starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Search fails due to search cache populated with non-existent models after plugin removal ('NoneType' object has no attribute 'objects') #8486

New Issue
Closed
opened 2025-12-29 20:37:19 +01:00 by adam · 9 comments
adam commented 2025-12-29 20:37:19 +01:00
Owner
Copy Link

Originally created by @pv2b on GitHub (Aug 18, 2023).

Originally assigned to: @abhi1693 on GitHub.

NetBox version

v3.5.8

Python version

Steps to Reproduce

Note: The full and exact repro steps aren't verified right now, but I agreed with @kkthxbye-code that I should file this as a bug anyway, as per our discussion on Slack.

  1. Install a plugin, such as netbox-topology-views
  2. Create a device, and make sure that somehow a "coordinate" is saved for that device. (Don't have exact steps for this right now.)
  3. Uninstall the plugin.
  4. Search for the name of the device that was created.

Expected Behavior

I expect to get a page of search results including that object.

Observed Behavior

I an get error screen with the python exception 'NoneType' object has no attribute 'objects'

Originally created by @pv2b on GitHub (Aug 18, 2023). Originally assigned to: @abhi1693 on GitHub. ### NetBox version v3.5.8 ### Python version ### Steps to Reproduce Note: The full and exact repro steps aren't verified right now, but I agreed with @kkthxbye-code that I should file this as a bug anyway, as per our [discussion on Slack](https://netdev-community.slack.com/archives/C01P0FRSXRV/p1692368097880469). 1. Install a plugin, such as [netbox-topology-views](https://github.com/mattieserver/netbox-topology-views) 2. Create a device, and make sure that somehow a "coordinate" is saved for that device. (Don't have exact steps for this right now.) 3. Uninstall the plugin. 4. Search for the name of the device that was created. ### Expected Behavior I expect to get a page of search results including that object. ### Observed Behavior I an get error screen with the python exception 'NoneType' object has no attribute 'objects'
adam added the type: bugstatus: acceptedseverity: medium labels 2025-12-29 20:37:19 +01:00
adam closed this issue 2025-12-29 20:37:19 +01:00
adam commented 2025-12-29 20:37:19 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 18, 2023):

Here's some analysis based on my conversation with @kkthxbye-code on Slack and a workaround that got me out of this condition:

The pre-condition seems to be that a plugin (in my case, (netbox-topology-views)[https://github.com/mattieserver/netbox-topology-views] has been installed, and has populated some data into the database. Specifically in my case, there were some "coordinates" (i.e. on a topology graph, not lat/lng) that were set on a device.

After the plugin was uninstalled, a search on the name of the object with coordinates associated with it would error out with this message:

'NoneType' object has no attribute 'objects'

By running this with DEBUG on, I was able to repro that this crashes on this line: https://github.com/netbox-community/netbox/blob/v3.5.8/netbox/utilities/fields.py#L117

The exception seems to be hit because ct.model_class() returns None, and therefore you can't get .objects on it. Looking closer at what ct is, it seems to be <ContentType: coordinate>, which I believe is a content type added by netbox-topology-views. And then of course when the plugin is uninstalled, the model class doesn't exist (ct.model_class() return None) any more, and thus causes a problem.

In trying to fix this problem on my system, I first tried just dropping all the tables relating to netbox_topology_views from Postgres, but that wasn't enough. With the help of @kkthxbye-code, we were able to identify that the problem was that, in addition to stale data still existing in the database (which I had previously manually removed), cached data related to this was still present in the extras_cachedvalue table.

We expected that rebuilding the search index using python netbox/manage.py reindex would fix this, but that didn't happen. As a workaround, I truncated the entire extras_cachedvalue table and then ran reindex to build a search index from scratch, and that seems to have got rid of the issue completely.

From my analysis there's at least few places where this could be addressed and my initial thoughts for how to address this:

  1. At the point where the exception was thrown, maybe a check for ct.model_class() being None before referencing it would help steer the search code into some less crashy behaviour when this condition is encountered.
  2. The python netbox/manage.py reindex should clear out cached values for models that no longer exist.
  3. Maybe some kind of housekeeping task that checks for cached data with non-present models in the index? Maybe as a self-check on startup?
@pv2b commented on GitHub (Aug 18, 2023): Here's some analysis based on my conversation with @kkthxbye-code on Slack and a workaround that got me out of this condition: The pre-condition seems to be that a plugin (in my case, (netbox-topology-views)[https://github.com/mattieserver/netbox-topology-views] has been installed, and has populated some data into the database. Specifically in my case, there were some "coordinates" (i.e. on a topology graph, not lat/lng) that were set on a device. After the plugin was uninstalled, a search on the name of the object with coordinates associated with it would error out with this message: 'NoneType' object has no attribute 'objects' By running this with DEBUG on, I was able to repro that this crashes on this line: https://github.com/netbox-community/netbox/blob/v3.5.8/netbox/utilities/fields.py#L117 The exception seems to be hit because `ct.model_class()` returns `None`, and therefore you can't get `.objects` on it. Looking closer at what `ct` is, it seems to be `<ContentType: coordinate>`, which I believe is a content type added by netbox-topology-views. And then of course when the plugin is uninstalled, the model class doesn't exist (`ct.model_class()` return `None`) any more, and thus causes a problem. In trying to fix this problem on my system, I first tried just dropping all the tables relating to netbox_topology_views from Postgres, but that wasn't enough. With the help of @kkthxbye-code, we were able to identify that the problem was that, in addition to stale data still existing in the database (which I had previously manually removed), cached data related to this was still present in the `extras_cachedvalue` table. We expected that rebuilding the search index using `python netbox/manage.py reindex` would fix this, but that didn't happen. As a workaround, I truncated the entire `extras_cachedvalue` table and then ran reindex to build a search index from scratch, and that seems to have got rid of the issue completely. From my analysis there's at least few places where this could be addressed and my initial thoughts for how to address this: 1. At the point where the exception was thrown, maybe a check for `ct.model_class()` being `None` before referencing it would help steer the search code into some less crashy behaviour when this condition is encountered. 2. The `python netbox/manage.py reindex` should clear out cached values for models that no longer exist. 3. Maybe some kind of housekeeping task that checks for cached data with non-present models in the index? Maybe as a self-check on startup?
adam commented 2025-12-29 20:37:20 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 18, 2023):

Oh by the way, I've saved a copy of the database that is corrupted in this state, so in case the proposed repro doesn't quite work out, that data will be available for more investigation. However I'm not comfortable sharing this SQL dump because it contains confidential information.

@pv2b commented on GitHub (Aug 18, 2023): Oh by the way, I've saved a copy of the database that is corrupted in this state, so in case the proposed repro doesn't quite work out, that data will be available for more investigation. However I'm not comfortable sharing this SQL dump because it contains confidential information.
adam commented 2025-12-29 20:37:20 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 18, 2023):

For anyone else running into this, until there's a code fix for this, here's how you can completely re-build the search index from scratch. That should clear this issue up. However, as for any time you're working with nbshell and and working at this level, and especially when running commands posted by some random guy on the Internet, make sure you have a full backup of your system, in case something goes off the rails.

  1. Issue the following commands in nbshell to completely clear the search index. (This will completely break search until a reindex is run, which is the next step.)
from netbox.search.backends import search_backend
search_backend.clear()
exit()
  1. Issue the following command to re-build the search index. This might take some time if you have a lot of data and/or a slow computer.
python manage.py reindex

This has worked for me and for one other person who's had this issue.

@pv2b commented on GitHub (Aug 18, 2023): For anyone else running into this, until there's a code fix for this, here's how you can completely re-build the search index from scratch. That should clear this issue up. However, as for any time you're working with nbshell and and working at this level, and especially when running commands posted by some random guy on the Internet, **make sure you have a full backup of your system**, in case something goes off the rails. 1. Issue the following commands in [nbshell](https://demo.netbox.dev/static/docs/administration/netbox-shell/) to completely clear the search index. (This will completely break search until a reindex is run, which is the next step.) ``` from netbox.search.backends import search_backend search_backend.clear() exit() ``` 2. Issue the following command to re-build the search index. This might take some time if you have a lot of data and/or a slow computer. ``` python manage.py reindex ``` This has worked for me and for one other person who's had this issue.
adam commented 2025-12-29 20:37:20 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 18, 2023):

FYI: I've written up a fix for the missing "None" check (and I found a second one as well) but due to project rules I won't be able to open a real PR until this has been accepted, so I've opened it as a draft PR.
https://github.com/netbox-community/netbox/pull/13508

@pv2b commented on GitHub (Aug 18, 2023): FYI: I've written up a fix for the missing "None" check (and I found a second one as well) but due to project rules I won't be able to open a real PR until this has been accepted, so I've opened it as a draft PR. https://github.com/netbox-community/netbox/pull/13508
adam commented 2025-12-29 20:37:20 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Aug 21, 2023):

Replicates fine, thank you for the thorough report. My steps:

  • pip install netbox-topology-views==3.6.1
  • Add netbox-topology-views to PLUGINS and PLUGINS_CONFIG
  • Start netbox, go to Coordinates and just add one.
  • Search for the group name.
  • Remove plugin from PLUGINS

This issue is similar to https://github.com/netbox-community/netbox/issues/11335 - fixed by https://github.com/netbox-community/netbox/pull/11709 and https://github.com/netbox-community/netbox/pull/13117

I'm not sure your draft PR fixes the underlying issue and trying it out quick doesn't fix it, but does move the issue to another method. Maybe @jeremystretch or @abhi1693 can have a look, I'm not really that familiar with the custom code we have for gfk lookups.

@kkthxbye-code commented on GitHub (Aug 21, 2023): Replicates fine, thank you for the thorough report. My steps: * `pip install netbox-topology-views==3.6.1` * Add `netbox-topology-views` to `PLUGINS` and `PLUGINS_CONFIG` * Start netbox, go to `Coordinates` and just add one. * Search for the group name. * Remove plugin from `PLUGINS` This issue is similar to https://github.com/netbox-community/netbox/issues/11335 - fixed by https://github.com/netbox-community/netbox/pull/11709 and https://github.com/netbox-community/netbox/pull/13117 I'm not sure your draft PR fixes the underlying issue and trying it out quick doesn't fix it, but does move the issue to another method. Maybe @jeremystretch or @abhi1693 can have a look, I'm not really that familiar with the custom code we have for gfk lookups.
adam commented 2025-12-29 20:37:21 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 21, 2023):

Replicates fine, thank you for the thorough report. My steps:

* `pip install netbox-topology-views==3.6.1`

* Add `netbox-topology-views` to `PLUGINS` and `PLUGINS_CONFIG`

* Start netbox, go to `Coordinates` and just add one.

* Search for the group name.

This issue is similar to #11335 - fixed by #11709 and #13117

I'm not sure your draft PR fixes the underlying issue and trying it out quick doesn't fix it, but does move the issue to another method. Maybe @jeremystretch or @abhi1693 can have a look, I'm not really that familiar with the custom code we have for gfk lookups.

Depends what you mean is the underlying issue, in my opinion there are at least two underlying issues:

  1. There's old data left over in the search cache for content types that no longer exists.
  2. There's no error checking to handle the case where content type no longer exists.

Cleaning up the search cache will address 1, and adding checks for "None" will address 2, it's possible your repro is slightly different than the one I ran into myself, because my fix "works for me". But I did a quick code search before and I found a lot of cases of the pattern of just calling get_model() on a contenttype and expecting it to work, so it's possible that I might have missed a code path.

In my opinion it should be fixed on both ends.

@pv2b commented on GitHub (Aug 21, 2023): > Replicates fine, thank you for the thorough report. My steps: > > * `pip install netbox-topology-views==3.6.1` > > * Add `netbox-topology-views` to `PLUGINS` and `PLUGINS_CONFIG` > > * Start netbox, go to `Coordinates` and just add one. > > * Search for the group name. > > > This issue is similar to #11335 - fixed by #11709 and #13117 > > I'm not sure your draft PR fixes the underlying issue and trying it out quick doesn't fix it, but does move the issue to another method. Maybe @jeremystretch or @abhi1693 can have a look, I'm not really that familiar with the custom code we have for gfk lookups. Depends what you mean is the underlying issue, in my opinion there are at least two underlying issues: 1. There's old data left over in the search cache for content types that no longer exists. 2. There's no error checking to handle the case where content type no longer exists. Cleaning up the search cache will address 1, and adding checks for "None" will address 2, it's possible your repro is slightly different than the one I ran into myself, because my fix "works for me". But I did a quick code search before and I found a lot of cases of the pattern of just calling get_model() on a contenttype and expecting it to work, so it's possible that I might have missed a code path. In my opinion it should be fixed on both ends.
adam commented 2025-12-29 20:37:21 +01:00
Author
Owner
Copy Link

@pv2b commented on GitHub (Aug 21, 2023):

Replicates fine, thank you for the thorough report. My steps:

* `pip install netbox-topology-views==3.6.1`

* Add `netbox-topology-views` to `PLUGINS` and `PLUGINS_CONFIG`

* Start netbox, go to `Coordinates` and just add one.

* Search for the group name.

Are you missing a repro step here, did you not need to uninstall the plugin?

@pv2b commented on GitHub (Aug 21, 2023): > Replicates fine, thank you for the thorough report. My steps: > > * `pip install netbox-topology-views==3.6.1` > > * Add `netbox-topology-views` to `PLUGINS` and `PLUGINS_CONFIG` > > * Start netbox, go to `Coordinates` and just add one. > > * Search for the group name. Are you missing a repro step here, did you not need to uninstall the plugin?
adam commented 2025-12-29 20:37:21 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Aug 21, 2023):

Are you missing a repro step here, did you not need to uninstall the plugin?

Just missed the last step, added it now, thanks.

@kkthxbye-code commented on GitHub (Aug 21, 2023): > Are you missing a repro step here, did you not need to uninstall the plugin? Just missed the last step, added it now, thanks.
adam commented 2025-12-29 20:37:21 +01:00
Author
Owner
Copy Link

@bitcollector1 commented on GitHub (Aug 21, 2023):

This fix worked for me but I've not had the time to restore the old DB and try to narrow down the culprit with DEBUG. That said it looks like you fine folks have a good grasp on this issue.

I just wanted to say thanks again to @pv2b for the fix as it was annoying me for months!

@bitcollector1 commented on GitHub (Aug 21, 2023): This fix worked for me but I've not had the time to restore the old DB and try to narrow down the culprit with DEBUG. That said it looks like you fine folks have a good grasp on this issue. I just wanted to say thanks again to @pv2b for the fix as it was annoying me for months!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label severity: medium status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8486
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?