starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Remove queryset caching support #5017

New Issue
Closed
opened 2025-12-29 19:23:12 +01:00 by adam · 7 comments
adam commented 2025-12-29 19:23:12 +01:00
Owner
Copy Link

Originally created by @jeremystretch on GitHub (Jun 21, 2021).

Originally assigned to: @jeremystretch on GitHub.

Proposed Changes

Remove the support for query caching that was introduced in v2.6 under #2647. This would remove django-cacheops as a dependency and eliminate the REDS['caching'] and CACHE_TIMEOUT configuration parameters.

Justification

Since its implementation, we have been constantly chasing numerous bugs attributable to limitations in the caching engine. I have addressed some of these issues within django-cacheops itself, however we continue to run into problems, particularly where complex workflows are involved.

In my opinion, it does not make sense to continue burning time chasing these bugs to support a feature with relatively little benefit. I would prefer to optimize NetBox itself where necessary to meet performance expectations rather than trying to keep the caching layer intact. I've opened this issue for discussion to see what the community thinks.

Originally created by @jeremystretch on GitHub (Jun 21, 2021). Originally assigned to: @jeremystretch on GitHub. ### Proposed Changes Remove the support for query caching that was introduced in v2.6 under #2647. This would remove django-cacheops as a dependency and eliminate the ~`REDS['caching']` and~ `CACHE_TIMEOUT` configuration parameters. ### Justification Since its implementation, we have been constantly chasing [numerous bugs](https://github.com/netbox-community/netbox/issues?q=is%3Aissue+caching+label%3A%22type%3A+bug%22+) attributable to limitations in the caching engine. I have addressed some of these issues within django-cacheops itself, however we continue to run into problems, particularly where complex workflows are involved. In my opinion, it does not make sense to continue burning time chasing these bugs to support a feature with relatively little benefit. I would prefer to optimize NetBox itself where necessary to meet performance expectations rather than trying to keep the caching layer intact. I've opened this issue for discussion to see what the community thinks.
adam added the status: acceptedtype: deprecation labels 2025-12-29 19:23:12 +01:00
adam closed this issue 2025-12-29 19:23:12 +01:00
adam commented 2025-12-29 19:23:13 +01:00
Author
Owner
Copy Link

@jhujhiti commented on GitHub (Jun 21, 2021):

Are there any quantitative numbers on how much the removal of caching would hurt specific API endpoints, or a way to disable the cache in an existing 2.11 environment to evaluate what the impact of this change would be? As a user, I'm certainly in favor of reducing the complexity of the stack, but my deployment has a number of identical queries run by external monitoring repeatedly and losing the cache there could put a lot more easily-cached load on the database. (Although I do suspect Redis isn't making much difference, I don't have a way to prove it.)

@jhujhiti commented on GitHub (Jun 21, 2021): Are there any quantitative numbers on how much the removal of caching would hurt specific API endpoints, or a way to disable the cache in an existing 2.11 environment to evaluate what the impact of this change would be? As a user, I'm certainly in favor of reducing the complexity of the stack, but my deployment has a number of identical queries run by external monitoring repeatedly and losing the cache there could put a lot more easily-cached load on the database. (Although I do suspect Redis isn't making much difference, I don't have a way to prove it.)
adam commented 2025-12-29 19:23:13 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jun 21, 2021):

@jhujhiti You can test on your own deployment by toggling the caching feature on and off. Setting CACHE_TIMEOUT = 0 in configuration.py and restarting the NetBox service will disable caching.

@jeremystretch commented on GitHub (Jun 21, 2021): @jhujhiti You can test on your own deployment by toggling the caching feature on and off. Setting `CACHE_TIMEOUT = 0` in `configuration.py` and restarting the NetBox service will disable caching.
adam commented 2025-12-29 19:23:13 +01:00
Author
Owner
Copy Link

@jhujhiti commented on GitHub (Jun 22, 2021):

Great! No discernible impact on the 95th or 50th percentile latency on the API endpoints I've been monitoring. Consider this a vote in favor.

@jhujhiti commented on GitHub (Jun 22, 2021): Great! No discernible impact on the 95th or 50th percentile latency on the API endpoints I've been monitoring. Consider this a vote in favor.
adam commented 2025-12-29 19:23:14 +01:00
Author
Owner
Copy Link

@dreng commented on GitHub (Jun 24, 2021):

Very low impact in our environment. Not noticeable in every day use. I'd say: kick it!

@dreng commented on GitHub (Jun 24, 2021): Very low impact in our environment. Not noticeable in every day use. I'd say: kick it!
adam commented 2025-12-29 19:23:14 +01:00
Author
Owner
Copy Link

@maximumG commented on GitHub (Jul 1, 2021):

As mentioned in issue #6651. The check_release process will be impacted by cache removal.

However check_release can easily rely on RQ result caching (result_ttl) and RQ registries to achieve the same result as the current implementation.

When browsing the home page it is possible to check in the RQ registries for the latest check_release job and its corresponding result. If the result has expired (> result_ttl time), we can queue a new check_release job.

I think that setting the check_release job with a result_ttl to 86400s is acceptable to check for an update every day.

@maximumG commented on GitHub (Jul 1, 2021): As mentioned in issue #6651. The [check_release](https://github.com/netbox-community/netbox/blob/develop/netbox/utilities/background_tasks.py) process will be impacted by cache removal. However _check_release_ can easily rely on RQ [result caching](https://python-rq.org/docs/results/) (`result_ttl`) and RQ [registries](https://python-rq.org/docs/job_registries/) to achieve the same result as the current implementation. When browsing the home page it is possible to check in the RQ registries for the latest `check_release` job and its corresponding result. If the result has expired (> result_ttl time), we can queue a new `check_release` job. I think that setting the check_release job with a result_ttl to 86400s is acceptable to check for an update every day.
adam commented 2025-12-29 19:23:14 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Jul 2, 2021):

Just to add a +1 here. Databases like postgres have very good internal cache mechanisms, and I never saw any benefit in adding a layer on top.

Caching of HTML rendering might have some justification, but only for pages which are viewed by multiple people and fetched repeatedly - which would probably only be the front page, and even then it varies by the user's permissions.

@candlerb commented on GitHub (Jul 2, 2021): Just to add a +1 here. Databases like postgres have very good internal cache mechanisms, and I never saw any benefit in adding a layer on top. Caching of HTML rendering *might* have some justification, but only for pages which are viewed by multiple people and fetched repeatedly - which would probably only be the front page, and even then it varies by the user's permissions.
adam commented 2025-12-29 19:23:14 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jul 7, 2021):

I should clarify that this is just removing queryset caching. We will still maintain a Redis-based cache for generic caching functions (e.g. checking for the latest NetBox release). We will likely replace django-cacheops with django-redis, which includes support for Redis Sentinel.

@jeremystretch commented on GitHub (Jul 7, 2021): I should clarify that this is just removing _queryset_ caching. We will still maintain a Redis-based cache for generic caching functions (e.g. checking for the latest NetBox release). We will likely replace django-cacheops with [django-redis](https://github.com/jazzband/django-redis), which includes support for Redis Sentinel.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: deprecation
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#5017
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?