Use background processing for non-real-time tasks (search caching, change logging, etc.) #7364

New Issue

adam · 2025-12-29T20:22:20+01:00

adam commented

2025-12-29 20:22:20 +01:00

Originally created by @jeremystretch on GitHub (Dec 15, 2022).

NetBox version

v3.4.0

Feature type

Change to existing functionality

Proposed functionality

The purpose of this FR is to explore the feasibility and value of converting certain tasks which currently happen inline with the request/response cycle to be handled in the background. The two most prominent examples are:

Search indexing
Change logging

Both of these currently happen immediately in response to the creation, modification, or deletion of an object in NetBox. Rather than performing these tasks in real-time, each could be queued to be processed by a background worker, similar to how we currently handle webhooks.

Change logging, for instance, is currently performed by the handle_changed_object and handle_deleted_object signal receivers. These each create and save to the database a new ObjectChange instance in response to some change being applied. Rather than doing this directly, the handlers could instead queue a job to effect the creation of the ObjectChange, allowing us to proceed with delivering a response immediately. A background worker would then extract this job from the queue for processing out of sync with the request/response process.

Use case

Queuing these functions for background processing would remove some amount of overhead from the request/response cycle, reducing the overall time required to process add, change, and delete requests. In some cases, particularly requests which effect the creation of many new objects at once, the reduction of response time could be drastic.

However, this change would also impose a delay between the time a change is made and when it is reflected in other areas of the application. For instance, it's possible that a recently modified object might not have its changelog updated until some noticeable time after the change was made, depending on the business and resourcing of the background workers.

Database changes

No response

External dependencies

No response

Originally created by @jeremystretch on GitHub (Dec 15, 2022). ### NetBox version v3.4.0 ### Feature type Change to existing functionality ### Proposed functionality The purpose of this FR is to explore the feasibility and value of converting certain tasks which currently happen inline with the request/response cycle to be handled in the background. The two most prominent examples are: * Search indexing * Change logging Both of these currently happen immediately in response to the creation, modification, or deletion of an object in NetBox. Rather than performing these tasks in real-time, each could be queued to be processed by a background worker, similar to how we currently handle webhooks. Change logging, for instance, is currently performed by the [`handle_changed_object`](https://github.com/netbox-community/netbox/blob/f8685ad7aa276f1ccc7e15f6d2e65c2d0f1ad897/netbox/extras/signals.py#L38) and [`handle_deleted_object`](https://github.com/netbox-community/netbox/blob/f8685ad7aa276f1ccc7e15f6d2e65c2d0f1ad897/netbox/extras/signals.py#L98) signal receivers. These each create and save to the database a new ObjectChange instance in response to some change being applied. Rather than doing this directly, the handlers could instead queue a job to effect the creation of the ObjectChange, allowing us to proceed with delivering a response immediately. A background worker would then extract this job from the queue for processing out of sync with the request/response process. ### Use case Queuing these functions for background processing would remove some amount of overhead from the request/response cycle, reducing the overall time required to process add, change, and delete requests. In some cases, particularly requests which effect the creation of many new objects at once, the reduction of response time could be drastic. However, this change would also impose a delay between the time a change is made and when it is reflected in other areas of the application. For instance, it's possible that a recently modified object might not have its changelog updated until some noticeable time after the change was made, depending on the business and resourcing of the background workers. ### Database changes _No response_ ### External dependencies _No response_

adam added the type: feature pending closure status: under review labels 2025-12-29 20:22:20 +01:00

adam closed this issue

2025-12-29 20:22:20 +01:00

adam commented

2025-12-29 20:22:21 +01:00

@hagbarddenstore commented on GitHub (Dec 15, 2022):

I use this method in my plugins and I haven’t heard any complaints so far.Unless you’re on the Changelog page it’s unlikely you’re gonna notice it at all, so I say go for it!

@hagbarddenstore commented on GitHub (Dec 15, 2022): I use this method in my plugins and I haven’t heard any complaints so far.Unless you’re on the Changelog page it’s unlikely you’re gonna notice it at all, so I say go for it!

adam commented

2025-12-29 20:22:23 +01:00

@kkthxbye-code commented on GitHub (Dec 18, 2022):

Example of a user experiencing issues with bulk import: https://github.com/netbox-community/netbox/discussions/11198

I explained the reason here: https://github.com/netbox-community/netbox/discussions/11198#discussioncomment-4439757

@kkthxbye-code commented on GitHub (Dec 18, 2022): Example of a user experiencing issues with bulk import: https://github.com/netbox-community/netbox/discussions/11198 I explained the reason here: https://github.com/netbox-community/netbox/discussions/11198#discussioncomment-4439757

adam commented

2025-12-29 20:22:23 +01:00

@tyler-8 commented on GitHub (Dec 27, 2022):

Search indexing makes a lot of sense for background workers in different processes from the request/response cycle.

The object changelog I'm less certain about. While I haven't yet used the async view capabilities of Django today, it does have them - and they will continue to be implemented across the codebase. The object changelog functionality would be better suited for an asgi environment. IMO it should be closely tied to the request/response as it the way changes (as a result of request/response) are tracked.

Pushing changelog to separate worker processes would require more worker management to ensure there is always a worker available - in competition with scripts, reports, housekeeping, plugins, etc. And if the worker(s) process or redis is down for unrelated reasons, you potentially have data loss (of the object changes) without any clear indicators in the request/response cycle.

My recommendation would be to hold off on moving the object changelogs to the rq workers and either table it for future async-improvements in Django, or if/when asyncio/asgi is introduced into NetBox.

@tyler-8 commented on GitHub (Dec 27, 2022): Search indexing makes a lot of sense for background workers in different processes from the request/response cycle. The object changelog I'm less certain about. While I haven't yet used the async view capabilities of Django today, [it does have them](https://docs.djangoproject.com/en/4.1/topics/async/) - and they will continue to be implemented across the codebase. The object changelog functionality would be better suited for an `asgi` environment. IMO it _should_ be closely tied to the request/response as it the way changes (as a result of request/response) are tracked. Pushing changelog to separate worker processes would require more worker management to ensure there is _always_ a worker available - in competition with scripts, reports, housekeeping, plugins, etc. And if the worker(s) process or redis is down for unrelated reasons, you potentially have data loss (of the object changes) without any clear indicators in the request/response cycle. My recommendation would be to hold off on moving the object changelogs to the rq workers and either table it for future async-improvements in Django, or if/when `asyncio/asgi` is introduced into NetBox.

adam commented

2025-12-29 20:22:23 +01:00

@jeremystretch commented on GitHub (Dec 27, 2022):

The object changelog functionality would be better suited for an asgi environment.

Granted I haven't looked too much into async workflows yet, however I don't believe leveraging them would allow us to actually return the response any faster (other than token gains resulting from query optimization); we would still need to wait for the change record to be recorded before returning a response to the user. Hence the desire to remove the recording from the request/response cycle entirely.

Pushing changelog to separate worker processes would require more worker management to ensure there is always a worker available

This is already the case; the RQ worker is part of the NetBox deployment. Heavy workloads may require the configuration of additional workers, but this is in exchange for unlocking greater performance on the front end.

And if the worker(s) process or redis is down for unrelated reasons, you potentially have data loss (of the object changes) without any clear indicators in the request/response cycle.

We can handle this just as we do for scripts and reports today: If the Redis queue is unavailable, changes will not be permitted. (I've noticed that lot of people misconceive Redis as an optional component of NetBox, where in reality is a first class citizen just as the PostgreSQL database.) We can also detect the operation of RQ workers, though their absence is less impeding since the backlog will be preserved until a worker comes online to process it.

@jeremystretch commented on GitHub (Dec 27, 2022): > The object changelog functionality would be better suited for an asgi environment. Granted I haven't looked too much into async workflows yet, however I don't believe leveraging them would allow us to actually return the response any faster (other than token gains resulting from query optimization); we would still need to wait for the change record to be recorded before returning a response to the user. Hence the desire to remove the recording from the request/response cycle entirely. > Pushing changelog to separate worker processes would require more worker management to ensure there is always a worker available This is already the case; the RQ worker is part of the NetBox deployment. Heavy workloads may require the configuration of _additional_ workers, but this is in exchange for unlocking greater performance on the front end. > And if the worker(s) process or redis is down for unrelated reasons, you potentially have data loss (of the object changes) without any clear indicators in the request/response cycle. We can handle this just as we do for scripts and reports today: If the Redis queue is unavailable, changes will not be permitted. (I've noticed that lot of people misconceive Redis as an optional component of NetBox, where in reality is a first class citizen just as the PostgreSQL database.) We can also detect the operation of RQ workers, though their absence is less impeding since the backlog will be preserved until a worker comes online to process it.

adam commented

2025-12-29 20:22:23 +01:00

@tyler-8 commented on GitHub (Dec 27, 2022):

Granted I haven't looked too much into async workflows yet, however I don't believe leveraging them would allow us to actually return the response any faster (other than token gains resulting from query optimization); we would still need to wait for the change record to be recorded before returning a response to the user. Hence the desire to remove the recording from the request/response cycle entirely.

The benefit would primarily be for the server itself. asgi is going to be able to handle more simultaneous requests, and their response time isn't chewing up the entire thread/process as in wsgi.

This is already the case; the RQ worker is part of the NetBox deployment. Heavy workloads may require the configuration of additional workers, but this is in exchange for unlocking greater performance on the front end.

Yes, however it does change the story of what a "minimum" deployment looks like IMO. Changelog service should come above all else, or at least that's how it's operated til now. Updates to the documentation could be enough here perhaps.

We can handle this just as we do for scripts and reports today: If the Redis queue is unavailable, changes will not be permitted.

That sounds reasonable.

I think https://github.com/netbox-community/netbox/issues/11254 may actually be important for this change. Returning the request ID in the responses gives you some sort of record of your change (other than the object itself that you're modifying) and a way to validate that your changes do in fact have related ObjectChange entries.

@tyler-8 commented on GitHub (Dec 27, 2022): > Granted I haven't looked too much into async workflows yet, however I don't believe leveraging them would allow us to actually return the response any faster (other than token gains resulting from query optimization); we would still need to wait for the change record to be recorded before returning a response to the user. Hence the desire to remove the recording from the request/response cycle entirely. The benefit would primarily be for the server itself. `asgi` is going to be able to handle more simultaneous requests, and their response time isn't chewing up the entire thread/process as in `wsgi`. > This is already the case; the RQ worker is part of the NetBox deployment. Heavy workloads may require the configuration of additional workers, but this is in exchange for unlocking greater performance on the front end. Yes, however it does change the story of what a "minimum" deployment looks like IMO. Changelog service should come above all else, or at least that's how it's operated til now. Updates to the documentation could be enough here perhaps. > We can handle this just as we do for scripts and reports today: If the Redis queue is unavailable, changes will not be permitted. That sounds reasonable. I think https://github.com/netbox-community/netbox/issues/11254 may actually be important for this change. Returning the request ID in the responses gives you some sort of record of your change (other than the object itself that you're modifying) and a way to validate that your changes do in fact have related `ObjectChange` entries.

adam commented

2025-12-29 20:22:24 +01:00

@github-actions[bot] commented on GitHub (Mar 28, 2023):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Mar 28, 2023): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).

adam commented

2025-12-29 20:22:24 +01:00

@github-actions[bot] commented on GitHub (Apr 27, 2023):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@github-actions[bot] commented on GitHub (Apr 27, 2023): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

Sign in to join this conversation.

Branches Tags

main

update-changelog-comments-docs

feature-removal-issue-type

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#7364