starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Allow redefining internally used background queues #7282

New Issue
Closed
opened 2025-12-29 20:21:12 +01:00 by adam · 5 comments
adam commented 2025-12-29 20:21:12 +01:00
Owner
Copy Link

Originally created by @kkthxbye-code on GitHub (Nov 24, 2022).

Originally assigned to: @kkthxbye-code on GitHub.

NetBox version

v3.3.8

Feature type

Change to existing functionality

Proposed functionality

Allow changing the used RQ queue for webhook scheduling, custom scripts and reports to a specific queue.

Use case

Currently script/report and webhook execution uses the same queue ("default"). One RQ worker will only ever run one job at a time:

Each worker will process a single job at a time. Within a worker, there is no concurrent processing going on. If you want to perform jobs concurrently, simply start more workers.

For webhooks we want to only ever run one job at the time, as the order is very important to avoid a webhook referencing a related object that hasn't been sent in a webhook yet.

For scripts/reports the user might not care about the order of executions, but as it stands now if the user runs one rqworker, long running scripts will block both webhooks and other scripts from execution before it is done.

If the user starts multiple rqworkers that will allow multiple scripts to run at a time, but it will also allow multiple webhooks to be sent at a time, which in turn can create situations where a newer webhook is done processing before an older one - imagine three workers all picking up a job at the same time, the webhook will potentially send the HTTP request at roughly the same time, which one goes first depends on CPU scheduling or other external factors.

Allowing webhooks to be configured to use a seperate queue will allow the user to start one worker for webhooks and multiple workers for script/report execution, without risking data consistency issues with webhooks.

Database changes

None

External dependencies

None

Originally created by @kkthxbye-code on GitHub (Nov 24, 2022). Originally assigned to: @kkthxbye-code on GitHub. ### NetBox version v3.3.8 ### Feature type Change to existing functionality ### Proposed functionality Allow changing the used RQ queue for webhook scheduling, custom scripts and reports to a specific queue. ### Use case Currently script/report and webhook execution uses the same queue ("default"). One RQ worker will only ever run [one job at a time](https://python-rq.org/docs/workers/): > Each worker will process a single job at a time. Within a worker, there is no concurrent processing going on. If you want to perform jobs concurrently, simply start more workers. For webhooks we want to only ever run one job at the time, as the order is very important to avoid a webhook referencing a related object that hasn't been sent in a webhook yet. For scripts/reports the user might not care about the order of executions, but as it stands now if the user runs one rqworker, long running scripts will block both webhooks and other scripts from execution before it is done. If the user starts multiple rqworkers that will allow multiple scripts to run at a time, but it will also allow multiple webhooks to be sent at a time, which in turn can create situations where a newer webhook is done processing before an older one - imagine three workers all picking up a job at the same time, the webhook will potentially send the HTTP request at roughly the same time, which one goes first depends on CPU scheduling or other external factors. Allowing webhooks to be configured to use a seperate queue will allow the user to start one worker for webhooks and multiple workers for script/report execution, without risking data consistency issues with webhooks. ### Database changes None ### External dependencies None
adam added the status: acceptedtype: feature labels 2025-12-29 20:21:12 +01:00
adam closed this issue 2025-12-29 20:21:13 +01:00
adam commented 2025-12-29 20:21:13 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Nov 24, 2022):

If accepted I'll implement it. We also need to sync up with netbox-docker as there needs to be changes to the default docker-compose file to add the new queue.

@kkthxbye-code commented on GitHub (Nov 24, 2022): If accepted I'll implement it. We also need to sync up with netbox-docker as there needs to be changes to the default docker-compose file to add the new queue.
adam commented 2025-12-29 20:21:13 +01:00
Author
Owner
Copy Link

@julianstolp commented on GitHub (Nov 24, 2022):

If adding the ability to set the queue priority for webhooks, would it be possible to add the ability to set the priority in general at the URL with maybe the flag ?priority=high, ?priority=default, ?priority=low. The usecase would be to prioritize manual triggered scripts to run before the background scripts.

@julianstolp commented on GitHub (Nov 24, 2022): If adding the ability to set the queue priority for webhooks, would it be possible to add the ability to set the priority in general at the URL with maybe the flag ?priority=high, ?priority=default, ?priority=low. The usecase would be to prioritize manual triggered scripts to run before the background scripts.
adam commented 2025-12-29 20:21:13 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Nov 24, 2022):

@Bierfass - This feature request would be to set a static queue for webhooks, as there in my opinion is no reason to want to concurrently execute webhooks because of the mentioned pitfalls of the ordering not being perserved.

Allowing you to schedule scripts and reports in different queues is a good idea though, but would be another feature request. It would probably either be set in the Meta class of the script or as a form selection when executing the script though. Also note that the low, default and high queues only work as intended if running more than one rqworker as far as I can tell.

@kkthxbye-code commented on GitHub (Nov 24, 2022): @Bierfass - This feature request would be to set a static queue for webhooks, as there in my opinion is no reason to want to concurrently execute webhooks because of the mentioned pitfalls of the ordering not being perserved. Allowing you to schedule scripts and reports in different queues is a good idea though, but would be another feature request. It would probably either be set in the Meta class of the script or as a form selection when executing the script though. Also note that the low, default and high queues only work as intended if running more than one rqworker as far as I can tell.
adam commented 2025-12-29 20:21:14 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Dec 1, 2022):

Just recapping our discussion from today's maintainers' meeting. This is a great idea and I'd like to be able to configure queues per process type. For example, something like this in configuration.py:

QUEUE_MAPPINGS = {
    'webhooks': 'my-custom-queue',
    'reports': 'high',
}

This would grant users a good amount of flexibility to create and assign their own queues as needed (each of which can be serviced by however many RQ workers is appropriate).

@jeremystretch commented on GitHub (Dec 1, 2022): Just recapping our discussion from today's maintainers' meeting. This is a great idea and I'd like to be able to configure queues per process type. For example, something like this in `configuration.py`: ```python QUEUE_MAPPINGS = { 'webhooks': 'my-custom-queue', 'reports': 'high', } ``` This would grant users a good amount of flexibility to create and assign their own queues as needed (each of which can be serviced by however many RQ workers is appropriate).
adam commented 2025-12-29 20:21:14 +01:00
Author
Owner
Copy Link

@hagbarddenstore commented on GitHub (Dec 1, 2022):

Will this be extended to plugins as well, so that they register their keys in that map?

@hagbarddenstore commented on GitHub (Dec 1, 2022): Will this be extended to plugins as well, so that they register their keys in that map?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#7282
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?