Allow plugin to create their own django-rq queues #5024

New Issue

adam · 2025-12-29T19:23:16+01:00

adam commented

2025-12-29 19:23:16 +01:00

Originally created by @maximumG on GitHub (Jun 22, 2021).

Originally assigned to: @maximumG on GitHub.

NetBox version

v2.11.7

Feature type

New functionality

Proposed functionality

I'd like to propose the ability for netbox plugin to define some dedicated django-rq queue(s). This new feature has already been discussed in #4908 but was missing a owner back in time.

The idea would be to store queue names in a new attribute inside the PluginConfig class. This attributes would be a list in order to defined several queue to create by django-rq.

In order to avoid queue name collision, the actual queue names will be prepended by the plugin name.

As an exemple the following config will lead to 3 queues being created using django-rq:

my_super_plugin.high
my_super_plugin.medium
my_super_plugin.low

MyPluginConfig(PluginConfig):
    name = "my_super_plugin"
    queues = [
        "low"
        "medium"
        "high"
    ]

The creation of the queues will need to override django settings (RQ_QUEUES) and can be done in the ready method of the PluginConfig class.

We can also override django RQ_QUEUES settings directly in the settings.py, during init of all plugins.

If you are interested in such feature I can take care of a PR.

Use case

All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

Database changes

No response

External dependencies

No response

Originally created by @maximumG on GitHub (Jun 22, 2021). Originally assigned to: @maximumG on GitHub. ### NetBox version v2.11.7 ### Feature type New functionality ### Proposed functionality I'd like to propose the ability for netbox plugin to define some dedicated _django-rq_ queue(s). This new feature has already been discussed in #4908 but was missing a owner back in time. The idea would be to store queue names in a new attribute inside the `PluginConfig` class. This attributes would be a `list` in order to defined several queue to create by django-rq. In order to avoid queue name collision, the actual queue names will be prepended by the plugin name. As an exemple the following config will lead to 3 queues being created using django-rq: * _my_super_plugin.high_ * _my_super_plugin.medium_ * _my_super_plugin.low_ ```python MyPluginConfig(PluginConfig): name = "my_super_plugin" queues = [ "low" "medium" "high" ] ``` The creation of the queues will need to override django settings ([RQ_QUEUES](https://github.com/netbox-community/netbox/blob/607039f043f0a43437db05aae18bae6f1ef49f85/netbox/netbox/settings.py#L571)) and can be done in the `ready` method of the `PluginConfig` class. We can also override django RQ_QUEUES settings directly in the `settings.py`, during [init of all plugins](https://github.com/netbox-community/netbox/blob/4292d88a9230789ef0c5af627e7245d2d3dd4bda/netbox/netbox/settings.py#L597). If you are interested in such feature I can take care of a PR. ### Use case All netbox plugins have the possibility to use the _default_ queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script. ### Database changes _No response_ ### External dependencies _No response_

adam added the status: accepted type: feature labels 2025-12-29 19:23:16 +01:00

adam closed this issue

2025-12-29 19:23:16 +01:00

adam commented

2025-12-29 19:23:17 +01:00

@maximumG commented on GitHub (Jun 30, 2021):

I already added this feature in my own netbox fork along with some new unittest functions to ensure the implementation is working as expected

If we agree on this I can move on and update the netbox documentation to add some notes regarding dedicated plugin queues.

@maximumG commented on GitHub (Jun 30, 2021): I already added this feature in [my own netbox fork](https://github.com/maximumG/netbox/commit/9a0bc730a3b3b60098a57fd0a0e180d19983ecc2) along with some new unittest functions to ensure the implementation is working as expected If we agree on this I can move on and update the netbox documentation to add some notes regarding dedicated plugin queues.

adam commented

2025-12-29 19:23:17 +01:00

@jeremystretch commented on GitHub (Jul 1, 2021):

Thanks for this!

All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively. We need a way to order queues among both NetBox core and one or more plugins in a predictable manner. For example, suppose two different plugins each introduce their own high- and low-priority queues. We don't want to end up ordering them as:

[default, plugin1-high, plugin1-low, plugin2-high, plugin2-low]

Without such a mechanism in place, plugins would be relying on administrators to manually reconfigure their rqworker invocations accordingly, which is sure to lead to frustration. I'm open to suggestions.

NetBox also currently has a check_releases queue that is used solely for checking for new releases. I don't recall the motivation for this design decision initially, but it's probably best moved to a scheduled task within the default queue using rq-scheduler anyway. (In fact, this may become necessary if we decided to remove caching support as discussed under #6639.)

@jeremystretch commented on GitHub (Jul 1, 2021): Thanks for this! > All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script. Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that `rqworker` services each queue respectively. We need a way to order queues among both NetBox core and one or more plugins in a predictable manner. For example, suppose two different plugins each introduce their own high- and low-priority queues. We don't want to end up ordering them as: ``` [default, plugin1-high, plugin1-low, plugin2-high, plugin2-low] ``` Without such a mechanism in place, plugins would be relying on administrators to manually reconfigure their `rqworker` invocations accordingly, which is sure to lead to frustration. I'm open to suggestions. NetBox also currently has a `check_releases` queue that is used solely for checking for new releases. I don't recall the motivation for this design decision initially, but it's probably best moved to a scheduled task within the default queue using [`rq-scheduler`](https://github.com/rq/rq-scheduler) anyway. (In fact, this may become necessary if we decided to remove caching support as discussed under #6639.)

adam commented

2025-12-29 19:23:17 +01:00

@maximumG commented on GitHub (Jul 1, 2021):

Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively.

You are absolutely right, I forgot to mention half of the proposed solution...sorry for this 😞

We need a way to order queues among both NetBox core and one or more plugins in a predictable manner.

From my point of view, queue administration should be tackled at the sysadmin level. What I had in mind in the first place is to have:

per default, an rqworker instance listening to queue default and check_release, to handle netbox core jobs (already the case now)
a new rqworker instance configured per plugin. This new instance must be configured to listen only to the plugin's queues

The second point implies that the plugin maintainer inform potential user that they must create a dedicated rqworker instance listening to the plugin queues (container / systemd unit). No matter the underlying implementation to spawn new rqworker it is really simple nowadays to add an additional container/pod (cloud) or systemd unit (on-premise) to handle each plugin background workload.

Even if creating new rqworker instance is a manual operation done by sysadmin, I see the following benefits:

Each plugin background workload is isolated and does not impact other plugin or netbox core workload
Netbox source code is "kept clean" and focused on IRM only, as we do not add some code to handle queueing system
We keep it relatively simple by separating concerns of Netbox code and system implementation details

@maximumG commented on GitHub (Jul 1, 2021): > Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively. You are absolutely right, I forgot to mention half of the proposed solution...sorry for this 😞 > We need a way to order queues among both NetBox core and one or more plugins in a predictable manner. From my point of view, queue administration should be tackled at the sysadmin level. What I had in mind in the first place is to have: * per default, an rqworker instance listening to queue `default` and `check_release`, to handle netbox core jobs (already the case now) * a new rqworker instance configured per plugin. This new instance must be configured to listen only to the plugin's queues The second point implies that the plugin maintainer inform potential user that they must create a dedicated rqworker instance listening to the plugin queues (container / systemd unit). No matter the underlying implementation to spawn new rqworker it is really simple nowadays to add an additional container/pod (cloud) or systemd unit (on-premise) to handle each plugin background workload. Even if creating new rqworker instance is a manual operation done by sysadmin, I see the following benefits: - Each plugin background workload is isolated and does not impact other plugin or netbox core workload - Netbox source code is "kept clean" and focused on IRM only, as we do not add some code to handle queueing system - We keep it relatively simple by separating concerns of Netbox code and system implementation details

adam commented

2025-12-29 19:23:17 +01:00

@jeremystretch commented on GitHub (Jul 1, 2021):

Maybe there's a middle ground, where NetBox provides its own high- and low-priority queues (in addition to the default queue) for consumption by plugins. NetBox's rqworker would service all three of these accordingly by default. Then I'd feel more comfortable putting the burden on the user to service any additional queues needed by plugins.

@jeremystretch commented on GitHub (Jul 1, 2021): Maybe there's a middle ground, where NetBox provides its own high- and low-priority queues (in addition to the default queue) for consumption by plugins. NetBox's rqworker would service all three of these accordingly by default. Then I'd feel more comfortable putting the burden on the user to service any additional queues needed by plugins.

adam commented

2025-12-29 19:23:18 +01:00

@maximumG commented on GitHub (Jul 1, 2021):

I will try to sum up our two ideas to define a proposal target design.

What Netbox would provides out of the box

Netbox would define 3 queues : default, high-priority and low-priority. The default queue is already present and dedicated for netbox internal-use (reports and scripts jobs). The high-priority and low-priority queues are provided out of the box to plugin developer if they need background job and do not want to implement their own queues.

Netbox would provides a single rq-worker (systemd/container) which listen to the 3 queues above. As stated by the RQ documentation, the queue order is important here and should stay as default, high-priority and low-priority. This way we ensure default job > high-priority jobs > low-priority jobs

What Netbox would allow for plugin developers

Plugin developers have the ability to define their own dedicated queue(s) in the plugin's config. This implies a new rq-worker instance must be created and configured to listen to the plugin dedicated queue(s). In such case, end-user of the plugin must ensure this rq-worker is installed and configured properly according to any plugin install documentation.

@maximumG commented on GitHub (Jul 1, 2021): I will try to sum up our two ideas to define a proposal target design. ### What Netbox would provides out of the box **Netbox would define 3 queues** : `default`, `high-priority` and `low-priority`. The `default` queue is already present and dedicated for netbox internal-use (reports and scripts jobs). The `high-priority` and `low-priority` queues are provided out of the box to plugin developer if they need background job and do not want to implement their own queues. **Netbox would provides a single _rq-worker_** (systemd/container) which listen to the 3 queues above. As stated by the [RQ documentation](https://python-rq.org/docs/workers/), the queue order is important here and should stay as `default`, `high-priority` and `low-priority`. This way we ensure `default` job > `high-priority` jobs > `low-priority` jobs ### What Netbox would allow for plugin developers Plugin developers have the ability to define their own dedicated queue(s) in the plugin's config. This implies a new rq-worker instance must be created and configured to listen to the plugin dedicated queue(s). In such case, end-user of the plugin must ensure this rq-worker is installed and configured properly according to any plugin install documentation.

adam commented

2025-12-29 19:23:18 +01:00

@jeremystretch commented on GitHub (Jul 1, 2021):

I like that! Just one tweak:

the queue order is important here and should stay as default, high-priority and low-priority.

I would arrange them high > default > low. NetBox would continue to queue webhooks, reports, etc. in the default queue.

@jeremystretch commented on GitHub (Jul 1, 2021): I like that! Just one tweak: > the queue order is important here and should stay as default, high-priority and low-priority. I would arrange them high > default > low. NetBox would continue to queue webhooks, reports, etc. in the default queue.

adam commented

2025-12-29 19:23:18 +01:00

@jeremystretch commented on GitHub (Jul 1, 2021):

I pinged the Docker folks for their take on this as well. Per @tobiasge in the #netbox-docker channel on the NetDev Slack:

In the docker-compose.yaml we have defined a container to run the default rqworker. See this configuration. To define additional workers one would need to add a new worker instance to the docker-compose.override.yml file, which is also explained in our documentation.
From that point of view, I think it is easy to add new workers for plugins. So no objections from me.

@jeremystretch commented on GitHub (Jul 1, 2021): I pinged the Docker folks for their take on this as well. Per @tobiasge in the #netbox-docker channel on the NetDev Slack: > In the docker-compose.yaml we have defined a container to run the default rqworker. See this configuration. To define additional workers one would need to add a new worker instance to the docker-compose.override.yml file, which is also explained in our documentation. > From that point of view, I think it is easy to add new workers for plugins. So no objections from me.

adam commented

2025-12-29 19:23:18 +01:00

@jeremystretch commented on GitHub (Jul 6, 2021):

I should also note that the PR for this should be based on the feature branch, as it will need to go into the next major release.

@jeremystretch commented on GitHub (Jul 6, 2021): I should also note that the PR for this should be based on the `feature` branch, as it will need to go into the next major release.

adam commented

2025-12-29 19:23:18 +01:00

@maximumG commented on GitHub (Jul 6, 2021):

Thanks for the update @jeremystretch

I'd also wanted to confirm where should I put some words in the documentation regarding this new feature (default queue + plugin related) ?

I was thinking about a new dedicated section inside the plugin page right after the caching configuration.

@maximumG commented on GitHub (Jul 6, 2021): Thanks for the update @jeremystretch I'd also wanted to confirm where should I put some words in the documentation regarding this new feature (default queue + plugin related) ? I was thinking about a new dedicated section inside [the plugin page](https://netbox.readthedocs.io/en/stable/plugins/development/) right after the caching configuration.

adam commented

2025-12-29 19:23:18 +01:00

@jeremystretch commented on GitHub (Jul 6, 2021):

@maximumG Yep, that sounds perfect!

@jeremystretch commented on GitHub (Jul 6, 2021): @maximumG Yep, that sounds perfect!

adam commented

2025-12-29 19:23:19 +01:00

@jeremystretch commented on GitHub (Jul 8, 2021):

FYI with #6713 now closed, NetBox has only a single default queue. Should simplify this work a bit.

@jeremystretch commented on GitHub (Jul 8, 2021): FYI with #6713 now closed, NetBox has only a single `default` queue. Should simplify this work a bit.

Sign in to join this conversation.

Branches Tags

main

21050-device-oob-ip-may-become-orphaned

21102-fix-graphiql-explorer

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#5024