starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Allow user to create their own DJANGO-RQ queues #3906

New Issue
Closed
opened 2025-12-29 18:31:54 +01:00 by adam · 11 comments
adam commented 2025-12-29 18:31:54 +01:00
Owner
Copy Link

Originally created by @dgarros on GitHub (Jul 27, 2020).

Environment

  • Python version: 3.7.7
  • NetBox version: 2.8.8

Proposed Functionality

Allow users to create their own DJANGO-RQ queues
Currently NetBox creates 2 queues by default: default and check_releases.
As more features and plugins are leveraging the RQ queues system to execute asynchronous tasks, it would be useful to be able to create more queues.

Use Case

For the onboarding plugin (or other tasks that need to communicate with the devices), it would be useful to have a queue per region to be able to run specific workers per region that are authorized to connect to the network devices (jumphost).

Database Changes

No change to the database.

External Dependencies

No external dependency.

Originally created by @dgarros on GitHub (Jul 27, 2020). ### Environment * Python version: 3.7.7 * NetBox version: 2.8.8 ### Proposed Functionality Allow users to create their own DJANGO-RQ queues Currently NetBox creates 2 queues by default: `default` and `check_releases`. As more features and plugins are leveraging the RQ queues system to execute asynchronous tasks, it would be useful to be able to create more queues. ### Use Case For the onboarding plugin (or other tasks that need to communicate with the devices), it would be useful to have a queue per region to be able to run specific workers per region that are authorized to connect to the network devices (jumphost). ### Database Changes No change to the database. ### External Dependencies No external dependency.
adam added the type: featurestatus: needs ownerpending closuretopic: plugins labels 2025-12-29 18:31:54 +01:00
adam closed this issue 2025-12-29 18:31:54 +01:00
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jul 28, 2020):

Allow users to create their own DJANGO-RQ queues

Could you expand on exactly what's being proposed? What is the workflow for users to create new queues?

@jeremystretch commented on GitHub (Jul 28, 2020): > Allow users to create their own DJANGO-RQ queues Could you expand on exactly what's being proposed? What is the workflow for users to create new queues?
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@dgarros commented on GitHub (Jul 29, 2020):

@jeremystretch My initial thought was to define new queues in the configuration.py file, in a similar way than what django-rq is doing.
If possible, I think a good solution would be to allow the user to define their own RQ_QUEUES, completely or partially. NetBox could add its own queues on top of what the user provides default, check_releases.

@dgarros commented on GitHub (Jul 29, 2020): @jeremystretch My initial thought was to define new queues in the `configuration.py` file, in a similar way than what [django-rq](https://github.com/rq/django-rq) is doing. If possible, I think a good solution would be to allow the user to define their own RQ_QUEUES, completely or partially. NetBox could add its own queues on top of what the user provides `default`, `check_releases`.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@cpmills1975 commented on GitHub (Jul 29, 2020):

Would it make more sense that if a plugin requires a queue for a specific purpose, it is able to be specified in the plugin's setup.py file or PluginConfig sub-class (sorry - I'm not sufficiently knowledgeable to understand the difference)? That way, the requirement to run a queue is kept contained in the plugin code itself and wouldn't cause a problem if a user imported a plugin and didn't modify their site wide configuration.py file.

@cpmills1975 commented on GitHub (Jul 29, 2020): Would it make more sense that if a plugin requires a queue for a specific purpose, it is able to be specified in the plugin's setup.py file or PluginConfig sub-class (sorry - I'm not sufficiently knowledgeable to understand the difference)? That way, the requirement to run a queue is kept contained in the plugin code itself and wouldn't cause a problem if a user imported a plugin and didn't modify their site wide configuration.py file.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@dgarros commented on GitHub (Jul 29, 2020):

@cpmills1975 in some cases it could make sense but not for all cases.
For example, if the goal is to have different queues to control on which jumphost/server a task is running, it won't be possible for the plugin creator to know in advance how many queues are required. Some users will need 1, other will need dozen or more.

@dgarros commented on GitHub (Jul 29, 2020): @cpmills1975 in some cases it could make sense but not for all cases. For example, if the goal is to have different queues to control on which jumphost/server a task is running, it won't be possible for the plugin creator to know in advance how many queues are required. Some users will need 1, other will need dozen or more.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@tb-killa commented on GitHub (Aug 6, 2020):

I agree with @cpmills1975: Plugins should be able to build and use own rq-queue if needing.

@tb-killa commented on GitHub (Aug 6, 2020): I agree with @cpmills1975: Plugins should be able to build and use own rq-queue if needing.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Sep 29, 2020):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@stale[bot] commented on GitHub (Sep 29, 2020): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@dgarros commented on GitHub (Sep 29, 2020):

Looks like different use cases could have different requirements for this feature.
The main use cases I have in mind is to allow the users to control where a given task will be executed.

For example for the onboarding plugin, It's likely that not all servers will be able to connect directly to the network devices so the worker needs to run on a Jumphost that can connect to the devices. If the Jumphosts are different per region (US/APAC/EMEA) and no jumphost can reach all network devices, it won't be possible to use the onboarding plugin currently because we can't control where a task is running.
From a plugin developer point of view, I would like to provide an option to define where a given onboarding task should run (onboard, 1.2.3.4 in APAC). In this case APAC would be the name of queue previously created by the user which would map to a group of workers running in the APAC jumphost.

In this scenario, the different queues needs to be statically defined by the users ahead of time, in the configuration file. The plugin (or netbox) will just provide an option to select what queue should be used for a given job.

@dgarros commented on GitHub (Sep 29, 2020): Looks like different use cases could have different requirements for this feature. The main use cases I have in mind is to allow the users to control where a given task will be executed. For example for the onboarding plugin, It's likely that not all servers will be able to connect directly to the network devices so the worker needs to run on a Jumphost that can connect to the devices. If the Jumphosts are different per region (US/APAC/EMEA) and no jumphost can reach all network devices, it won't be possible to use the onboarding plugin currently because we can't control where a task is running. From a plugin developer point of view, I would like to provide an option to define where a given onboarding task should run (onboard, 1.2.3.4 in APAC). In this case APAC would be the name of queue previously created by the user which would map to a group of workers running in the APAC jumphost. In this scenario, the different queues needs to be statically defined by the users ahead of time, in the configuration file. The plugin (or netbox) will just provide an option to select what queue should be used for a given job.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Sep 29, 2020):

I see the use case for plugins defining their queues statically, however extending that ability to the end user seems excessive. We would need to handle naming collisions, garbage collection, permissions, and related housekeeping tasks. Ensuring a deterministic set of queues per plugin is easier to manage.

As far as the use case cited, you should be able to use a single queue for all jobs and disseminate them to distributed worker processes based on geographic location. Additionally, if you decide you need the ability to create queues dynamically, it's certainly possible to do so by interfacing with Redis/django-rq directly.

@jeremystretch commented on GitHub (Sep 29, 2020): I see the use case for plugins defining their queues statically, however extending that ability to the end user seems excessive. We would need to handle naming collisions, garbage collection, permissions, and related housekeeping tasks. Ensuring a deterministic set of queues per plugin is easier to manage. As far as the use case cited, you should be able to use a single queue for all jobs and disseminate them to distributed worker processes based on geographic location. Additionally, if you decide you _need_ the ability to create queues dynamically, it's certainly possible to do so by interfacing with Redis/django-rq directly.
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Oct 14, 2020):

Tagging this as needs owner for whoever wants to take it on. Please be sure to include tests using extras.tests.dummy_plugin. Also note that because this effects a change to the plugins API, work will need to be done against the current develop-2.x branch (as opposed to develop).

@jeremystretch commented on GitHub (Oct 14, 2020): Tagging this as `needs owner` for whoever wants to take it on. Please be sure to include tests using `extras.tests.dummy_plugin`. Also note that because this effects a change to the plugins API, work will need to be done against the current `develop-2.x` branch (as opposed to `develop`).
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Nov 29, 2020):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@stale[bot] commented on GitHub (Nov 29, 2020): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 18:31:55 +01:00
Author
Owner
Copy Link

@stale[bot] commented on GitHub (Dec 15, 2020):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@stale[bot] commented on GitHub (Dec 15, 2020): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pending closure status: needs owner topic: plugins type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3906
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?