starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Run reports/scripts in the background #1654

New Issue
Closed
opened 2025-12-29 16:34:00 +01:00 by adam · 7 comments
adam commented 2025-12-29 16:34:00 +01:00
Owner
Copy Link

Originally created by @arionl on GitHub (Apr 4, 2018).

Originally assigned to: @lampwins on GitHub.

Issue type

[X] Feature request
[ ] Bug report
[ ] Documentation

Environment

  • Python version: 3.6.4
  • NetBox version: 2.3.2

Description

I have some reports that take a long time to run. They are usually reports that are comparing NetBox data with data in another system. For example, I'm using a custom field to track if a device or VM should be actively monitored and then I have a report that runs every day to see if the object exists or if it is missing from the monitoring platform. With thousands of objects, these reports take a while to run. They work fine when they run on the daily schedule or if I run them via manage.py but if I use the "Run Report" button within the UI, I usually get a timeout message from my nginx reverse proxy. I could increase the timeouts, but I think it would be more useful if a report could be marked as "Running" and then I could refresh the page later to check for results. Could this be managed as a DB flag or something?

Originally created by @arionl on GitHub (Apr 4, 2018). Originally assigned to: @lampwins on GitHub. <!-- Before opening a new issue, please search through the existing issues to see if your topic has already been addressed. Note that you may need to remove the "is:open" filter from the search bar to include closed issues. Check the appropriate type for your issue below by placing an x between the brackets. For assistance with installation issues, or for any other issues other than those listed below, please raise your topic for discussion on our mailing list: https://groups.google.com/forum/#!forum/netbox-discuss Please note that issues which do not fall under any of the below categories will be closed. Due to an excessive backlog of feature requests, we are not currently accepting any proposals which extend NetBox's feature scope. not prepend any sort of tag to your issue's title. An administrator will review your issue and assign labels as appropriate. ---> ### Issue type [X] Feature request <!-- An enhancement of existing functionality --> [ ] Bug report <!-- Unexpected or erroneous behavior --> [ ] Documentation <!-- A modification to the documentation --> <!-- Please describe the environment in which you are running NetBox. (Be sure to verify that you are running the latest stable release of NetBox before submitting a bug report.) If you are submitting a bug report and have made any changes to the code base, please first validate that your bug can be recreated while running an official release. --> ### Environment * Python version: 3.6.4 * NetBox version: 2.3.2 ### Description I have some reports that take a long time to run. They are usually reports that are comparing NetBox data with data in another system. For example, I'm using a custom field to track if a device or VM should be actively monitored and then I have a report that runs every day to see if the object exists or if it is missing from the monitoring platform. With thousands of objects, these reports take a while to run. They work fine when they run on the daily schedule or if I run them via `manage.py` but if I use the "Run Report" button within the UI, I usually get a timeout message from my nginx reverse proxy. I could increase the timeouts, but I think it would be more useful if a report could be marked as "Running" and then I could refresh the page later to check for results. Could this be managed as a DB flag or something?
adam added the status: acceptedtype: feature labels 2025-12-29 16:34:00 +01:00
adam closed this issue 2025-12-29 16:34:01 +01:00
adam commented 2025-12-29 16:34:01 +01:00
Author
Owner
Copy Link

@Grokzen commented on GitHub (Apr 10, 2018):

Would it not be better to have this type of task outsourced to a celery runner and run the report async to free up the call to the GUI and avoid having to increase the timeout?

@Grokzen commented on GitHub (Apr 10, 2018): Would it not be better to have this type of task outsourced to a celery runner and run the report async to free up the call to the GUI and avoid having to increase the timeout?
adam commented 2025-12-29 16:34:01 +01:00
Author
Owner
Copy Link

@arionl commented on GitHub (May 8, 2018):

I'm not sure of the proper way to tackle this from within Django. For me, I'm running all reports via a cron job that just does python3 manage.py runreport REPORTNAME every day.. but in this feature request I was just pointing out that in many cases, the GUI will be unresponsive or timeout when a report takes a long time to run and it would be nice to allow my users to run reports on demand if they really don't want to wait until the next day when a report is run via cron. Not a high priority request, but I think others might have a similar use case and there may be other tasks lurking within NetBox that could be offloaded like this..

@arionl commented on GitHub (May 8, 2018): I'm not sure of the proper way to tackle this from within Django. For me, I'm running all reports via a cron job that just does `python3 manage.py runreport REPORTNAME` every day.. but in this feature request I was just pointing out that in many cases, the GUI will be unresponsive or timeout when a report takes a long time to run and it would be nice to allow my users to run reports on demand if they really don't want to wait until the next day when a report is run via cron. Not a high priority request, but I think others might have a similar use case and there may be other tasks lurking within NetBox that could be offloaded like this..
adam commented 2025-12-29 16:34:02 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Apr 24, 2019):

Beginning with v2.6, NetBox will require a Redis database, so I think it makes sense to also require django-rq to enable asynchronous processing of reports. (django-rq is currently an optional component, required only to enable webhooks.)

@jeremystretch commented on GitHub (Apr 24, 2019): Beginning with v2.6, NetBox will require a Redis database, so I think it makes sense to also require `django-rq` to enable asynchronous processing of reports. (`django-rq` is currently an optional component, required only to enable webhooks.)
adam commented 2025-12-29 16:34:02 +01:00
Author
Owner
Copy Link

@tb-killa commented on GitHub (Aug 19, 2019):

Maybe this FR could be used by some adapted Fragments from https://github.com/netbox-community/netbox/issues/3415 ?

@tb-killa commented on GitHub (Aug 19, 2019): Maybe this FR could be used by some adapted Fragments from https://github.com/netbox-community/netbox/issues/3415 ?
adam commented 2025-12-29 16:34:02 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Apr 24, 2020):

We need to figure out the optimal workflow when running reports via the REST API. If the request is queued for background processing, we need to return some kind of confirmation to the client.

It also raises the question of whether we should immediately delete the result from the previous run, or wait until the new run has completed.

@jeremystretch commented on GitHub (Apr 24, 2020): We need to figure out the optimal workflow when running reports via the REST API. If the request is queued for background processing, we need to return some kind of confirmation to the client. It also raises the question of whether we should immediately delete the result from the previous run, or wait until the new run has completed.
adam commented 2025-12-29 16:34:02 +01:00
Author
Owner
Copy Link

@lampwins commented on GitHub (Jun 4, 2020):

My proposal for this is a new model JobResults to take the place of the current ReportResults model in extras. It will track mostly the name information but will be generic in that it will support report results, script results, and presumably anything else we come up with in the future. It could also conceivably be something we expose to plugins at some point down the road.

It would have these fields:

  • created_at - datetime
  • completed_at - datetime
  • user - FK to user
  • type - choice field (report, script, etc)
  • name - char (report/script name)
  • data - json
  • status - choice field (queued, running, completed, failed)
  • job_id - char - RQ job info

Basically, when a user runs a report either through the API or the UI, we create one of these objects as we enqueue the job and pass this object's info back to the user.

We would expose a new list and retrieve endpoints for the JobResults model, but we will also build in the intelligence on the existing report results and script endpoints to query based on the name and type fields to retrieve the status of the job. The user will be able to keep up with the results by watching as the status field changes over time and the data field is finally populated upon completion. This does open the door the better tracking of results historically, but it would not be necessary to do this day one, as a part of this functionality.

Scripts will work in mostly the same way, we would just need to convert the plumbing of script output to use the similar report results method.

We would modify the UI for both reports and scripts a little bit to dynamically query the API every few seconds until results are available.

@lampwins commented on GitHub (Jun 4, 2020): My proposal for this is a new model `JobResults` to take the place of the current `ReportResults` model in extras. It will track mostly the name information but will be generic in that it will support report results, script results, and presumably anything else we come up with in the future. It could also conceivably be something we expose to plugins at some point down the road. It would have these fields: - created_at - datetime - completed_at - datetime - user - FK to user - type - choice field (report, script, etc) - name - char (report/script name) - data - json - status - choice field (queued, running, completed, failed) - job_id - char - RQ job info Basically, when a user runs a report either through the API or the UI, we create one of these objects as we enqueue the job and pass this object's info back to the user. We would expose a new list and retrieve endpoints for the `JobResults` model, but we will also build in the intelligence on the existing report results and script endpoints to query based on the name and type fields to retrieve the status of the job. The user will be able to keep up with the results by watching as the status field changes over time and the data field is finally populated upon completion. This does open the door the better tracking of results historically, but it would not be necessary to do this day one, as a part of this functionality. Scripts will work in mostly the same way, we would just need to convert the plumbing of script output to use the similar report results method. We would modify the UI for both reports and scripts a little bit to dynamically query the API every few seconds until results are available.
adam commented 2025-12-29 16:34:02 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jun 9, 2020):

Sounds good to me. Super nit-picky point: We might want to name the model JobResult (singular) to avoid confusion when dealing with multiple instances.

Related: I've opened #4735 to propose merging report functionality into custom scripts. The current assumption is that this work (#2006) will occur first, but I'm open to either approach.

@jeremystretch commented on GitHub (Jun 9, 2020): Sounds good to me. Super nit-picky point: We might want to name the model `JobResult` (singular) to avoid confusion when dealing with multiple instances. Related: I've opened #4735 to propose merging report functionality into custom scripts. The current assumption is that this work (#2006) will occur first, but I'm open to either approach.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#1654
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?