starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Pre-generate the OpenAPI schema #4910

New Issue
Closed
opened 2025-12-29 19:22:10 +01:00 by adam · 10 comments
adam commented 2025-12-29 19:22:10 +01:00
Owner
Copy Link

Originally created by @candlerb on GitHub (May 15, 2021).

Originally assigned to: @jeremystretch on GitHub.

NetBox version

v2.11.3

Feature type

Change to existing functionality

Proposed functionality

A feature like "collectstatic" which builds the openapi schema once in ./upgrade.sh, then serves it as a static file.

Currently it's served dynamically and is very slow, taking consistently 7-8 seconds on my system:

# time curl -sS localhost:8001/api/docs/?format=openapi | wc
      0   78785  799797

real	0m7.805s
user	0m0.016s
sys	0m0.005s

Alternatively, cache it in redis (it will take nearly a megabyte of RAM, but worth it)

Use case

ansible netbox inventory

The ansible netbox inventory fetches the whole openapi schema each time it runs. When combined with the actual data queries it does for devices, vms etc, this gives around a 10-second lead time to any playbook which is hitting the netbox API.

Whilst this inventory module has a feature to cache the results of the inventory (if you configure a suitable cache plugin), I want to get the "real-time" up-to-the-minute data from Netbox. The plugin can't be configured to cache the openapi schema only, whilst still making data queries to Netbox.

pynetbox

Any code which uses pynetbox, and calls nb.openapi(), will also benefit.

import pynetbox
nb = pynetbox.api(API_URL, token=API_TOKEN)
nb.openapi()

Database changes

None

External dependencies

None

Originally created by @candlerb on GitHub (May 15, 2021). Originally assigned to: @jeremystretch on GitHub. ### NetBox version v2.11.3 ### Feature type Change to existing functionality ### Proposed functionality A feature like "collectstatic" which builds the openapi schema once in `./upgrade.sh`, then serves it as a static file. Currently it's served dynamically and is very slow, taking consistently 7-8 seconds on my system: ``` # time curl -sS localhost:8001/api/docs/?format=openapi | wc 0 78785 799797 real 0m7.805s user 0m0.016s sys 0m0.005s ``` Alternatively, cache it in redis (it will take nearly a megabyte of RAM, but worth it) ### Use case #### ansible netbox inventory The [ansible netbox inventory](https://docs.ansible.com/ansible/latest/collections/netbox/netbox/nb_inventory_inventory.html) fetches the whole openapi schema each time it runs. When combined with the actual data queries it does for devices, vms etc, this gives around a 10-second lead time to any playbook which is hitting the netbox API. Whilst this inventory module has a feature to cache the *results* of the inventory (if you configure a suitable cache plugin), I want to get the "real-time" up-to-the-minute data from Netbox. The plugin can't be configured to cache the openapi schema only, whilst still making data queries to Netbox. #### pynetbox Any code which uses pynetbox, and calls `nb.openapi()`, will also benefit. ``` import pynetbox nb = pynetbox.api(API_URL, token=API_TOKEN) nb.openapi() ``` ### Database changes None ### External dependencies None
adam added the status: acceptedtype: feature labels 2025-12-29 19:22:10 +01:00
adam closed this issue 2025-12-29 19:22:10 +01:00
adam commented 2025-12-29 19:22:10 +01:00
Author
Owner
Copy Link

@mraerino commented on GitHub (May 16, 2021):

This really got worse on 2.11 - my webserver even times out while generating the spec.

Is there a console command you can run to get the openapi json?

@mraerino commented on GitHub (May 16, 2021): This really got worse on 2.11 - my webserver even times out while generating the spec. Is there a console command you can run to get the openapi json?
adam commented 2025-12-29 19:22:10 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 17, 2021):

This really got worse on 2.11

I haven't noticed any difference from v2.10, and I can't recall any changes we've made that would have a significant impact on it.

@jeremystretch commented on GitHub (May 17, 2021): > This really got worse on 2.11 I haven't noticed any difference from v2.10, and I can't recall any changes we've made that would have a significant impact on it.
adam commented 2025-12-29 19:22:10 +01:00
Author
Owner
Copy Link

@sdktr commented on GitHub (Jun 2, 2021):

While serving a static version of the spec seems harmless since it's version bound anyway, I'm not sure that it makes sense to request the complete schema each time the plugin starts? No matter how fast netbox spits out the file, is that a roundtrip we can avoid?

@sdktr commented on GitHub (Jun 2, 2021): While serving a static version of the spec seems harmless since it's version bound anyway, I'm not sure that it makes sense to request the complete schema each time the plugin starts? No matter how fast netbox spits out the file, is that a roundtrip we can avoid?
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Jun 2, 2021):

There are two ways to bite the cherry.

  1. At the server end: pre-generate the spec and serve it statically
  2. At the client end: read the spec once and cache it

Whilst both are possible, (1) gives the maximum benefit:

  • It only has to be implemented once
  • It benefits all clients, even dumb ones
  • Clients don't have to worry about running with a stale copy of the spec when Netbox changes
  • TBH, transferring 1MB over the network is cheap these days

Ideally, Netbox would pregenerate the spec and give an ETag - then smart clients can check and don't have to refetch unless it has changed.

@candlerb commented on GitHub (Jun 2, 2021): There are two ways to bite the cherry. 1. At the server end: pre-generate the spec and serve it statically 2. At the client end: read the spec once and cache it Whilst both are possible, (1) gives the maximum benefit: * It only has to be implemented once * It benefits all clients, even dumb ones * Clients don't have to worry about running with a stale copy of the spec when Netbox changes * TBH, transferring 1MB over the network is cheap these days Ideally, Netbox would pregenerate the spec *and* give an ETag - then smart clients can check and don't have to refetch unless it has changed.
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Sep 11, 2021):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Sep 11, 2021): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Oct 1, 2021):

I don't think we can pre-generate the full API spec as part of the release process, given that it contains some values unique to the installation (i.e. the base API URL). We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if BASE_PATH is changed).

What we can do is specify a cache timeout for the rendered page. This isn't perfect, but it should at least greatly reduce load times for bringing up the API docs. I'm going to try a 24-hour timeout for the rendered specs.

@jeremystretch commented on GitHub (Oct 1, 2021): I don't think we can pre-generate the full API spec as part of the release process, given that it contains some values unique to the installation (i.e. the base API URL). We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if `BASE_PATH` is changed). What we _can_ do is specify a cache timeout for the rendered page. This isn't perfect, but it should at least greatly reduce load times for bringing up the API docs. I'm going to try a 24-hour timeout for the rendered specs.
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Oct 1, 2021):

We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if BASE_PATH is changed).

I think it would be reasonable to require the user to re-run some command like manage.py genapi if the BASE_PATH is changed (but to have it run automatically on ./update.sh, like collectstatic is)

@candlerb commented on GitHub (Oct 1, 2021): > We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if `BASE_PATH` is changed). I think it would be reasonable to require the user to re-run some command like `manage.py genapi` if the `BASE_PATH` is changed (but to have it run automatically on `./update.sh`, like `collectstatic` is)
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Oct 1, 2021):

It would be very difficult to convey that to the user and ensure that it's undertaken consistently. IMO it's not worth the additional overhead for such a minor impact.

@jeremystretch commented on GitHub (Oct 1, 2021): It would be very difficult to convey that to the user and ensure that it's undertaken consistently. IMO it's not worth the additional overhead for such a minor impact.
adam commented 2025-12-29 19:22:11 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Oct 1, 2021):

IMO this is a reasonable optimization that avoids introducing any additional overhead. Just wanted to make some progress on this as no one else has volunteered to own this.

I'm happy to consider further improvements, though it's worth noting that we're somewhat limited by the capabilities of the drf-yasg package. Any suggestions for improvements to the actual spec rendering should be made upstream.

@jeremystretch commented on GitHub (Oct 1, 2021): IMO this is a reasonable optimization that avoids introducing any additional overhead. Just wanted to make _some_ progress on this as no one else has volunteered to own this. I'm happy to consider further improvements, though it's worth noting that we're somewhat limited by the capabilities of the `drf-yasg` package. Any suggestions for improvements to the actual spec rendering should be made upstream.
adam commented 2025-12-29 19:22:12 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Oct 1, 2021):

Given that the solution turns out to be just a case of adding cache_timeout=86400, that seems like a very reasonable cost/benefit tradeoff :-)

@candlerb commented on GitHub (Oct 1, 2021): Given that the solution turns out to be just a case of adding `cache_timeout=86400`, that seems like a very reasonable cost/benefit tradeoff :-)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#4910
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?