starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Improve performance of object type custom field lookups for API responses #8200

New Issue
Closed
opened 2025-12-29 20:33:44 +01:00 by adam · 2 comments
adam commented 2025-12-29 20:33:44 +01:00
Owner
Copy Link

Originally created by @kkthxbye-code on GitHub (Jun 16, 2023).

NetBox version

v3.5.3

Feature type

Data model extension

Proposed functionality

Currently when listing objects (let's use device as an example) with custom fields of type Object or Multiple Objects, if the field is populated a query is made for every single custom field for every single object type custom field.

The lookup is done here:
cdce500d90/netbox/extras/models/customfields.py (L348-L353)

The reason we do this, is because object type custom fields are implemented like GenericForeignKeys, with the difference being that the type and id of the custom field is stored in the JSONB custom_field_data column.

This is very ineffective and slows API responses to a crawl for larger pagination sizes or with multiple custom fields.

As an example imagine the device model having two custom fields, cf_test1 and cf_test2 both being object type custom fields, both pointing towards the Interface model. Fetching 100 devices in the API results in 422 queries in my test instance. 200 of the queries are for getting the interface matching the ID, and 200 further queries are for getting the device of the interface as the interface serializer contains the device.

We had a quick talk about this on slack, and @jeremystretch suggested that it might be possible to replicate the prefetch logic that exists for generic foreign keys. I don't have enough in-depth knowledge to suggest if this is a viable approach.

An alternative workaround to the immediate problem would be to allow excluding the custom_field field in the API, just as we allow excluding config_context also for performance issues. An even nicer solution would be to allow the user to define exactly which fields are included in the response, however I'm not sure how feasible this is.

Another solution yet again would be to allow the user to disable the deserialization of custom fields in the API response.

Use case

We currently have 6 object type custom fields on the device model. We run a script from another system to get the name and status of all devices of a specific type (1000+ devices). Getting a single page in the API of 1000 devices takes upwards of 90 seconds. The same request in a test environment with the same data, but sans custom fields takes seconds. This is not optimal when the only two fields needed is name and status.

Database changes

Probably not, but might depend on solution chosen.

External dependencies

None

Originally created by @kkthxbye-code on GitHub (Jun 16, 2023). ### NetBox version v3.5.3 ### Feature type Data model extension ### Proposed functionality Currently when listing objects (let's use device as an example) with custom fields of type `Object` or `Multiple Objects`, if the field is populated a query is made for every single custom field for every single object type custom field. The lookup is done here: https://github.com/netbox-community/netbox/blob/cdce500d909ffc91d24107fa60191da9d00828ec/netbox/extras/models/customfields.py#L348-L353 The reason we do this, is because object type custom fields are implemented like GenericForeignKeys, with the difference being that the type and id of the custom field is stored in the JSONB `custom_field_data` column. This is very ineffective and slows API responses to a crawl for larger pagination sizes or with multiple custom fields. As an example imagine the device model having two custom fields, cf_test1 and cf_test2 both being object type custom fields, both pointing towards the Interface model. Fetching 100 devices in the API results in 422 queries in my test instance. 200 of the queries are for getting the interface matching the ID, and 200 further queries are for getting the device of the interface as the interface serializer contains the device. We had a quick talk about this on slack, and @jeremystretch suggested that it might be possible to replicate the prefetch logic that exists for generic foreign keys. I don't have enough in-depth knowledge to suggest if this is a viable approach. An alternative workaround to the immediate problem would be to allow excluding the custom_field field in the API, just as we allow excluding config_context also for performance issues. An even nicer solution would be to allow the user to define exactly which fields are included in the response, however I'm not sure how feasible this is. Another solution yet again would be to allow the user to disable the deserialization of custom fields in the API response. ### Use case We currently have 6 object type custom fields on the device model. We run a script from another system to get the name and status of all devices of a specific type (1000+ devices). Getting a single page in the API of 1000 devices takes upwards of 90 seconds. The same request in a test environment with the same data, but sans custom fields takes seconds. This is not optimal when the only two fields needed is name and status. ### Database changes Probably not, but might depend on solution chosen. ### External dependencies None
adam added the type: featurepending closure labels 2025-12-29 20:33:44 +01:00
adam closed this issue 2025-12-29 20:33:45 +01:00
adam commented 2025-12-29 20:33:45 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Sep 15, 2023):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Sep 15, 2023): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 20:33:45 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Oct 15, 2023):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@github-actions[bot] commented on GitHub (Oct 15, 2023): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pending closure type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8200
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?