starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-14 14:23:32 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

[PR #11943] [MERGED] 11291 optimize GraphQL queries #13872

New Issue
Closed
opened 2025-12-29 23:21:17 +01:00 by adam · 0 comments
adam commented 2025-12-29 23:21:17 +01:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/netbox-community/netbox/pull/11943
Author: @arthanson
Created: 3/9/2023
Status: ✅ Merged
Merged: 3/23/2023
Merged by: @jeremystretch

Base: feature ← Head: 11291-graphql-slow-queries-2

📝 Commits (4)

  • 6b9d576 11291 initial optimize graphql queries
  • 103e69d 11291 add optimizer to schemas
  • 49ac5b6 11291 cleanup fields.py
  • ee61e2a 11291 fix fragment query

📊 Changes

11 files changed (+547 additions, -3 deletions)

View changed files

📝 netbox/circuits/graphql/schema.py (+17 -0)
📝 netbox/core/graphql/schema.py (+8 -0)
📝 netbox/dcim/graphql/schema.py (+122 -0)
📝 netbox/extras/graphql/schema.py (+32 -0)
📝 netbox/ipam/graphql/schema.py (+57 -0)
📝 netbox/netbox/graphql/fields.py (+6 -3)
📝 netbox/tenancy/graphql/schema.py (+20 -0)
📝 netbox/users/graphql/schema.py (+8 -0)
➕ netbox/utilities/graphql_optimizer.py (+249 -0)
📝 netbox/virtualization/graphql/schema.py (+17 -0)
📝 netbox/wireless/graphql/schema.py (+11 -0)

📄 Description

Fixes: #11291

This optimizes the GraphQL queries by automatically putting in prefetch_related to queried fields. Results:

I made a bunch of VLANs and a query that shows the query issue resulting in a 10x or 20x reduction of queries, here are the query counts:

  • old code (before filtering at all levels): 985 queries
  • current code (with filtering at all levels): 1852 queries
  • with query optimizer: 84 queries

Tested with sample queries for nested types, nested filters, fragments (triple dot notation) and they all worked successfully.

Couple Notes on implementation:

There is some interaction between graphene / FilterSets and the optimizer. The optimizer works fine with graphene but without FilterSets and works fine outside of graphene with FilterSets, but combining Graphene, FilterSets and the optimizer produces extra queries. So netbox/netbox/graphql/fields.py if there are no filter params then it doesn't instantiate the FilterSet.

Propose this as a temporary solution as it results in 10x less queries in the non-filtered case and figuring out how to fix the interaction between graphql / filterset will be a big time sink I think so can create a separate issue for that.

There is a separate issue where this code: https://github.com/netbox-community/netbox/blob/develop/netbox/netbox/filtersets.py#L249 is adding in extra queries for custom-fields even if they are not referenced in the query-set. This is what causes the number of queries to double with the filtering at all levels feature that was added in NetBox 3.4. It isn't a bug in that feature, rather an existing bug that was just exposed by this change as it now applies filters at all levels in the schema. If the custom fields issue is fixed the querycount comes back down to 985 with filtering.

In the schmea files the lines:

    def resolve_vrf_list(root, info, **kwargs):
        return gql_query_optimizer(models.VRF.objects.all(), info)

were needed as unfortunately the optimizer code could not be put into fields.py which generically handles getting the schema class and filterset. The reason for this is two fold:

  • The optimizer has to be inserted in code in the parent list_resolver initializer after it queries the class but before it passes it down into Django.
  • second: fields.py list_resolver gets called for all lists, but you only want the optimizer called for the root resolver so it needs to go into the resolver functions in the schema files.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbox-community/netbox/pull/11943 **Author:** [@arthanson](https://github.com/arthanson) **Created:** 3/9/2023 **Status:** ✅ Merged **Merged:** 3/23/2023 **Merged by:** [@jeremystretch](https://github.com/jeremystretch) **Base:** `feature` ← **Head:** `11291-graphql-slow-queries-2` --- ### 📝 Commits (4) - [`6b9d576`](https://github.com/netbox-community/netbox/commit/6b9d576f146de5626b0870accffd9c3cf3a59b1b) 11291 initial optimize graphql queries - [`103e69d`](https://github.com/netbox-community/netbox/commit/103e69d9e8ccdc810ae155ada69c72c38598685a) 11291 add optimizer to schemas - [`49ac5b6`](https://github.com/netbox-community/netbox/commit/49ac5b6f28e26c704975d82e07b8f4cbbf6249da) 11291 cleanup fields.py - [`ee61e2a`](https://github.com/netbox-community/netbox/commit/ee61e2a5bece3674eec96c1491f9f63248f6d424) 11291 fix fragment query ### 📊 Changes **11 files changed** (+547 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `netbox/circuits/graphql/schema.py` (+17 -0) 📝 `netbox/core/graphql/schema.py` (+8 -0) 📝 `netbox/dcim/graphql/schema.py` (+122 -0) 📝 `netbox/extras/graphql/schema.py` (+32 -0) 📝 `netbox/ipam/graphql/schema.py` (+57 -0) 📝 `netbox/netbox/graphql/fields.py` (+6 -3) 📝 `netbox/tenancy/graphql/schema.py` (+20 -0) 📝 `netbox/users/graphql/schema.py` (+8 -0) ➕ `netbox/utilities/graphql_optimizer.py` (+249 -0) 📝 `netbox/virtualization/graphql/schema.py` (+17 -0) 📝 `netbox/wireless/graphql/schema.py` (+11 -0) </details> ### 📄 Description <!-- Thank you for your interest in contributing to NetBox! Please note that our contribution policy requires that a feature request or bug report be approved and assigned prior to opening a pull request. This helps avoid waste time and effort on a proposed change that we might not be able to accept. IF YOUR PULL REQUEST DOES NOT REFERENCE AN ISSUE WHICH HAS BEEN ASSIGNED TO YOU, IT WILL BE CLOSED AUTOMATICALLY. Please specify your assigned issue number on the line below. --> ### Fixes: #11291 <!-- Please include a summary of the proposed changes below. --> This optimizes the GraphQL queries by automatically putting in prefetch_related to queried fields. Results: I made a bunch of VLANs and a query that shows the query issue resulting in a 10x or 20x reduction of queries, here are the query counts: - old code (before filtering at all levels): 985 queries - current code (with filtering at all levels): 1852 queries - with query optimizer: 84 queries Tested with sample queries for nested types, nested filters, fragments (triple dot notation) and they all worked successfully. **Couple Notes on implementation:** There is some interaction between graphene / FilterSets and the optimizer. The optimizer works fine with graphene but without FilterSets and works fine outside of graphene with FilterSets, but combining Graphene, FilterSets and the optimizer produces extra queries. So [netbox/netbox/graphql/fields.py](https://github.com/netbox-community/netbox/pull/11943/files#diff-453a32882215ad6bdb3c106adeeff83bcf4d2667375fe4130a8643f10be5e872) if there are no filter params then it doesn't instantiate the FilterSet. Propose this as a temporary solution as it results in 10x less queries in the non-filtered case and figuring out how to fix the interaction between graphql / filterset will be a big time sink I think so can create a separate issue for that. There is a separate issue where this code: https://github.com/netbox-community/netbox/blob/develop/netbox/netbox/filtersets.py#L249 is adding in extra queries for custom-fields even if they are not referenced in the query-set. This is what causes the number of queries to double with the filtering at all levels feature that was added in NetBox 3.4. It isn't a bug in that feature, rather an existing bug that was just exposed by this change as it now applies filters at all levels in the schema. If the custom fields issue is fixed the querycount comes back down to 985 with filtering. In the schmea files the lines: ``` def resolve_vrf_list(root, info, **kwargs): return gql_query_optimizer(models.VRF.objects.all(), info) ``` were needed as unfortunately the optimizer code could not be put into fields.py which generically handles getting the schema class and filterset. The reason for this is two fold: - The optimizer has to be inserted in code in the parent list_resolver initializer after it queries the class but before it passes it down into Django. - second: fields.py list_resolver gets called for all lists, but you only want the optimizer called for the root resolver so it needs to go into the resolver functions in the schema files. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
adam added the pull-request label 2025-12-29 23:21:17 +01:00
adam closed this issue 2025-12-29 23:21:17 +01:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#13872
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?