starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Improve performance when rendering the prefix hierarchy #4721

New Issue
Closed
opened 2025-12-29 19:19:51 +01:00 by adam · 7 comments
adam commented 2025-12-29 19:19:51 +01:00
Owner
Copy Link

Originally created by @jeremystretch on GitHub (Apr 4, 2021).

Originally assigned to: @jeremystretch on GitHub.

NetBox version

v2.10.8

Feature type

Change to existing functionality

Proposed functionality

Employ MPTT for constructing the prefix hierarchy. This entails automatically assigning each prefix to its parent (if any) and recalculating the hierarchy. NetBox currently employs MPTT for several models (e.g. nesting regions) but adopting it for prefixes will require the automatic assignment and re-assignment of parents as prefixes are added and removed.

Use case

MPTT allows us to more efficiently model the prefix hierarchy, and obviates the need to make verbose SQL annotations to count parent and child prefixes. However, it further consideration is needed regarding the automatic assignment of parent prefixes.

Database changes

Extend ipam.Prefix to an MPTT-enabled model by adding the parent_id, tree_id, level, lft, and rght fields.

External dependencies

No response

Originally created by @jeremystretch on GitHub (Apr 4, 2021). Originally assigned to: @jeremystretch on GitHub. ### NetBox version v2.10.8 ### Feature type Change to existing functionality ### Proposed functionality Employ MPTT for constructing the prefix hierarchy. This entails automatically assigning each prefix to its parent (if any) and recalculating the hierarchy. NetBox currently employs MPTT for several models (e.g. nesting regions) but adopting it for prefixes will require the automatic assignment and re-assignment of parents as prefixes are added and removed. ### Use case MPTT allows us to more efficiently model the prefix hierarchy, and obviates the need to make verbose SQL annotations to count parent and child prefixes. However, it further consideration is needed regarding the automatic assignment of parent prefixes. ### Database changes Extend ipam.Prefix to an MPTT-enabled model by adding the `parent_id`, `tree_id`, `level`, `lft`, and `rght` fields. ### External dependencies _No response_
adam added the status: acceptedtype: feature labels 2025-12-29 19:19:51 +01:00
adam closed this issue 2025-12-29 19:19:52 +01:00
adam commented 2025-12-29 19:19:52 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Apr 6, 2021):

What would be the performance impacts, good or bad, (in terms of response-time to load prefix pages) of doing this?

@tyler-8 commented on GitHub (Apr 6, 2021): What would be the performance impacts, good or bad, (in terms of response-time to load prefix pages) of doing this?
adam commented 2025-12-29 19:19:52 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Apr 6, 2021):

@tyler-8 there would be some degree of performance impact at write time, but it should greatly reduce the amount of time required to render the prefixes list. This proposal is still just a rough idea: further testing is needed to determine its viability.

@jeremystretch commented on GitHub (Apr 6, 2021): @tyler-8 there would be some degree of performance impact at write time, but it should greatly reduce the amount of time required to render the prefixes list. This proposal is still just a rough idea: further testing is needed to determine its viability.
adam commented 2025-12-29 19:19:53 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Apr 6, 2021):

What implications are there for the REST API, if any? I presume new fields as stated in the OP, but if users have code around the existing fields, would their work be impacted?

@tyler-8 commented on GitHub (Apr 6, 2021): What implications are there for the REST API, if any? I presume new fields as stated in the OP, but if users have code around the existing fields, would their work be impacted?
adam commented 2025-12-29 19:19:53 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 27, 2021):

Status update: I ended up forgoing MPTT or any similar solution, and opted to simply cache the depth and child count of each prefix. It's not a super-elegant solution, but it works well and seems pretty robust. Load times for the full prefix hierarchy have been dramatically reduced. Plus, we can filter by depth and child count now!

netbox_depth_filter

I still want to do some more testing, especially around the migration, but I expect this to debut in v2.11.5.

@jeremystretch commented on GitHub (May 27, 2021): Status update: I ended up forgoing MPTT or any similar solution, and opted to simply cache the depth and child count of each prefix. It's not a super-elegant solution, but it works well and seems pretty robust. Load times for the full prefix hierarchy have been _dramatically_ reduced. Plus, we can filter by depth and child count now! ![netbox_depth_filter](https://user-images.githubusercontent.com/13487278/119874278-6e317f00-bef3-11eb-9058-d87a1dd4797d.gif) I still want to do some more testing, especially around the migration, but I expect this to debut in v2.11.5.
adam commented 2025-12-29 19:19:53 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 27, 2021):

I did a quick benchmark of the #6087 branch against develop using the following parameters:

  • 1,049,660 prefixes defined (1,092 of which are IPv6)
  • Showing 100 per page
  • Utilization column hidden
  • DEBUG off

I made five requests for /ipam/prefixes/?per_page=100 and recorded the total load time reported by the browser. Times are below.

v2.11.4 #6087
21.92s 0.92s
21.52s 0.90s
21.65s 0.93s
21.51s 0.90s
21.58s 0.93s

Clearly, some pretty huge gains in performance.

It's worth reiterating that I've hidden the utilization column from the prefixes tables for these tests, since that imposes additional overhead. (All other default columns were retained.) Still, with the utilization column included (though no child IP addresses defined), load times averaged around five seconds with #6087 in place.

Migrating the roughly one million prefixes took about eight minutes on my development machine. A more realistic deployment with perhaps around 100 thousand prefixes should take less than a minute.

$ time ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization
Running migrations:
  Applying ipam.0047_prefix_depth_children... OK
  Applying ipam.0048_prefix_populate_depth_children...
Updating 1049660 prefixes...
 OK

real	8m9.779s
user	7m17.770s
sys	0m0.679s

Obviously this is far from a scientific test, though I believe it illustrates the enormity of the performance improvements gained with this work.

@jeremystretch commented on GitHub (May 27, 2021): I did a quick benchmark of the #6087 branch against `develop` using the following parameters: * 1,049,660 prefixes defined (1,092 of which are IPv6) * Showing 100 per page * Utilization column hidden * `DEBUG` off I made five requests for `/ipam/prefixes/?per_page=100` and recorded the total load time reported by the browser. Times are below. | v2.11.4 | #6087 | |---------|-------| | 21.92s | 0.92s | | 21.52s | 0.90s | | 21.65s | 0.93s | | 21.51s | 0.90s | | 21.58s | 0.93s | Clearly, some pretty huge gains in performance. It's worth reiterating that I've hidden the utilization column from the prefixes tables for these tests, since that imposes additional overhead. (All other default columns were retained.) Still, with the utilization column included (though no child IP addresses defined), load times averaged around five seconds with #6087 in place. Migrating the roughly one million prefixes took about eight minutes on my development machine. A more realistic deployment with perhaps around 100 thousand prefixes should take less than a minute. ``` $ time ./manage.py migrate Operations to perform: Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization Running migrations: Applying ipam.0047_prefix_depth_children... OK Applying ipam.0048_prefix_populate_depth_children... Updating 1049660 prefixes... OK real 8m9.779s user 7m17.770s sys 0m0.679s ``` Obviously this is far from a scientific test, though I believe it illustrates the enormity of the performance improvements gained with this work.
adam commented 2025-12-29 19:19:53 +01:00
Author
Owner
Copy Link

@sdktr commented on GitHub (May 27, 2021):

Nice work @jeremystretch ! Will the children be available in the API response as well? Or the other way around: will the parent prefix reference be exposed in the API?

https://demo.netbox.dev/api/ipam/prefixes/?parent=123 ?

https://demo.netbox.dev/api/ipam/prefixes/456/?include_children=true ?

@sdktr commented on GitHub (May 27, 2021): Nice work @jeremystretch ! Will the children be available in the API response as well? Or the other way around: will the parent prefix reference be exposed in the API? https://demo.netbox.dev/api/ipam/prefixes/?parent=123 ? https://demo.netbox.dev/api/ipam/prefixes/456/?include_children=true ?
adam commented 2025-12-29 19:19:53 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 27, 2021):

@sdktr this doesn't enforce any explicit relationship between parent and child prefixes. This would actually be infeasible given that we (sometimes) allow duplicate prefixes to coexist.

We could extend the REST API serializers to annotate the depth and child count for each prefix with negligible performance impact, but that would be the extent of the changes (beyond the new filtering ability introduced).

@jeremystretch commented on GitHub (May 27, 2021): @sdktr this doesn't enforce any explicit relationship between parent and child prefixes. This would actually be infeasible given that we (sometimes) allow duplicate prefixes to coexist. We could extend the REST API serializers to annotate the depth and child count for each prefix with negligible performance impact, but that would be the extent of the changes (beyond the new filtering ability introduced).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#4721
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?