starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Performance of the Interface API should be improved #8031

New Issue
Closed
opened 2025-12-29 20:31:30 +01:00 by adam · 4 comments
adam commented 2025-12-29 20:31:30 +01:00
Owner
Copy Link

Originally created by @jiuka on GitHub (May 10, 2023).

NetBox version

v3.5.1

Python version

3.8

Steps to Reproduce

  1. Request many interfaces (https://demo.netbox.dev/api/dcim/interfaces/?limit=1000)

This has surfaced when using the Ansible invnetory plugin netbox.netbox.nb_inventory with the interfaces option set to return inteface informations.

Expected Behavior

The request resturns reasonably quickly

Observed Behavior

We see a 7-9s request time per 1000 interfaces.

Originally created by @jiuka on GitHub (May 10, 2023). ### NetBox version v3.5.1 ### Python version 3.8 ### Steps to Reproduce 1. Request many interfaces (https://demo.netbox.dev/api/dcim/interfaces/?limit=1000) This has surfaced when using the Ansible invnetory plugin [netbox.netbox.nb_inventory](https://github.com/netbox-community/ansible_modules/blob/devel/plugins/inventory/nb_inventory.py) with the `interfaces` option set to return inteface informations. ### Expected Behavior The request resturns reasonably quickly ### Observed Behavior We see a 7-9s request time per 1000 interfaces.
adam closed this issue 2025-12-29 20:31:30 +01:00
adam commented 2025-12-29 20:31:30 +01:00
Author
Owner
Copy Link

@abhi1693 commented on GitHub (May 10, 2023):

I dont think this can be considered a bug because pulling 1000 objects for any model will always be time consuming. You should limit the number of objects so that the API serializer can return much faster.

Here is a single object for Interface

{
            "id": 1,
            "url": "https://demo.netbox.dev/api/dcim/interfaces/1/",
            "display": "GigabitEthernet0/0/0",
            "device": {
                "id": 1,
                "url": "https://demo.netbox.dev/api/dcim/devices/1/",
                "display": "dmi01-akron-rtr01",
                "name": "dmi01-akron-rtr01"
            },
            "vdcs": [],
            "module": null,
            "name": "GigabitEthernet0/0/0",
            "label": "",
            "type": {
                "value": "1000base-x-sfp",
                "label": "SFP (1GE)"
            },
            "enabled": true,
            "parent": null,
            "bridge": null,
            "lag": null,
            "mtu": null,
            "mac_address": null,
            "speed": null,
            "duplex": null,
            "wwn": null,
            "mgmt_only": false,
            "description": "",
            "mode": null,
            "rf_role": null,
            "rf_channel": null,
            "poe_mode": null,
            "poe_type": null,
            "rf_channel_frequency": null,
            "rf_channel_width": null,
            "tx_power": null,
            "untagged_vlan": null,
            "tagged_vlans": [],
            "mark_connected": false,
            "cable": {
                "id": 7,
                "url": "https://demo.netbox.dev/api/dcim/cables/7/",
                "display": "#7",
                "label": ""
            },
            "cable_end": "B",
            "wireless_link": null,
            "link_peers": [
                {
                    "id": 1,
                    "url": "https://demo.netbox.dev/api/circuits/circuit-terminations/1/",
                    "display": "KKDG4923: Termination Z",
                    "circuit": {
                        "id": 1,
                        "url": "https://demo.netbox.dev/api/circuits/circuits/1/",
                        "display": "KKDG4923",
                        "cid": "KKDG4923"
                    },
                    "term_side": "Z",
                    "cable": 7,
                    "_occupied": true
                }
            ],
            "link_peers_type": "circuits.circuittermination",
            "wireless_lans": [],
            "vrf": null,
            "l2vpn_termination": null,
            "connected_endpoints": [
                {
                    "id": 1,
                    "url": "https://demo.netbox.dev/api/circuits/provider-networks/1/",
                    "display": "Level3 MPLS",
                    "name": "Level3 MPLS"
                }
            ],
            "connected_endpoints_type": "circuits.providernetwork",
            "connected_endpoints_reachable": true,
            "tags": [],
            "custom_fields": {},
            "created": "2021-04-14T00:00:00Z",
            "last_updated": "2021-04-14T17:36:01.841000Z",
            "count_ipaddresses": 0,
            "count_fhrp_groups": 0,
            "_occupied": true
        },

If I break this down a bit, the following are nested serializers which may be doing addiotional queries

  • device (object)
  • vdcs (list of objects)
  • parent (object)
  • bridge (object)
  • lag (object)
  • tagged vlans (list of objects)
  • cable ( object)
  • link peers (list of objects)
  • wireless lans (list of objects)
  • connected endpoints (list of objects)
  • tags (list of objects)
  • custom fields (object)

I'm sure I have missed a few fields in the list but if you understand the scope of how much data is being serialized here and multiply it by 1000 times. Considering this the API runs extrememly fast IMO.

@abhi1693 commented on GitHub (May 10, 2023): I dont think this can be considered a bug because pulling 1000 objects for any model will always be time consuming. You should limit the number of objects so that the API serializer can return much faster. Here is a single object for Interface ``` { "id": 1, "url": "https://demo.netbox.dev/api/dcim/interfaces/1/", "display": "GigabitEthernet0/0/0", "device": { "id": 1, "url": "https://demo.netbox.dev/api/dcim/devices/1/", "display": "dmi01-akron-rtr01", "name": "dmi01-akron-rtr01" }, "vdcs": [], "module": null, "name": "GigabitEthernet0/0/0", "label": "", "type": { "value": "1000base-x-sfp", "label": "SFP (1GE)" }, "enabled": true, "parent": null, "bridge": null, "lag": null, "mtu": null, "mac_address": null, "speed": null, "duplex": null, "wwn": null, "mgmt_only": false, "description": "", "mode": null, "rf_role": null, "rf_channel": null, "poe_mode": null, "poe_type": null, "rf_channel_frequency": null, "rf_channel_width": null, "tx_power": null, "untagged_vlan": null, "tagged_vlans": [], "mark_connected": false, "cable": { "id": 7, "url": "https://demo.netbox.dev/api/dcim/cables/7/", "display": "#7", "label": "" }, "cable_end": "B", "wireless_link": null, "link_peers": [ { "id": 1, "url": "https://demo.netbox.dev/api/circuits/circuit-terminations/1/", "display": "KKDG4923: Termination Z", "circuit": { "id": 1, "url": "https://demo.netbox.dev/api/circuits/circuits/1/", "display": "KKDG4923", "cid": "KKDG4923" }, "term_side": "Z", "cable": 7, "_occupied": true } ], "link_peers_type": "circuits.circuittermination", "wireless_lans": [], "vrf": null, "l2vpn_termination": null, "connected_endpoints": [ { "id": 1, "url": "https://demo.netbox.dev/api/circuits/provider-networks/1/", "display": "Level3 MPLS", "name": "Level3 MPLS" } ], "connected_endpoints_type": "circuits.providernetwork", "connected_endpoints_reachable": true, "tags": [], "custom_fields": {}, "created": "2021-04-14T00:00:00Z", "last_updated": "2021-04-14T17:36:01.841000Z", "count_ipaddresses": 0, "count_fhrp_groups": 0, "_occupied": true }, ``` If I break this down a bit, the following are nested serializers which may be doing addiotional queries - device (object) - vdcs (list of objects) - parent (object) - bridge (object) - lag (object) - tagged vlans (list of objects) - cable ( object) - link peers (list of objects) - wireless lans (list of objects) - connected endpoints (list of objects) - tags (list of objects) - custom fields (object) I'm sure I have missed a few fields in the list but if you understand the scope of how much data is being serialized here and multiply it by 1000 times. Considering this the API runs extrememly fast IMO.
adam commented 2025-12-29 20:31:30 +01:00
Author
Owner
Copy Link

@jiuka commented on GitHub (May 10, 2023):

Adding l2vpn_terminationsand vdcs to the prefetch_related in the InterfaceViewSet reduced amount of DB queries in our setup from 3185 down to 1183. Removing the connected_endpoints, connected_endpoints_type, connected_endpoints_reachable fields from the InterfaceSerializer reduced queries down to 545 and removing link_peers, link_peers_type too down to 57. However I could not get thouse filed prefetched. With this "optimisations" we see requesttimes of ~3s compared to ~7s.

@jiuka commented on GitHub (May 10, 2023): Adding `l2vpn_terminations`and `vdcs` to the [prefetch_related](https://github.com/netbox-community/netbox/blob/develop/netbox/dcim/api/views.py#L494) in the `InterfaceViewSet` reduced amount of DB queries in our setup from 3185 down to 1183. Removing the `connected_endpoints`, `connected_endpoints_type`, `connected_endpoints_reachable` fields from the [InterfaceSerializer](https://github.com/netbox-community/netbox/blob/develop/netbox/dcim/api/serializers.py#L866) reduced queries down to 545 and removing `link_peers`, `link_peers_type` too down to 57. However I could not get thouse filed prefetched. With this "optimisations" we see requesttimes of ~3s compared to ~7s.
adam commented 2025-12-29 20:31:30 +01:00
Author
Owner
Copy Link

@jiuka commented on GitHub (May 10, 2023):

@abhi1693 To use the Interface informations with the Ansible Inventory plugin all interfaces need to be queried. But yes it's not realy a bug more a non functional improvment, but this was the most matching issue template available.

@jiuka commented on GitHub (May 10, 2023): @abhi1693 To use the Interface informations with the Ansible Inventory plugin all interfaces need to be queried. But yes it's not realy a bug more a non functional improvment, but this was the most matching issue template available.
adam commented 2025-12-29 20:31:31 +01:00
Author
Owner
Copy Link

@abhi1693 commented on GitHub (May 10, 2023):

We probably wont remove the connected_endpoints, connected_endpoints_type, connected_endpoints_reachable from the API without more discussion about it. But if you want re-open this an FR with your sugestion and take it from there. I'll close this for now as there is not much to be done.

@abhi1693 commented on GitHub (May 10, 2023): We probably wont remove the `connected_endpoints`, `connected_endpoints_type`, `connected_endpoints_reachable` from the API without more discussion about it. But if you want re-open this an FR with your sugestion and take it from there. I'll close this for now as there is not much to be done.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8031
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?