starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Huge load on a database from excessive queries #7906

New Issue
Closed
opened 2025-12-29 20:29:46 +01:00 by adam · 1 comment
adam commented 2025-12-29 20:29:46 +01:00
Owner
Copy Link

Originally created by @doc-sheet on GitHub (Apr 13, 2023).

NetBox version

v3.4.7

Python version

3.10

Steps to Reproduce

  1. Fill netbox with lots of devices, vms, interfaces etc (usual data collection for many years)
  2. Get full list of interfaces through api (e.g. https://demo.netbox.dev/api/dcim/interfaces/?limit=0)

Hellow.
I don't know the root cause but i believe it's prefetching mechanism in api models.
I have couple examples of queries that result in most cpu load from database in our case

figure 1 
SELECT
  "ipam_l2vpntermination"."id",
  "ipam_l2vpntermination"."created",
  "ipam_l2vpntermination"."last_updated",
  "ipam_l2vpntermination"."custom_field_data",
  "ipam_l2vpntermination"."l2vpn_id",
  "ipam_l2vpntermination"."assigned_object_type_id",
  "ipam_l2vpntermination"."assigned_object_id"
FROM
  "ipam_l2vpntermination"
  INNER JOIN "ipam_l2vpn" ON (
    "ipam_l2vpntermination"."l2vpn_id" = "ipam_l2vpn"."id"
  )
WHERE
  (
    "ipam_l2vpntermination"."assigned_object_id" = ?
    AND "ipam_l2vpntermination"."assigned_object_type_id" = ?
  )
ORDER BY
  "ipam_l2vpn"."name" ASC,
  "ipam_l2vpn"."identifier" ASC
LIMIT
  ?
figure 2 
SELECT
  "dcim_virtualdevicecontext"."id",
  "dcim_virtualdevicecontext"."created",
  "dcim_virtualdevicecontext"."last_updated",
  "dcim_virtualdevicecontext"."custom_field_data",
  "dcim_virtualdevicecontext"."description",
  "dcim_virtualdevicecontext"."device_id",
  "dcim_virtualdevicecontext"."name",
  "dcim_virtualdevicecontext"."status",
  "dcim_virtualdevicecontext"."identifier",
  "dcim_virtualdevicecontext"."primary_ip4_id",
  "dcim_virtualdevicecontext"."primary_ip6_id",
  "dcim_virtualdevicecontext"."tenant_id",
  "dcim_virtualdevicecontext"."comments"
FROM
  "dcim_virtualdevicecontext"
  INNER JOIN "dcim_interface_vdcs" ON (
    "dcim_virtualdevicecontext"."id" = "dcim_interface_vdcs"."virtualdevicecontext_id"
  )
WHERE
  "dcim_interface_vdcs"."interface_id" = ?
ORDER BY
  "dcim_virtualdevicecontext"."name" ASC

Usual case is: get full lists or devices/ipaddresses/vms with pytnetbox.
No subrequests done from python code, just general endpoints with maximum page size.

But on sql side there are lots of joins with stuff like l2vpn, vdc which we don't use at all.

Expected Behavior

Same as now but less excessive

Observed Behavior

CPU load from database and lots of unneeded queries

Originally created by @doc-sheet on GitHub (Apr 13, 2023). ### NetBox version v3.4.7 ### Python version 3.10 ### Steps to Reproduce 1. Fill netbox with lots of devices, vms, interfaces etc (usual data collection for many years) 2. Get full list of interfaces through api (e.g. https://demo.netbox.dev/api/dcim/interfaces/?limit=0) Hellow. I don't know the root cause but i believe it's prefetching mechanism in api models. I have couple examples of queries that result in most cpu load from database in our case <details> <summary>figure 1</summary> ```sql SELECT "ipam_l2vpntermination"."id", "ipam_l2vpntermination"."created", "ipam_l2vpntermination"."last_updated", "ipam_l2vpntermination"."custom_field_data", "ipam_l2vpntermination"."l2vpn_id", "ipam_l2vpntermination"."assigned_object_type_id", "ipam_l2vpntermination"."assigned_object_id" FROM "ipam_l2vpntermination" INNER JOIN "ipam_l2vpn" ON ( "ipam_l2vpntermination"."l2vpn_id" = "ipam_l2vpn"."id" ) WHERE ( "ipam_l2vpntermination"."assigned_object_id" = ? AND "ipam_l2vpntermination"."assigned_object_type_id" = ? ) ORDER BY "ipam_l2vpn"."name" ASC, "ipam_l2vpn"."identifier" ASC LIMIT ? ```` </details> <details> <summary>figure 2</summary> ```sql SELECT "dcim_virtualdevicecontext"."id", "dcim_virtualdevicecontext"."created", "dcim_virtualdevicecontext"."last_updated", "dcim_virtualdevicecontext"."custom_field_data", "dcim_virtualdevicecontext"."description", "dcim_virtualdevicecontext"."device_id", "dcim_virtualdevicecontext"."name", "dcim_virtualdevicecontext"."status", "dcim_virtualdevicecontext"."identifier", "dcim_virtualdevicecontext"."primary_ip4_id", "dcim_virtualdevicecontext"."primary_ip6_id", "dcim_virtualdevicecontext"."tenant_id", "dcim_virtualdevicecontext"."comments" FROM "dcim_virtualdevicecontext" INNER JOIN "dcim_interface_vdcs" ON ( "dcim_virtualdevicecontext"."id" = "dcim_interface_vdcs"."virtualdevicecontext_id" ) WHERE "dcim_interface_vdcs"."interface_id" = ? ORDER BY "dcim_virtualdevicecontext"."name" ASC ``` </details> Usual case is: get full lists or devices/ipaddresses/vms with pytnetbox. No subrequests done from python code, just general endpoints with maximum page size. But on sql side there are lots of joins with stuff like l2vpn, vdc which we don't use at all. ### Expected Behavior Same as now but less excessive ### Observed Behavior CPU load from database and lots of unneeded queries
adam closed this issue 2025-12-29 20:29:46 +01:00
adam commented 2025-12-29 20:29:52 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Apr 13, 2023):

This is not a valid or productive bug report. If you'd like to learn more about the queries being made and why, please consider starting a discussion instead.

@jeremystretch commented on GitHub (Apr 13, 2023): This is not a valid or productive bug report. If you'd like to learn more about the queries being made and why, please consider starting a discussion instead.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#7906
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?