starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Bulk importing devices extremely slow #11083

New Issue
Closed
opened 2025-12-29 21:40:05 +01:00 by adam · 8 comments
adam commented 2025-12-29 21:40:05 +01:00
Owner
Copy Link

Originally created by @martijnremmen on GitHub (Apr 25, 2025).

Originally assigned to: @jeremystretch on GitHub.

Deployment Type

Self-hosted

NetBox Version

v4.2.6

Python Version

3.12

Steps to Reproduce

  1. create all required models for a device (role, manufacturer, status, site)
  2. import device_type Catalyst 9200L-48P-4X
  3. import 10 devices using this device_type as csv.

Notes:

  • I am using a locally hosted Postgres instance in Docker with image postgres:15-alpine

Expected Behavior

the import completing within seconds

Observed Behavior

the import completes in more than 1 minute. Performs more than 11K SQL queries. And gives no progress indicator. for just 10 devices

Image
Originally created by @martijnremmen on GitHub (Apr 25, 2025). Originally assigned to: @jeremystretch on GitHub. ### Deployment Type Self-hosted ### NetBox Version v4.2.6 ### Python Version 3.12 ### Steps to Reproduce 1. create all required models for a device (role, manufacturer, status, site) 2. import device_type [Catalyst 9200L-48P-4X](https://github.com/netbox-community/devicetype-library/blob/master/device-types/Cisco/C9200L-48P-4X.yaml) 3. import 10 devices using this device_type as csv. Notes: - I am using a locally hosted Postgres instance in Docker with image postgres:15-alpine ### Expected Behavior the import completing within seconds ### Observed Behavior the import completes in more than 1 minute. Performs more than 11K SQL queries. And gives no progress indicator. for just 10 devices <img width="213" alt="Image" src="https://github.com/user-attachments/assets/c4fd2583-8e04-4d70-b031-43e21d394e4a" />
adam added the type: bugstatus: acceptedseverity: low labels 2025-12-29 21:40:05 +01:00
adam closed this issue 2025-12-29 21:40:05 +01:00
adam commented 2025-12-29 21:40:05 +01:00
Author
Owner
Copy Link

@jnovinger commented on GitHub (Apr 28, 2025):

Hey @martijnremmen , thanks for the bug report. While I'm able to reproduce some slowness using your STR (or at least how I interpreted them), I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows.

On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries.

Image

This is the CSV I used to do the import of the 10 devices:

name,status,tenant,site,location,rack,role,manufacturer,device_type
Cisco Catalyst 9200L-48P-4X-1,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-2,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-3,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-4,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-5,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-6,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-7,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-8,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-9,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-10,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X

Can you provide any more details that would help us reproduce this?

@jnovinger commented on GitHub (Apr 28, 2025): Hey @martijnremmen , thanks for the bug report. While I'm able to reproduce some slowness using your STR (or at least how I interpreted them), I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows. On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries. ![Image](https://github.com/user-attachments/assets/fed2a3a2-aea6-44cb-9e0d-f6ea3b610b84) This is the CSV I used to do the import of the 10 devices: ```csv name,status,tenant,site,location,rack,role,manufacturer,device_type Cisco Catalyst 9200L-48P-4X-1,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-2,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-3,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-4,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-5,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-6,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-7,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-8,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-9,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X Cisco Catalyst 9200L-48P-4X-10,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X ``` Can you provide any more details that would help us reproduce this?
adam commented 2025-12-29 21:40:06 +01:00
Author
Owner
Copy Link

@kkthxbye-code commented on GitHub (Apr 28, 2025):

I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows.

On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries.

Image

You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request.

@kkthxbye-code commented on GitHub (Apr 28, 2025): > I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows. > > On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries. > > ![Image](https://github.com/user-attachments/assets/fed2a3a2-aea6-44cb-9e0d-f6ea3b610b84) You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request.
adam commented 2025-12-29 21:40:06 +01:00
Author
Owner
Copy Link

@jnovinger commented on GitHub (Apr 28, 2025):

You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request.

Doh, thanks, you are correct. I'm seeing similar output now.

Image

@jnovinger commented on GitHub (Apr 28, 2025): > You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request. Doh, thanks, you are correct. I'm seeing similar output now. ![Image](https://github.com/user-attachments/assets/344d89ad-31e6-4cf7-b34d-09d0cd223d7c)
adam commented 2025-12-29 21:40:07 +01:00
Author
Owner
Copy Link

@jnovinger commented on GitHub (Apr 28, 2025):

FWIW, batch delete on the 10 created devices has the same issue--this time with ~16,000 queries.

@jnovinger commented on GitHub (Apr 28, 2025): FWIW, batch delete on the 10 created devices has the same issue--this time with ~16,000 queries.
adam commented 2025-12-29 21:40:07 +01:00
Author
Owner
Copy Link

@martijnremmen commented on GitHub (Apr 28, 2025):

Thanks for investigating the issue!

also note when removing the interfaces from the device type, importing gets a lot faster with 737 queries (still alot)

@martijnremmen commented on GitHub (Apr 28, 2025): Thanks for investigating the issue! also note when removing the interfaces from the device type, importing gets a lot faster with 737 queries (still alot)
adam commented 2025-12-29 21:40:07 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jul 28, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Jul 28, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/main/CONTRIBUTING.md).
adam commented 2025-12-29 21:40:07 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jul 31, 2025):

Although I haven't dug into this yet, the number of queries resulting from a rather small import does seem extreme.

As a potential interim solution, NetBox v4.4 will introduce the ability to perform bulk imports and other operations as background jobs (see #19589), which would at least unblock the user while the import is completed.

@jeremystretch commented on GitHub (Jul 31, 2025): Although I haven't dug into this yet, the number of queries resulting from a rather small import does seem extreme. As a potential interim solution, NetBox v4.4 will introduce the ability to perform bulk imports and other operations as background jobs (see #19589), which would at least unblock the user while the import is completed.
adam commented 2025-12-29 21:40:08 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jul 31, 2025):

Pleased to say I was able to achieve a substantial improvement by prefetching related objects for newly-created device components prior to triggering creation of the changelog records.

Testing using a device type with 48 interfaces, 1 console port, and 2 power ports, the total number of queries resulting from importing a single device dropped from 1173 to 366. (Note that a fairly large number of queries still expected due to the overhead of creating all the device components and their resulting search and changelog records.) A similar improvement was observed when creating individual devices via the model form as well.

@jeremystretch commented on GitHub (Jul 31, 2025): Pleased to say I was able to achieve a substantial improvement by prefetching related objects for newly-created device components prior to triggering creation of the changelog records. Testing using a device type with 48 interfaces, 1 console port, and 2 power ports, the total number of queries resulting from importing a single device dropped from 1173 to 366. (Note that a _fairly_ large number of queries still expected due to the overhead of creating all the device components and their resulting search and changelog records.) A similar improvement was observed when creating individual devices via the model form as well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label severity: low status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#11083
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?