starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Migrate from using uint32 to uint64 for identifiers #1347

New Issue
Closed
opened 2025-12-29 16:31:38 +01:00 by adam · 9 comments
adam commented 2025-12-29 16:31:38 +01:00
Owner
Copy Link

Originally created by @bdlamprecht on GitHub (Oct 23, 2017).

Issue type

[X] Feature request

  • NetBox version: 2.2.2

Description

According to the conversation in the NetworkToCode (Slack Channel) on October 19th, NetBox currently uses uint32 for all of its id values. This provides up to a maximum of 4,294,967,295 unique numbers.

While this quantity appears quite large at first glance, I can foresee this becoming a limiting factor when there are multiple devices with unique interfaces and the possibility of each interface consuming several IP addresses (ipv6).

Also, each new ip-address uses an id that never gets reused (even when that ip-address is deleted) and when scaled out against several VRFs multiplied by each device in the DCIM, that only further exposes this limitation.

By transitioning now to use uint64, this future headache (if not addressed now) will allow for 18,446,744,073,709,551,615 unique id numbers which allows for a much larger quantity of values that should not be exhausted any time soon (not going to say "never").

Originally created by @bdlamprecht on GitHub (Oct 23, 2017). ### Issue type [X] Feature request <!-- Requesting the implementation of a new feature --> * NetBox version: 2.2.2 ### Description According to the conversation in the NetworkToCode (Slack Channel) on October 19th, NetBox currently uses `uint32` for all of its `id` values. This provides up to a maximum of `4,294,967,295` unique numbers. While this quantity appears quite large at first glance, I can foresee this becoming a limiting factor when there are multiple devices with unique interfaces and the possibility of each interface consuming several IP addresses (ipv6). Also, each new `ip-address` uses an `id` that never gets reused (even when that `ip-address` is deleted) and when scaled out against several VRFs multiplied by each device in the `DCIM`, that only further exposes this limitation. By transitioning now to use `uint64`, this future headache (if not addressed now) will allow for `18,446,744,073,709,551,615` unique `id` numbers which allows for a much larger quantity of values that _should_ not be exhausted any time soon (not going to say "_never_").
adam added the status: acceptedtype: housekeeping labels 2025-12-29 16:31:38 +01:00
adam closed this issue 2025-12-29 16:31:38 +01:00
adam commented 2025-12-29 16:31:39 +01:00
Author
Owner
Copy Link

@darkstar commented on GitHub (Oct 24, 2017):

I think this is not as urgent as you make it seem.

If you create 10 new objects every second, for 24/7, you will hit the limit somewhere around 2030.

If we assume that each object in the database takes up about 250 bytes (just a rough guess), you will hit the limit around the time your database exceeds 1TB in size.

I guess we can safely assume that both of these scenarios are still a while off, even for use-cases with lots and lots of tenants and devices :)

@darkstar commented on GitHub (Oct 24, 2017): I think this is not as urgent as you make it seem. If you create 10 new objects every second, for 24/7, you will hit the limit somewhere around 2030. If we assume that each object in the database takes up about 250 bytes (just a rough guess), you will hit the limit around the time your database exceeds 1TB in size. I guess we can safely assume that both of these scenarios are still a while off, even for use-cases with lots and lots of tenants and devices :)
adam commented 2025-12-29 16:31:39 +01:00
Author
Owner
Copy Link

@RyanBreaker commented on GitHub (Oct 24, 2017):

@darkstar /r/theydidthemath

@RyanBreaker commented on GitHub (Oct 24, 2017): @darkstar /r/theydidthemath
adam commented 2025-12-29 16:31:39 +01:00
Author
Owner
Copy Link

@bdlamprecht commented on GitHub (Oct 24, 2017):

Yes, you are correct, it is not as "urgent" as in I need it by tomorrow.

That being said, although I sense the implied sarcasm, but in my circumstances, my customer currently has over 10,000 networking devices which I will eventually need to import. Each of those devices has several VRFs and in each of those VRFs is at least 25 ip-addresses (ipv4 & ipv6). Similar math came be done for interfaces.

When you account for the changing environment of this customer and that an id is never re-used once it has been "created" (even upon deletion), this makes your comment less than helpful.

So claiming that if I only added 10 objects every second I would be good until 2030 is a little naive.
I appreciate you doing the math, but your comment came of as somewhat arrogant.

@bdlamprecht commented on GitHub (Oct 24, 2017): Yes, you are correct, it is not as "urgent" as in I need it by tomorrow. That being said, although I sense the implied sarcasm, but in my circumstances, my customer currently has over 10,000 networking devices which I will eventually need to import. Each of those devices has several VRFs and in each of those VRFs is at least 25 `ip-addresses` (ipv4 & ipv6). Similar math came be done for `interfaces`. When you account for the changing environment of this customer and that an `id` is never re-used once it has been "created" (even upon deletion), this makes your comment less than helpful. So claiming that if I only added 10 objects every second I would be good until 2030 is a little naive. I appreciate you doing the math, but your comment came of as somewhat arrogant.
adam commented 2025-12-29 16:31:39 +01:00
Author
Owner
Copy Link

@RyanBreaker commented on GitHub (Oct 24, 2017):

We're just having some fun discussion, no arrogance intended, apologies if it came off as such as tones don't always come through text the best.

By that sort of use case I can understand the request for this and would agree on a "why not" basis in the first place, no harm in it anyway. I'll look into how this can be changed, and pinging @jeremystretch on if he knows right away of how to do so or if he has any additional input.

@RyanBreaker commented on GitHub (Oct 24, 2017): We're just having some fun discussion, no arrogance intended, apologies if it came off as such as tones don't always come through text the best. By that sort of use case I can understand the request for this and would agree on a "why not" basis in the first place, no harm in it anyway. I'll look into how this can be changed, and pinging @jeremystretch on if he knows right away of how to do so or if he has any additional input.
adam commented 2025-12-29 16:31:39 +01:00
Author
Owner
Copy Link

@darkstar commented on GitHub (Oct 24, 2017):

Sorry if I missed your sense of humo(u)r. There was no sarcasm intended, I just wanted to show that most people don't realize how big a number 4.3 billion is (...and, continuing with the maths, even in your case you could import a couple hundred of those customers into NetBox, or have that one customer change his whole infrastructure/IP range a couple of hundred times ;) ).

But I see your point, and I agree with Ryan on the "why not" basis. It's just an ID field anyway...

@darkstar commented on GitHub (Oct 24, 2017): Sorry if I missed your sense of humo(u)r. There was no sarcasm intended, I just wanted to show that most people don't realize how big a number 4.3 billion is (...and, continuing with the maths, even in your case you could import a couple hundred of those customers into NetBox, or have that one customer change his whole infrastructure/IP range a couple of hundred times ;) ). But I see your point, and I agree with Ryan on the "why not" basis. It's just an ID field anyway...
adam commented 2025-12-29 16:31:40 +01:00
Author
Owner
Copy Link

@RyanBreaker commented on GitHub (Oct 24, 2017):

Looking at this actually, the id types that the application sets up in the database are of Postgres type INTEGER, a 4-byte signed type, but it also uses a custom sequencer which appears to reuse unused IDs instead of auto-incrementing.

All of this appears to go into the inner-workings of the Postgresql driver the application uses with how it sets up the tables in the first place.

@RyanBreaker commented on GitHub (Oct 24, 2017): Looking at this actually, the `id` types that the application sets up in the database are of Postgres type `INTEGER`, a 4-byte signed type, but it also uses a custom sequencer which appears to reuse unused IDs instead of auto-incrementing. All of this appears to go into the inner-workings of the Postgresql driver the application uses with how it sets up the tables in the first place.
adam commented 2025-12-29 16:31:40 +01:00
Author
Owner
Copy Link

@lampwins commented on GitHub (Oct 24, 2017):

Just to chime in; not to throw any fuel on the fire here...

2^32 is a really big number. Also keep in mind the ID's are per table. Meaning you roughly get 2^32 of each kind of object in NetBox, not 2^32 objects in total.

Just a trivial example to show that I really think this is a non issue (for now): taking an example line card from a Nexus 9K which supports 208,000 IPv4 entries, times 10,000 devices, equals 2.08e9 which is less than half of 2^32.

At that scale I question if it is even worth keeping track of information in this manor. If you do manage to get all that data in netbox at some point, I would really like to know how it performs. I have been wanting to so some stress testing in a variety of scenarios for a while but I simple don't have data at that scale.

@lampwins commented on GitHub (Oct 24, 2017): Just to chime in; not to throw any fuel on the fire here... 2^32 is a _really_ big number. Also keep in mind the ID's are _per table_. Meaning you roughly get 2^32 of each kind of object in NetBox, not 2^32 objects in total. Just a trivial example to show that I really think this is a non issue (for now): taking an example line card from a Nexus 9K which supports 208,000 IPv4 entries, times 10,000 devices, equals 2.08e9 which is less than half of 2^32. At that scale I question if it is even worth keeping track of information in this manor. If you do manage to get all that data in netbox at some point, I would really like to know how it performs. I have been wanting to so some stress testing in a variety of scenarios for a while but I simple don't have data at that scale.
adam commented 2025-12-29 16:31:40 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Oct 25, 2017):

My primary concern with the 32-bit primary key size relates to inventory items. Many devices can have dozens or hundreds of individual components. I have a Juniper EX9200, for instance, with 220 discrete inventory items (most of which are optics).

This is a concern because the most efficient way to update a device's inventory data is to wipe out all of its existing items and create a new object for each detected item. (Yes, there are more intelligent approaches, but they involve somewhat rather complex logic regarding parent relationships and would result in much longer run times.) So, if you're updating inventory data for 5,000 devices, with an average of say 50 items per device, each run increments the PK value by around 250,000. This will exhaust the 32-bit PK space after around 17,000 runs.

Granted, these are fairly large numbers, and 17 thousand runs allows for 46 years of daily inventory runs, but it's still a bit too tangible a number for comfort. Whether it makes sense to expand the field size for all objects or just InventoryItems I don't know.

@jeremystretch commented on GitHub (Oct 25, 2017): My primary concern with the 32-bit primary key size relates to inventory items. Many devices can have dozens or hundreds of individual components. I have a Juniper EX9200, for instance, with 220 discrete inventory items (most of which are optics). This is a concern because the most efficient way to update a device's inventory data is to wipe out all of its existing items and create a new object for each detected item. (Yes, there are more intelligent approaches, but they involve somewhat rather complex logic regarding parent relationships and would result in much longer run times.) So, if you're updating inventory data for 5,000 devices, with an average of say 50 items per device, each run increments the PK value by around 250,000. This will exhaust the 32-bit PK space after around 17,000 runs. Granted, these are fairly large numbers, and 17 thousand runs allows for 46 years of daily inventory runs, but it's still a bit too tangible a number for comfort. Whether it makes sense to expand the field size for _all_ objects or just InventoryItems I don't know.
adam commented 2025-12-29 16:31:41 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 19, 2021):

Casually commenting three years later...

Looks like DEFAULT_AUTO_FIELD is coming in Django 3.2, which should help with this.

@jeremystretch commented on GitHub (Jan 19, 2021): Casually commenting three years later... Looks like [`DEFAULT_AUTO_FIELD`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-DEFAULT_AUTO_FIELD) is coming in Django 3.2, which should help with this.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: housekeeping
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#1347
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?