starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Rack list contains duplicate entries in paginated API responses #3111

New Issue
Closed
opened 2025-12-29 18:25:42 +01:00 by adam · 13 comments
adam commented 2025-12-29 18:25:42 +01:00
Owner
Copy Link

Originally created by @henyxia on GitHub (Jan 3, 2020).

Originally assigned to: @jeremystretch on GitHub.

Environment

  • Python version: 3.7
  • NetBox version: 2.6.10

Steps to Reproduce

  1. GET /api/dcim/racks/ and iterate over all elements

Expected Behavior

List all racks without duplicates.

Expected Behavior

Some duplicated racks.

Context

We have around 4500 racks and we have only one duplicate.

Script used to reproduce the isse

#!/usr/bin/perl

use strict;
use warnings;

use JSON;

my $next = 'http://127.0.0.1:32769/api/dcim/racks/';
my $response;
my $res;

do {
    $response = `curl -s '$next'`;
    $res = decode_json($response);
    $next = $res->{next};
    foreach my $e (@{$res->{results}}) {
        print "$e->{id}\n";
    }
} while ($next);

And then run dup.pl | sort | uniq -d

Links

This might be a regression, or at least link to: #2938

Originally created by @henyxia on GitHub (Jan 3, 2020). Originally assigned to: @jeremystretch on GitHub. <!-- NOTE: This form is only for reproducible bugs. If you need assistance with NetBox installation, or if you have a general question, DO NOT open an issue. Instead, post to our mailing list: https://groups.google.com/forum/#!forum/netbox-discuss Please describe the environment in which you are running NetBox. Be sure that you are running an unmodified instance of the latest stable release before submitting a bug report. --> ### Environment * Python version: 3.7 * NetBox version: 2.6.10 <!-- Describe in detail the exact steps that someone else can take to reproduce this bug using the current stable release of NetBox (or the current beta release where applicable). Begin with the creation of any necessary database objects and call out every operation being performed explicitly. If reporting a bug in the REST API, be sure to reconstruct the raw HTTP request(s) being made: Don't rely on a wrapper like pynetbox. --> ### Steps to Reproduce 1. GET /api/dcim/racks/ and iterate over all elements <!-- What did you expect to happen? --> ### Expected Behavior List all racks without duplicates. <!-- What happened instead? --> ### Expected Behavior Some duplicated racks. ### Context We have around 4500 racks and we have only one duplicate. ### Script used to reproduce the isse ``` #!/usr/bin/perl use strict; use warnings; use JSON; my $next = 'http://127.0.0.1:32769/api/dcim/racks/'; my $response; my $res; do { $response = `curl -s '$next'`; $res = decode_json($response); $next = $res->{next}; foreach my $e (@{$res->{results}}) { print "$e->{id}\n"; } } while ($next); ``` And then run `dup.pl | sort | uniq -d` ### Links This might be a regression, or at least link to: #2938
adam added the type: bugstatus: accepted labels 2025-12-29 18:25:42 +01:00
adam closed this issue 2025-12-29 18:25:42 +01:00
adam commented 2025-12-29 18:25:43 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 3, 2020):

First, this is going to be very difficult to reproduce without any example data. Can you see if you can narrow it down to a specific set of conditions or naming pattern?

Second, if this is related to NaturalOrderingManager, it will be be rendered moot by #3799 pretty soon, so it may not be worth digging too far into it.

@jeremystretch commented on GitHub (Jan 3, 2020): First, this is going to be very difficult to reproduce without any example data. Can you see if you can narrow it down to a specific set of conditions or naming pattern? Second, if this is related to `NaturalOrderingManager`, it will be be rendered moot by #3799 pretty soon, so it may not be worth digging too far into it.
adam commented 2025-12-29 18:25:43 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 3, 2020):

@henyxia can you try commenting out this line under class Rack in dcim/models.py and see if you can still reproduce this issue?

    objects = NaturalOrderingManager()
@jeremystretch commented on GitHub (Jan 3, 2020): @henyxia can you try commenting out [this line](https://github.com/netbox-community/netbox/blob/develop/netbox/dcim/models.py#L548) under `class Rack` in `dcim/models.py` and see if you can still reproduce this issue? ``` objects = NaturalOrderingManager() ```
adam commented 2025-12-29 18:25:43 +01:00
Author
Owner
Copy Link

@henyxia commented on GitHub (Jan 7, 2020):

Oy!
So, commenting the line (and restarting gunicorn) does not change anything.
I still got the same amount of duplicates in both (with and without the comment) cases.
I added even more racks and it seems to be even more duplicates.
Right now, I'm trying to isolate the replicate the issue with some random data for the issue to be more reproductible :)

@henyxia commented on GitHub (Jan 7, 2020): Oy! So, commenting the line (and restarting gunicorn) does not change anything. I still got the same amount of duplicates in both (with and without the comment) cases. I added even more racks and it seems to be even more duplicates. Right now, I'm trying to isolate the replicate the issue with some random data for the issue to be more reproductible :)
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@henyxia commented on GitHub (Jan 7, 2020):

So, right now I am now able to reproduce it on a bare new install.
Steps:

  1. Spawn a new netbox instance (latest version)
  2. Create a new site
  3. Create 8k racks in this site
for i in {1..8000}; do curl -H "Content-Type: application/json" -H "Accept: application/json" -d '{"name":"rack1", "site":1}' -X POST -H 'Authorization: Token  0123456789abcdef0123456789abcdef01234567' http://127.0.0.1:32769/api/dcim/racks/; done

And then, multiple duplicates are returned with dup.pl | sort | uniq -d

@henyxia commented on GitHub (Jan 7, 2020): So, right now I am now able to reproduce it on a bare new install. Steps: 1. Spawn a new netbox instance (latest version) 2. Create a new site 3. Create 8k racks in this site ``` for i in {1..8000}; do curl -H "Content-Type: application/json" -H "Accept: application/json" -d '{"name":"rack1", "site":1}' -X POST -H 'Authorization: Token 0123456789abcdef0123456789abcdef01234567' http://127.0.0.1:32769/api/dcim/racks/; done ``` And then, multiple duplicates are returned with `dup.pl | sort | uniq -d`
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@henyxia commented on GitHub (Jan 7, 2020):

I'm still searching on my side but I think the bug is now easily reproducible (through the steps described before).
Do you think anything need to be added to help?

@henyxia commented on GitHub (Jan 7, 2020): I'm still searching on my side but I think the bug is now easily reproducible (through the steps described before). Do you think anything need to be added to help?
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@hSaria commented on GitHub (Jan 7, 2020):

I can confirm that this is happening when there are racks with the same name. You don't even need to go crazy with the numbers; 50 racks with 10 on the paging limit can recreate this issue.

The lower the page limit, the more often you'll see duplicates. I'm also running NetBox directly using manage.py runserver to eliminate as many variables as possible.

This happens with or without brief; I just added it to spare you from the long output.

>>> import json, requests
>>> 
>>> headers = {'Authorization': 'Token ef1764f45a5dd423e7bb6ba8502aabb9838b0c36', 'Content-type': 'application/json'}
>>> 
>>> for i in range(50):
...     resp = requests.post('http://netbox/api/dcim/racks/', data='{"name": "rack1", "site": 1}', headers=headers)
... 
>>> url = 'http://netbox/api/dcim/racks/?limit=10&brief=true'
>>> results = []
>>> 
>>> while url:
...     data = json.loads(requests.get(url, headers=headers).content)
...     url = data['next']
...     results += data['results']
... 
>>> for index, result in enumerate(results):
...     if results.count(result) > 1:
...         print('duplicate', result['id'], 'at', index)
... 
duplicate 50 at 9
duplicate 50 at 19
>>> results[9]
{'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None}
>>> results[19]
{'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None}
@hSaria commented on GitHub (Jan 7, 2020): I can confirm that this is happening when there are racks with the same name. You don't even need to go crazy with the numbers; 50 racks with 10 on the paging limit can recreate this issue. The lower the page limit, the more often you'll see duplicates. I'm also running NetBox directly using `manage.py runserver` to eliminate as many variables as possible. > This happens with or without `brief`; I just added it to spare you from the long output. ```python >>> import json, requests >>> >>> headers = {'Authorization': 'Token ef1764f45a5dd423e7bb6ba8502aabb9838b0c36', 'Content-type': 'application/json'} >>> >>> for i in range(50): ... resp = requests.post('http://netbox/api/dcim/racks/', data='{"name": "rack1", "site": 1}', headers=headers) ... >>> url = 'http://netbox/api/dcim/racks/?limit=10&brief=true' >>> results = [] >>> >>> while url: ... data = json.loads(requests.get(url, headers=headers).content) ... url = data['next'] ... results += data['results'] ... >>> for index, result in enumerate(results): ... if results.count(result) > 1: ... print('duplicate', result['id'], 'at', index) ... duplicate 50 at 9 duplicate 50 at 19 >>> results[9] {'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None} >>> results[19] {'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None} ```
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@hSaria commented on GitHub (Jan 7, 2020):

I haven't gone around testing it, but this should be affecting other models too as ordering is not guaranteed if left unspecified as warned about in Django's docs.

@hSaria commented on GitHub (Jan 7, 2020): I haven't gone around testing it, but this should be affecting other models too as ordering is not guaranteed if left unspecified as [warned](https://docs.djangoproject.com/en/2.2/ref/models/options/#ordering) about in Django's docs.
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 7, 2020):

@hSaria Yeah, I think what's happening is that the ordering of Rack objects for which the set (site, group, name) is identical is non-deterministic. As you point out, this is likely affecting other models with non-unique ordering fields (though that doesn't seem to come up much). We'll need to comb through the models and identify any similar cases.

@jeremystretch commented on GitHub (Jan 7, 2020): @hSaria Yeah, I think what's happening is that the ordering of Rack objects for which the set `(site, group, name)` is identical is non-deterministic. As you point out, this is likely affecting other models with non-unique ordering fields (though that doesn't seem to come up much). We'll need to comb through the models and identify any similar cases.
adam commented 2025-12-29 18:25:44 +01:00
Author
Owner
Copy Link

@hSaria commented on GitHub (Jan 7, 2020):

I've tested the same thing on devices by leaving the name empty and it does also return duplicates. I suppose it's rare that people create into duplicate objects.

I'll go through the models to try to figure out which ones can be dup-ed with the current order, but the eyes of someone more experienced would definitely prove worthy.

@hSaria commented on GitHub (Jan 7, 2020): I've tested the same thing on devices by leaving the name empty and it does also return duplicates. I suppose it's rare that people create into duplicate objects. I'll go through the models to try to figure out which ones can be dup-ed with the current order, but the eyes of someone more experienced would definitely prove worthy.
adam commented 2025-12-29 18:25:45 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 7, 2020):

FYI the apparent solution here is to simply append id to the list of fields by which each affected model is ordered.

@jeremystretch commented on GitHub (Jan 7, 2020): FYI the apparent solution here is to simply append `id` to the list of fields by which each affected model is ordered.
adam commented 2025-12-29 18:25:45 +01:00
Author
Owner
Copy Link

@hSaria commented on GitHub (Jan 7, 2020):

Found and confirmed (tested) the following:

  • dcim.Device
  • extras.ImageAttachment
  • ipam.VRF
  • ipam.Prefix
  • ipam.IPAddress
  • ipam.VLANGroup (not sure if you're supposed to be able to create a duplicate global group)
  • ipam.VLAN

I'll work on a PR for this. I don't mind if someone else wants to.

@hSaria commented on GitHub (Jan 7, 2020): Found and confirmed (tested) the following: * dcim.Device * extras.ImageAttachment * ipam.VRF * ipam.Prefix * ipam.IPAddress * ipam.VLANGroup (not sure if you're supposed to be able to create a duplicate global group) * ipam.VLAN I'll work on a PR for this. I don't mind if someone else wants to.
adam commented 2025-12-29 18:25:45 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 8, 2020):

Hold off on the PR for now. This will require introducing new migrations and I'd rather avoid having to refactor the significant number of migrations pending for v2.7.

@jeremystretch commented on GitHub (Jan 8, 2020): Hold off on the PR for now. This will require introducing new migrations and I'd rather avoid having to refactor the significant number of migrations pending for v2.7.
adam commented 2025-12-29 18:25:45 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 15, 2020):

This will be fixed in the v2.7 release.

@jeremystretch commented on GitHub (Jan 15, 2020): This will be fixed in the v2.7 release.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3111
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?