Inconsistent data being returned from new dcim.mac_addresses endpoint #11400

New Issue

Closed

opened 2025-12-29 21:44:40 +01:00 by adam · 19 comments

adam commented 2025-12-29 21:44:40 +01:00

Owner

Copy Link

Originally created by @jseifeddine on GitHub (Jul 20, 2025).

Originally assigned to: @jseifeddine on GitHub.

Deployment Type

Self-hosted

NetBox Version

v4.3.4

Python Version

3.11

Steps to Reproduce

BUG: Total interfaces from dcim.mac_addresses endpoint (count 7978) does not match total interfaces from dcim.interfaces endpoint (count 8628)

After having stumbled upon a bug present in v4.1.5 with IPAM, which I successfully reproduced here https://github.com/jseifeddine/netbox-ipam-bug/

Realizing that it was fixed in latest version from v4.3.3 - we upgraded to the newest v4.3.4 - and now experiencing a very similar issue with the new MAC addresses objects on the new dcim.mac_addresses endpoint. (mac objects and endpoint introduced in v4.2.0)

We use pynetbox to maintain "synchronization" between devices and NetBox.

Syncing interface information (Status, Description, IPs, speed, type, mac address etc)

Our sync script only makes changes when it needs to, doing a delta check before hand to figure out what in NetBox needs updating/deleting/creating.

IP addresses were an issue, as per the bug repository attached above - causing unnecessary operations.

That was fixed with IP addresses, but MAC addresses now have similar issues. I haven't yet been able to reproduce it exactly how I did with the IPAM repo above, but in our dev environment, which is a clone of production - I can see inconsistent data being returned from the dcim.mac_addresses endpoint, here is the best way I can display that:

NetBox v4.3.4

Found 8748 MAC addresses in NetBox for device 13756 from dcim.mac_addresses endpoint.
Found 9262 interfaces in NetBox for device 13756 from dcim.interfaces endpoint.
Found 8748 MAC addresses on 7978 interfaces with from dcim.mac_addresses endpoint.
Found 8748 MAC addresses on 8628 interfaces from dcim.interfaces endpoint.
BUG: Total interfaces from dcim.mac_addresses endpoint (7978) does not match total interfaces from dcim.interfaces endpoint (8628)

Test complete

Running this script;

#!venv/bin/python

from dotenv import load_dotenv
import os
import pynetbox

load_dotenv()

nb = pynetbox.api(
    url=os.getenv("NETBOX_API_URL"),
    token=os.getenv("NETBOX_API_TOKEN"),
    threading=True
)

DEVICE_ID = 13756

all_netbox_macs = list(nb.dcim.mac_addresses.filter(device_id=DEVICE_ID))
all_netbox_interfaces = list(nb.dcim.interfaces.filter(device_id=DEVICE_ID))

print(f"Found {len(all_netbox_macs)} MAC addresses in NetBox for device {DEVICE_ID} from dcim.mac_addresses endpoint.")
print(f"Found {len(all_netbox_interfaces)} interfaces in NetBox for device {DEVICE_ID} from dcim.interfaces endpoint.")

mac_interfaces = {}
interface_macs = {}

for mac in all_netbox_macs:
    if mac.assigned_object and mac.assigned_object.name:
        if not mac_interfaces.get(mac.assigned_object.name):
            mac_interfaces[mac.assigned_object.name] = []
        mac_interfaces[mac.assigned_object.name].append(mac)

total_macs = sum(len(macs) for macs in mac_interfaces.values())
print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces with from dcim.mac_addresses endpoint.")

for interface in all_netbox_interfaces:
    if interface.mac_addresses:
        if not interface_macs.get(interface.name):
            interface_macs[interface.name] = []
        for mac in interface.mac_addresses:
            interface_macs[interface.name].append(mac)

total_macs_from_interface = sum(len(macs) for macs in interface_macs.values())
print(f"Found {total_macs_from_interface} MAC addresses on {len(interface_macs)} interfaces from dcim.interfaces endpoint.")

# Expose bug in mac / interface counts returned from different endpoints

# total_macs is the total number of MAC addresses on the device from the dcim.mac_addresses endpoint
# total_macs_from_interface is the total number of MAC addresses on all interfaces from the dcim.interfaces endpoint
if total_macs != total_macs_from_interface:
    print(f"BUG: Total MAC addresses from dcim.mac_addresses endpoint ({total_macs}) does not match total MAC addresses from dcim.interfaces endpoint ({total_macs_from_interface})")

# mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint
# interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint
if len(mac_interfaces) != len(interface_macs):
    print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})")

print()
print("Test complete")

Expected Behavior

Consistent data from the new MAC Addresses endpoint

For example, correct interface counts derived from assigned_object

Observed Behavior

BUG: Total interfaces from dcim.mac_addresses endpoint (7978) does not match total interfaces from dcim.interfaces endpoint (8628)

# mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint
# interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint
if len(mac_interfaces) != len(interface_macs):
    print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})")

Originally created by @jseifeddine on GitHub (Jul 20, 2025). Originally assigned to: @jseifeddine on GitHub. ### Deployment Type Self-hosted ### NetBox Version v4.3.4 ### Python Version 3.11 ### Steps to Reproduce BUG: Total interfaces from `dcim.mac_addresses` endpoint (count 7978) does not match total interfaces from `dcim.interfaces` endpoint (count 8628) After having stumbled upon a bug present in `v4.1.5` with `IPAM`, which I successfully reproduced here https://github.com/jseifeddine/netbox-ipam-bug/ Realizing that it was fixed in latest version from v4.3.3 - we upgraded to the newest `v4.3.4` - and now experiencing a very similar issue with the new MAC addresses objects on the new `dcim.mac_addresses` endpoint. (mac objects and endpoint introduced in v4.2.0) We use `pynetbox` to maintain "synchronization" between devices and NetBox. Syncing interface information (Status, Description, IPs, speed, type, mac address etc) Our sync script only makes changes when it needs to, doing a delta check before hand to figure out what in NetBox needs updating/deleting/creating. IP addresses were an issue, as per the bug repository attached above - causing unnecessary operations. That was fixed with IP addresses, but MAC addresses now have similar issues. I haven't yet been able to reproduce it exactly how I did with the IPAM repo above, but in our dev environment, which is a clone of production - I can see inconsistent data being returned from the `dcim.mac_addresses` endpoint, here is the best way I can display that: NetBox v4.3.4 ```bash Found 8748 MAC addresses in NetBox for device 13756 from dcim.mac_addresses endpoint. Found 9262 interfaces in NetBox for device 13756 from dcim.interfaces endpoint. Found 8748 MAC addresses on 7978 interfaces with from dcim.mac_addresses endpoint. Found 8748 MAC addresses on 8628 interfaces from dcim.interfaces endpoint. BUG: Total interfaces from dcim.mac_addresses endpoint (7978) does not match total interfaces from dcim.interfaces endpoint (8628) Test complete ``` Running this script; ```python #!venv/bin/python from dotenv import load_dotenv import os import pynetbox load_dotenv() nb = pynetbox.api( url=os.getenv("NETBOX_API_URL"), token=os.getenv("NETBOX_API_TOKEN"), threading=True ) DEVICE_ID = 13756 all_netbox_macs = list(nb.dcim.mac_addresses.filter(device_id=DEVICE_ID)) all_netbox_interfaces = list(nb.dcim.interfaces.filter(device_id=DEVICE_ID)) print(f"Found {len(all_netbox_macs)} MAC addresses in NetBox for device {DEVICE_ID} from dcim.mac_addresses endpoint.") print(f"Found {len(all_netbox_interfaces)} interfaces in NetBox for device {DEVICE_ID} from dcim.interfaces endpoint.") mac_interfaces = {} interface_macs = {} for mac in all_netbox_macs: if mac.assigned_object and mac.assigned_object.name: if not mac_interfaces.get(mac.assigned_object.name): mac_interfaces[mac.assigned_object.name] = [] mac_interfaces[mac.assigned_object.name].append(mac) total_macs = sum(len(macs) for macs in mac_interfaces.values()) print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces with from dcim.mac_addresses endpoint.") for interface in all_netbox_interfaces: if interface.mac_addresses: if not interface_macs.get(interface.name): interface_macs[interface.name] = [] for mac in interface.mac_addresses: interface_macs[interface.name].append(mac) total_macs_from_interface = sum(len(macs) for macs in interface_macs.values()) print(f"Found {total_macs_from_interface} MAC addresses on {len(interface_macs)} interfaces from dcim.interfaces endpoint.") # Expose bug in mac / interface counts returned from different endpoints # total_macs is the total number of MAC addresses on the device from the dcim.mac_addresses endpoint # total_macs_from_interface is the total number of MAC addresses on all interfaces from the dcim.interfaces endpoint if total_macs != total_macs_from_interface: print(f"BUG: Total MAC addresses from dcim.mac_addresses endpoint ({total_macs}) does not match total MAC addresses from dcim.interfaces endpoint ({total_macs_from_interface})") # mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint # interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint if len(mac_interfaces) != len(interface_macs): print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})") print() print("Test complete") ``` ### Expected Behavior Consistent data from the new MAC Addresses endpoint For example, correct interface counts derived from assigned_object ### Observed Behavior BUG: Total interfaces from dcim.mac_addresses endpoint (7978) does not match total interfaces from dcim.interfaces endpoint (8628) ```python # mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint # interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint if len(mac_interfaces) != len(interface_macs): print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})") ```

adam added the type: bugstatus: acceptedseverity: low labels 2025-12-29 21:44:40 +01:00

adam closed this issue 2025-12-29 21:44:41 +01:00

adam commented 2025-12-29 21:44:41 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 20, 2025):

ok, I've updated that repo, to include testing MAC address endpoint bugs and able to reproduce it.

Steps to reproduce

#Requires docker, docker compose, curl, python3 etc. check the repo
git clone https://github.com/jseifeddine/netbox-ipam-bug
cd netbox-ipam-bug
NETBOX_VERSION=v4.3.4 ./initialize-and-test.sh

Output of running tests:

venv➜ netbox-ipam-bug (main) ✗ ./test-macs.py
Found 10000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint.
Found 10000 interfaces in NetBox for device 1 from dcim.interfaces endpoint.
Found 10000 MAC addresses on 9997 interfaces with from dcim.mac_addresses endpoint.
Found 10000 MAC addresses on 10000 interfaces from dcim.interfaces endpoint.
BUG: Total interfaces from dcim.mac_addresses endpoint (9997) does not match total interfaces from dcim.interfaces endpoint (10000)

Test complete

venv➜ netbox-ipam-bug (main) ✗ ./test-ipam.py                             
Found 10000 interfaces in NetBox for device 1
Found 20000 IP addresses in NetBox for device 1
Found 10000 MAC addresses in NetBox for device 1
Found 10000 interfaces with 20000 IP addresses
Found 9997 interfaces with 10000 MAC addresses

Each interface should have exactly two addresses
- 172.17.0.1/32
- ff1d:c7c:7b44:d39d:ab3d:6fde:f46a:4648/64

Each interface should have exactly one MAC address
- 18:2A:D3:65:90:2E


BUG: Interface dummy0 has 4 MAC addresses
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0


Test complete

@jseifeddine commented on GitHub (Jul 20, 2025): ok, I've updated that repo, to include testing MAC address endpoint bugs and able to reproduce it. Steps to reproduce ```bash #Requires docker, docker compose, curl, python3 etc. check the repo git clone https://github.com/jseifeddine/netbox-ipam-bug cd netbox-ipam-bug NETBOX_VERSION=v4.3.4 ./initialize-and-test.sh ``` Output of running tests: ```bash venv➜ netbox-ipam-bug (main) ✗ ./test-macs.py Found 10000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint. Found 10000 interfaces in NetBox for device 1 from dcim.interfaces endpoint. Found 10000 MAC addresses on 9997 interfaces with from dcim.mac_addresses endpoint. Found 10000 MAC addresses on 10000 interfaces from dcim.interfaces endpoint. BUG: Total interfaces from dcim.mac_addresses endpoint (9997) does not match total interfaces from dcim.interfaces endpoint (10000) Test complete ``` ```bash venv➜ netbox-ipam-bug (main) ✗ ./test-ipam.py Found 10000 interfaces in NetBox for device 1 Found 20000 IP addresses in NetBox for device 1 Found 10000 MAC addresses in NetBox for device 1 Found 10000 interfaces with 20000 IP addresses Found 9997 interfaces with 10000 MAC addresses Each interface should have exactly two addresses - 172.17.0.1/32 - ff1d:c7c:7b44:d39d:ab3d:6fde:f46a:4648/64 Each interface should have exactly one MAC address - 18:2A:D3:65:90:2E BUG: Interface dummy0 has 4 MAC addresses MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 Test complete ```

adam commented 2025-12-29 21:44:42 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 20, 2025):

A workaround, use the dcim.interfaces endpoint to get reliable mac address data

        # Get interfaces 
        self.logger.debug("Fetching existing NetBox interfaces...")
        all_netbox_interfaces = {
            iface.name: iface 
            for iface in netbox_device.api.dcim.interfaces.filter(device_id=netbox_device.id)
        }
        self.logger.debug(f"Found {len(all_netbox_interfaces)} existing interfaces in NetBox")

        # !!! as there is a bug in netbox, we can't use the mac_addresses endpoint !!!
        # # Get ALL existing MAC addresses for this device with their interface assignments
        # self.logger.debug("Fetching existing MAC addresses...")
        # netbox_macs = {}  # interface_name -> set of MAC addresses
        # all_netbox_macs = list(netbox_device.api.dcim.mac_addresses.filter(device_id=netbox_device.id))
        # self.logger.debug(f"Found {len(all_netbox_macs)} existing MAC addresses in NetBox")
        
        # for mac in all_netbox_macs:
        #     if mac.assigned_object:
        #         iface_name = mac.assigned_object.name
        #         if iface_name not in netbox_macs:
        #             netbox_macs[iface_name] = set()
        #         mac_str = self.normalize_mac_address(mac.mac_address)
        #         netbox_macs[iface_name].add(mac_str)

        # Get MAC addresses from interfaces instead of using the MAC address endpoint
        self.logger.debug("Extracting MAC addresses from interfaces...")
        netbox_macs = {}  # interface_name -> set of MAC addresses
        all_netbox_macs = []  # List of mock MAC address objects
        
        # Create a mock MAC address object with the necessary attributes
        class MockMacAddress:
            def __init__(self, mac_id, mac_address, interface_id, interface_name):
                self.id = mac_id
                self.mac_address = mac_address
                self.assigned_object = type('obj', (), {'id': interface_id, 'name': interface_name})

        # Create mock MAC address objects from interface data
        for iface in all_netbox_interfaces.values():
            for mac in iface.mac_addresses:
                mac_str = self.normalize_mac_address(mac.mac_address)
                
                # Create mock MAC address object
                mock_mac = MockMacAddress(mac.id, mac_str, iface.id, iface.name)
                all_netbox_macs.append(mock_mac)
                
                # Add to netbox_macs dictionary
                if iface.name not in netbox_macs:
                    netbox_macs[iface.name] = set()
                netbox_macs[iface.name].add(mac_str)
        
        self.logger.debug(f"Extracted {len(all_netbox_macs)} MAC addresses from {len(all_netbox_interfaces)} interfaces")

@jseifeddine commented on GitHub (Jul 20, 2025): A workaround, use the `dcim.interfaces` endpoint to get reliable mac address data ```python # Get interfaces self.logger.debug("Fetching existing NetBox interfaces...") all_netbox_interfaces = { iface.name: iface for iface in netbox_device.api.dcim.interfaces.filter(device_id=netbox_device.id) } self.logger.debug(f"Found {len(all_netbox_interfaces)} existing interfaces in NetBox") # !!! as there is a bug in netbox, we can't use the mac_addresses endpoint !!! # # Get ALL existing MAC addresses for this device with their interface assignments # self.logger.debug("Fetching existing MAC addresses...") # netbox_macs = {} # interface_name -> set of MAC addresses # all_netbox_macs = list(netbox_device.api.dcim.mac_addresses.filter(device_id=netbox_device.id)) # self.logger.debug(f"Found {len(all_netbox_macs)} existing MAC addresses in NetBox") # for mac in all_netbox_macs: # if mac.assigned_object: # iface_name = mac.assigned_object.name # if iface_name not in netbox_macs: # netbox_macs[iface_name] = set() # mac_str = self.normalize_mac_address(mac.mac_address) # netbox_macs[iface_name].add(mac_str) # Get MAC addresses from interfaces instead of using the MAC address endpoint self.logger.debug("Extracting MAC addresses from interfaces...") netbox_macs = {} # interface_name -> set of MAC addresses all_netbox_macs = [] # List of mock MAC address objects # Create a mock MAC address object with the necessary attributes class MockMacAddress: def __init__(self, mac_id, mac_address, interface_id, interface_name): self.id = mac_id self.mac_address = mac_address self.assigned_object = type('obj', (), {'id': interface_id, 'name': interface_name}) # Create mock MAC address objects from interface data for iface in all_netbox_interfaces.values(): for mac in iface.mac_addresses: mac_str = self.normalize_mac_address(mac.mac_address) # Create mock MAC address object mock_mac = MockMacAddress(mac.id, mac_str, iface.id, iface.name) all_netbox_macs.append(mock_mac) # Add to netbox_macs dictionary if iface.name not in netbox_macs: netbox_macs[iface.name] = set() netbox_macs[iface.name].add(mac_str) self.logger.debug(f"Extracted {len(all_netbox_macs)} MAC addresses from {len(all_netbox_interfaces)} interfaces") ```

adam commented 2025-12-29 21:44:42 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 21, 2025):

Also, for completeness - I should also add - MAC Address changes are not logged against the interface, as one might expect., Since they appear under the interface the same way IP addresses do.

No MAC Address changes referenced on the interface, even though IP addresses are, surely this is an oversight ?

The only reference is Primary MAC Address updates (ID) on the interface itself

@jseifeddine commented on GitHub (Jul 21, 2025): Also, for completeness - I should also add - MAC Address changes are not logged against the interface, as one might expect., Since they appear under the interface the same way IP addresses do. <img width="1621" height="679" alt="Image" src="https://github.com/user-attachments/assets/a0762176-2f3b-4e7a-bf2c-348fd109da51" /> No MAC Address changes referenced on the interface, even though IP addresses are, surely this is an oversight ? <img width="1208" height="384" alt="Image" src="https://github.com/user-attachments/assets/87cfdf45-02f9-445e-96b7-257f31a0bc9a" /> The only reference is Primary MAC Address updates (ID) on the interface itself <img width="949" height="409" alt="Image" src="https://github.com/user-attachments/assets/b334e54c-edb3-406d-9c35-0d66bd9d29c4" />

adam commented 2025-12-29 21:44:42 +01:00

Author

Owner

Copy Link

@DanSheps commented on GitHub (Jul 21, 2025):

As per the note above "repoduction steps", please provide reproduction steps using raw queries (nbshell python api, graphql, openapi, etc):

If reporting a bug in the REST API, be sure to reconstruct the raw HTTP request(s) being made: Don't rely on a client library such as pynetbox.

@DanSheps commented on GitHub (Jul 21, 2025): As per the note above "repoduction steps", please provide reproduction steps using raw queries (nbshell python api, graphql, openapi, etc): > If reporting a bug in the REST API, be sure to reconstruct the raw HTTP request(s) being made: Don't rely on a client library such as pynetbox.

adam commented 2025-12-29 21:44:43 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 21, 2025):

Similar results RAW - I did already test that, sorry should of provided as well.

venv➜ netbox-ipam-bug (main) ✗ ./test-ipam-raw.py
Found 1000 interfaces in NetBox for device 1
Found 2000 IP addresses in NetBox for device 1
Found 1000 MAC addresses in NetBox for device 1
Found 1000 interfaces with 2000 IP addresses
Found 992 interfaces with 1000 MAC addresses

Each interface should have exactly two addresses
- 172.17.0.1/32
- ff1d:c7c:7b44:d39d:ab3d:6fde:f46a:4648/64

Each interface should have exactly one MAC address
- 18:2A:D3:65:90:2E


BUG: Interface dummy0 has 9 MAC addresses
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0
MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0


Test complete

find all the related code here;
ae5465f70f

add requests logging
146d3cf4bb

all_netbox_interfaces = nb.get(f"dcim/interfaces/?device_id=1")
all_netbox_ips = nb.get(f"ipam/ip-addresses/?device_id=1")
if is_version_above(netbox_version, "4.2.0"):
    all_netbox_macs = nb.get(f"dcim/mac-addresses/?device_id=1")

@jseifeddine commented on GitHub (Jul 21, 2025): Similar results `RAW` - I did already test that, sorry should of provided as well. ```bash venv➜ netbox-ipam-bug (main) ✗ ./test-ipam-raw.py Found 1000 interfaces in NetBox for device 1 Found 2000 IP addresses in NetBox for device 1 Found 1000 MAC addresses in NetBox for device 1 Found 1000 interfaces with 2000 IP addresses Found 992 interfaces with 1000 MAC addresses Each interface should have exactly two addresses - 172.17.0.1/32 - ff1d:c7c:7b44:d39d:ab3d:6fde:f46a:4648/64 Each interface should have exactly one MAC address - 18:2A:D3:65:90:2E BUG: Interface dummy0 has 9 MAC addresses MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 MAC 18:2A:D3:65:90:2E (ID: 1) is assigned to dummy0 Test complete ``` find all the related code here; https://github.com/jseifeddine/netbox-ipam-bug/commit/ae5465f70f5416b492763de8cb0ec0d553e5746d add requests logging https://github.com/jseifeddine/netbox-ipam-bug/commit/146d3cf4bbdefb57c220eb1b4c0adacc98d7734c ```python all_netbox_interfaces = nb.get(f"dcim/interfaces/?device_id=1") all_netbox_ips = nb.get(f"ipam/ip-addresses/?device_id=1") if is_version_above(netbox_version, "4.2.0"): all_netbox_macs = nb.get(f"dcim/mac-addresses/?device_id=1") ```

adam commented 2025-12-29 21:44:43 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 21, 2025):

test-macs-raw.log
https://gist.github.com/jseifeddine/2077fde582b7079aa4175567f6cb3678

test-ipam-raw.log
https://gist.github.com/jseifeddine/9724f9f6cc957e8e59ad596a3fb739e0

You will find the results at the bottom of each gist - the results from the requests made...

Everything required to replicate this BUG, with RAW requests or with pynetbox is in https://github.com/jseifeddine/netbox-ipam-bug

I've added request logging, which you can see when running:

./test-ipam-raw.py or ./test-macs-raw.py

@jseifeddine commented on GitHub (Jul 21, 2025): `test-macs-raw.log` https://gist.github.com/jseifeddine/2077fde582b7079aa4175567f6cb3678 `test-ipam-raw.log` https://gist.github.com/jseifeddine/9724f9f6cc957e8e59ad596a3fb739e0 You will find the results at the bottom of each gist - the results from the requests made... Everything required to replicate this BUG, with RAW requests or with pynetbox is in https://github.com/jseifeddine/netbox-ipam-bug I've added request logging, which you can see when running: `./test-ipam-raw.py` or `./test-macs-raw.py`

adam commented 2025-12-29 21:44:43 +01:00

Author

Owner

Copy Link

@DanSheps commented on GitHub (Jul 21, 2025):

@jseifeddine You are still not providing recreating steps for our team to follow. Please properly document step-by-step recreation steps for us to follow.

@DanSheps commented on GitHub (Jul 21, 2025): @jseifeddine You are still not providing recreating steps for our team to follow. Please properly document step-by-step recreation steps for us to follow.

adam commented 2025-12-29 21:44:43 +01:00

Author

Owner

Copy Link

@jeremystretch commented on GitHub (Jul 21, 2025):

Please also ensure your reproduction steps do not require the use of pynetbox in any way. This ensures that we can isolate the API client as a potential source of unexpected behavior.

@jeremystretch commented on GitHub (Jul 21, 2025): Please also ensure your reproduction steps do not require the use of pynetbox in any way. This ensures that we can isolate the API client as a potential source of unexpected behavior.

adam commented 2025-12-29 21:44:44 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 22, 2025):

Thanks @DanSheps and @jeremystretch

I have provided everything that you have requested and more - ready to easily setup, run and reproduce - in the repository I've attached: https://github.com/jseifeddine/netbox-ipam-bug

This bug I've identified seems to only happen in certain conditions (lots of interfaces, exact same MAC addresses on multiple interfaces) - So, I set up a docker stack so one could easily reproduce the bug.

The repository simply;

• Initializes a docker stack with the NETBOX_VERSION of your choosing; in the bug report I have specified v4.3.4
• Creates an admin token and populates .env
• Using the newly created NetBox stack and .env for authentication, runs a insert_dummy_data.py script that will create, Dummy Site, Dummy Device Role, Dummy Manufacturer, Dummy Device Type, Dummy Device, 10,000 Dummy Interfaces, 20,000 IP Addresses assigned to those interfaces (2 distinct IPs each interface, interfaces having the same IP addresses), 10,000 MAC Addresses (interfaces have the same MAC Address)
• Runs tests
  -- test-ipam.py <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with pynetbox)
  -- test-ipam-raw.py <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with RAW API requests)
  -- test-macs.py <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with pynetbox)
  -- test-macs-raw.py <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with RAW API requests)

I've already run these tests, providing the RAW API scripts, and logs;

These logs show the RAW API requests (NOT pynetbox) and the result of testing, which exposes the BUG I am submitting.

test-macs-raw.log https://gist.github.com/jseifeddine/2077fde582b7079aa4175567f6cb3678

test-ipam-raw.log https://gist.github.com/jseifeddine/9724f9f6cc957e8e59ad596a3fb739e0

As result of the following (easily reproducible

#Requires docker, docker compose, curl, python3 etc. check the repo
git clone https://github.com/jseifeddine/netbox-ipam-bug
cd netbox-ipam-bug
# Initializes NetBox docker stack, creates API token, populates data, runs tests (using pynetbox)
NETBOX_VERSION=v4.3.4 ./initialize-and-test.sh #
# after the initialization command above, run the following for testing using RAW API requests
./test-macs-raw.py # <---- NOT pynetbox
./test-ipam-raw.py # <---- NOT pynetbox

Thanks

@jseifeddine commented on GitHub (Jul 22, 2025): Thanks @DanSheps and @jeremystretch I have provided everything that you have requested and more - ready to easily setup, run and reproduce - in the repository I've attached: https://github.com/jseifeddine/netbox-ipam-bug This bug I've identified seems to only happen in certain conditions (lots of interfaces, exact same MAC addresses on multiple interfaces) - So, I set up a docker stack so one could easily reproduce the bug. The repository simply; - Initializes a docker stack with the `NETBOX_VERSION` of your choosing; in the bug report I have specified `v4.3.4` - Creates an admin token and populates `.env` - Using the newly created NetBox stack and `.env` for authentication, runs a `insert_dummy_data.py` script that will create, Dummy Site, Dummy Device Role, Dummy Manufacturer, Dummy Device Type, Dummy Device, 10,000 Dummy Interfaces, 20,000 IP Addresses assigned to those interfaces (2 distinct IPs each interface, interfaces having the same IP addresses), 10,000 MAC Addresses (interfaces have the same MAC Address) - Runs tests -- `test-ipam.py` <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with `pynetbox`) -- `test-ipam-raw.py` <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with RAW API `requests`) -- `test-macs.py` <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with `pynetbox`) -- `test-macs-raw.py` <-- tests IPAM and MAC Address bug (Clearly identifying mac address bug with RAW API `requests`) I've already run these tests, providing the RAW API scripts, and logs; These logs show the RAW API `requests` (**NOT** `pynetbox`) and the result of testing, which exposes the `BUG` I am submitting. `test-macs-raw.log` https://gist.github.com/jseifeddine/2077fde582b7079aa4175567f6cb3678 `test-ipam-raw.log` https://gist.github.com/jseifeddine/9724f9f6cc957e8e59ad596a3fb739e0 As result of the following (easily reproducible ```bash #Requires docker, docker compose, curl, python3 etc. check the repo git clone https://github.com/jseifeddine/netbox-ipam-bug cd netbox-ipam-bug # Initializes NetBox docker stack, creates API token, populates data, runs tests (using pynetbox) NETBOX_VERSION=v4.3.4 ./initialize-and-test.sh # # after the initialization command above, run the following for testing using RAW API requests ./test-macs-raw.py # <---- NOT pynetbox ./test-ipam-raw.py # <---- NOT pynetbox ``` Thanks

adam commented 2025-12-29 21:44:44 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 22, 2025):

If you are unable to view the repository I have made, please find the below

import requests
import json
from typing import Dict, List, Optional, Any


class DotDict:
    """Dictionary subclass that allows attribute access to dictionary keys"""
    
    def __init__(self, data):
        """
        Initialize DotDict with dictionary data
        
        Args:
            data: Dictionary to convert to DotDict
        """
        for key, value in data.items():
            # Convert nested dictionaries and lists of dictionaries
            if isinstance(value, dict):
                value = DotDict(value)
            elif isinstance(value, list) and value and isinstance(value[0], dict):
                value = [DotDict(item) if isinstance(item, dict) else item for item in value]
            
            # Replace hyphens with underscores in key names for attribute access
            key = key.replace('-', '_') if isinstance(key, str) else key
            setattr(self, key, value)
    
    def __getitem__(self, key):
        return getattr(self, key)
    
    def __contains__(self, key):
        return hasattr(self, key)


class NetBoxAPI:
    """Class to interact directly with NetBox API without using pynetbox"""
    
    def __init__(self, url: str, token: str, verify_ssl: bool = True):
        """
        Initialize NetBox API client
        
        Args:
            url: NetBox API URL
            token: NetBox API token
            verify_ssl: Whether to verify SSL certificates
        """
        self.url = url.rstrip('/')
        self.token = token
        self.verify_ssl = verify_ssl
        self.headers = {
            'Authorization': f'Token {token}',
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        }
        
    def get(self, endpoint: str, params: Optional[Dict] = None) -> Dict:
        """
        Make a request to the NetBox API
        
        Args:
            endpoint: API endpoint (e.g. 'ipam/ip-addresses/')
            params: Query parameters
            
        Returns:
            API response as dictionary or list of DotDict objects
        """
        # Check if URL already ends with /api to avoid duplicate
        if self.url.endswith('/api'):
            url = f"{self.url}/{endpoint}"
        else:
            url = f"{self.url}/api/{endpoint}"
            
        # Initialize results and set up pagination parameters
        if params is None:
            params = {}
        
        results = []
        next_url = url
        
        # Follow pagination by getting all pages
        while next_url:
            response = requests.get(next_url, headers=self.headers, params=params, verify=self.verify_ssl)
            response.raise_for_status()
            data = response.json()
            
            # Add results from this page
            if 'results' in data:
                # Convert each result to DotDict for attribute access
                results.extend([DotDict(item) for item in data['results']])
                next_url = data.get('next')
                # Clear params after first request as they're included in the next URL
                params = {}
            else:
                # If no pagination, just return the data as DotDict
                return DotDict(data)
                
        # Return compiled results as list of DotDict objects
        return results
        
    def status(self) -> Dict:
        """
        Get NetBox status information
        
        Returns:
            Status information as a DotDict
        """
        if self.url.endswith('/api'):
            url = f"{self.url}/status/"
        else:
            url = f"{self.url}/api/status/"
            
        response = requests.get(url, headers=self.headers, verify=self.verify_ssl)
        response.raise_for_status()
        return DotDict(response.json())


nb = NetBoxAPI(
    url=os.getenv("NETBOX_URL"),
    token=os.getenv("NETBOX_TOKEN"),
)

DEVICE_ID = 1

all_netbox_macs = list(nb.get(f"dcim/mac-addresses/?device_id={DEVICE_ID}"))
all_netbox_interfaces = list(nb.get(f"dcim/interfaces/?device_id={DEVICE_ID}"))

print(f"Found {len(all_netbox_macs)} MAC addresses in NetBox for device {DEVICE_ID} from dcim.mac_addresses endpoint.")
print(f"Found {len(all_netbox_interfaces)} interfaces in NetBox for device {DEVICE_ID} from dcim.interfaces endpoint.")

mac_interfaces = {}
interface_macs = {}

for mac in all_netbox_macs:
    if mac.assigned_object and mac.assigned_object.name:
        if not mac_interfaces.get(mac.assigned_object.name):
            mac_interfaces[mac.assigned_object.name] = []
        mac_interfaces[mac.assigned_object.name].append(mac)

total_macs = sum(len(macs) for macs in mac_interfaces.values())
print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces with from dcim.mac_addresses endpoint.")

for interface in all_netbox_interfaces:
    if interface.mac_addresses:
        if not interface_macs.get(interface.name):
            interface_macs[interface.name] = []
        for mac in interface.mac_addresses:
            interface_macs[interface.name].append(mac)

total_macs_from_interface = sum(len(macs) for macs in interface_macs.values())
print(f"Found {total_macs_from_interface} MAC addresses on {len(interface_macs)} interfaces from dcim.interfaces endpoint.")

# Expose bug in mac / interface counts returned from different endpoints

# total_macs is the total number of MAC addresses on the device from the dcim.mac_addresses endpoint
# total_macs_from_interface is the total number of MAC addresses on all interfaces from the dcim.interfaces endpoint
if total_macs != total_macs_from_interface:
    print(f"BUG: Total MAC addresses from dcim.mac_addresses endpoint ({total_macs}) does not match total MAC addresses from dcim.interfaces endpoint ({total_macs_from_interface})")

# mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint
# interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint
if len(mac_interfaces) != len(interface_macs):
    print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})")

Requests logs and result:

--- NetBox API Request (Page 1) ---
URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1
Method: GET
Headers: {
  "Authorization": "[REDACTED]",
  "Content-Type": "application/json",
  "Accept": "application/json"
}
Params: {}

--- NetBox API Response (Page 1) ---
Status Code: 200
Response Time: 0.286 seconds
Response Size: 35054 bytes
Results: 50 items (Page 1, Total: 1000)
Next Page: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=50

--- NetBox API Request (Page 2) ---
URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=50
Method: GET
Headers: {
  "Authorization": "[REDACTED]",
  "Content-Type": "application/json",
  "Accept": "application/json"
}
Params: {}

--- NetBox API Response (Page 2) ---
Status Code: 200
Response Time: 0.163 seconds
Response Size: 35191 bytes
Results: 50 items (Page 2, Total: 1000)
Next Page: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=100

--- NetBox API Request (Page 3) ---
URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=100
Method: GET
Headers: {
  "Authorization": "[REDACTED]",
  "Content-Type": "application/json",
  "Accept": "application/json"
}
Params: {}
...

...
--- NetBox API Request (Page 20) ---
URL: http://localhost:8080/api/dcim/interfaces/?device_id=1&limit=50&offset=950
Method: GET
Headers: {
  "Authorization": "[REDACTED]",
  "Content-Type": "application/json",
  "Accept": "application/json"
}
Params: {}

--- NetBox API Response (Page 20) ---
Status Code: 200
Response Time: 0.077 seconds
Response Size: 78783 bytes
Results: 50 items (Page 20, Total: 1000)

Completed API requests: 20 page(s), 1000 total items retrieved
Found 1000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint.
Found 1000 interfaces in NetBox for device 1 from dcim.interfaces endpoint.
Found 1000 MAC addresses on 992 interfaces with from dcim.mac_addresses endpoint.
Found 1000 MAC addresses on 1000 interfaces from dcim.interfaces endpoint.
BUG: Total interfaces from dcim.mac_addresses endpoint (992) does not match total interfaces from dcim.interfaces endpoint (1000)

Test complete

@jseifeddine commented on GitHub (Jul 22, 2025): If you are unable to view the repository I have made, please find the below ```python import requests import json from typing import Dict, List, Optional, Any class DotDict: """Dictionary subclass that allows attribute access to dictionary keys""" def __init__(self, data): """ Initialize DotDict with dictionary data Args: data: Dictionary to convert to DotDict """ for key, value in data.items(): # Convert nested dictionaries and lists of dictionaries if isinstance(value, dict): value = DotDict(value) elif isinstance(value, list) and value and isinstance(value[0], dict): value = [DotDict(item) if isinstance(item, dict) else item for item in value] # Replace hyphens with underscores in key names for attribute access key = key.replace('-', '_') if isinstance(key, str) else key setattr(self, key, value) def __getitem__(self, key): return getattr(self, key) def __contains__(self, key): return hasattr(self, key) class NetBoxAPI: """Class to interact directly with NetBox API without using pynetbox""" def __init__(self, url: str, token: str, verify_ssl: bool = True): """ Initialize NetBox API client Args: url: NetBox API URL token: NetBox API token verify_ssl: Whether to verify SSL certificates """ self.url = url.rstrip('/') self.token = token self.verify_ssl = verify_ssl self.headers = { 'Authorization': f'Token {token}', 'Content-Type': 'application/json', 'Accept': 'application/json' } def get(self, endpoint: str, params: Optional[Dict] = None) -> Dict: """ Make a request to the NetBox API Args: endpoint: API endpoint (e.g. 'ipam/ip-addresses/') params: Query parameters Returns: API response as dictionary or list of DotDict objects """ # Check if URL already ends with /api to avoid duplicate if self.url.endswith('/api'): url = f"{self.url}/{endpoint}" else: url = f"{self.url}/api/{endpoint}" # Initialize results and set up pagination parameters if params is None: params = {} results = [] next_url = url # Follow pagination by getting all pages while next_url: response = requests.get(next_url, headers=self.headers, params=params, verify=self.verify_ssl) response.raise_for_status() data = response.json() # Add results from this page if 'results' in data: # Convert each result to DotDict for attribute access results.extend([DotDict(item) for item in data['results']]) next_url = data.get('next') # Clear params after first request as they're included in the next URL params = {} else: # If no pagination, just return the data as DotDict return DotDict(data) # Return compiled results as list of DotDict objects return results def status(self) -> Dict: """ Get NetBox status information Returns: Status information as a DotDict """ if self.url.endswith('/api'): url = f"{self.url}/status/" else: url = f"{self.url}/api/status/" response = requests.get(url, headers=self.headers, verify=self.verify_ssl) response.raise_for_status() return DotDict(response.json()) nb = NetBoxAPI( url=os.getenv("NETBOX_URL"), token=os.getenv("NETBOX_TOKEN"), ) DEVICE_ID = 1 all_netbox_macs = list(nb.get(f"dcim/mac-addresses/?device_id={DEVICE_ID}")) all_netbox_interfaces = list(nb.get(f"dcim/interfaces/?device_id={DEVICE_ID}")) print(f"Found {len(all_netbox_macs)} MAC addresses in NetBox for device {DEVICE_ID} from dcim.mac_addresses endpoint.") print(f"Found {len(all_netbox_interfaces)} interfaces in NetBox for device {DEVICE_ID} from dcim.interfaces endpoint.") mac_interfaces = {} interface_macs = {} for mac in all_netbox_macs: if mac.assigned_object and mac.assigned_object.name: if not mac_interfaces.get(mac.assigned_object.name): mac_interfaces[mac.assigned_object.name] = [] mac_interfaces[mac.assigned_object.name].append(mac) total_macs = sum(len(macs) for macs in mac_interfaces.values()) print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces with from dcim.mac_addresses endpoint.") for interface in all_netbox_interfaces: if interface.mac_addresses: if not interface_macs.get(interface.name): interface_macs[interface.name] = [] for mac in interface.mac_addresses: interface_macs[interface.name].append(mac) total_macs_from_interface = sum(len(macs) for macs in interface_macs.values()) print(f"Found {total_macs_from_interface} MAC addresses on {len(interface_macs)} interfaces from dcim.interfaces endpoint.") # Expose bug in mac / interface counts returned from different endpoints # total_macs is the total number of MAC addresses on the device from the dcim.mac_addresses endpoint # total_macs_from_interface is the total number of MAC addresses on all interfaces from the dcim.interfaces endpoint if total_macs != total_macs_from_interface: print(f"BUG: Total MAC addresses from dcim.mac_addresses endpoint ({total_macs}) does not match total MAC addresses from dcim.interfaces endpoint ({total_macs_from_interface})") # mac_interfaces[mac.assigned_object.name] -> list of MAC addresses from the dcim.mac_addresses endpoint # interface_macs[interface.name] -> list of MAC addresses from the dcim.interfaces endpoint if len(mac_interfaces) != len(interface_macs): print(f"BUG: Total interfaces from dcim.mac_addresses endpoint ({len(mac_interfaces)}) does not match total interfaces from dcim.interfaces endpoint ({len(interface_macs)})") ``` Requests logs and result: ```bash --- NetBox API Request (Page 1) --- URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1 Method: GET Headers: { "Authorization": "[REDACTED]", "Content-Type": "application/json", "Accept": "application/json" } Params: {} --- NetBox API Response (Page 1) --- Status Code: 200 Response Time: 0.286 seconds Response Size: 35054 bytes Results: 50 items (Page 1, Total: 1000) Next Page: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=50 --- NetBox API Request (Page 2) --- URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=50 Method: GET Headers: { "Authorization": "[REDACTED]", "Content-Type": "application/json", "Accept": "application/json" } Params: {} --- NetBox API Response (Page 2) --- Status Code: 200 Response Time: 0.163 seconds Response Size: 35191 bytes Results: 50 items (Page 2, Total: 1000) Next Page: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=100 --- NetBox API Request (Page 3) --- URL: http://localhost:8080/api/dcim/mac-addresses/?device_id=1&limit=50&offset=100 Method: GET Headers: { "Authorization": "[REDACTED]", "Content-Type": "application/json", "Accept": "application/json" } Params: {} ... ... --- NetBox API Request (Page 20) --- URL: http://localhost:8080/api/dcim/interfaces/?device_id=1&limit=50&offset=950 Method: GET Headers: { "Authorization": "[REDACTED]", "Content-Type": "application/json", "Accept": "application/json" } Params: {} --- NetBox API Response (Page 20) --- Status Code: 200 Response Time: 0.077 seconds Response Size: 78783 bytes Results: 50 items (Page 20, Total: 1000) Completed API requests: 20 page(s), 1000 total items retrieved Found 1000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint. Found 1000 interfaces in NetBox for device 1 from dcim.interfaces endpoint. Found 1000 MAC addresses on 992 interfaces with from dcim.mac_addresses endpoint. Found 1000 MAC addresses on 1000 interfaces from dcim.interfaces endpoint. BUG: Total interfaces from dcim.mac_addresses endpoint (992) does not match total interfaces from dcim.interfaces endpoint (1000) Test complete ```

adam commented 2025-12-29 21:44:44 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 22, 2025):

In summary

lots of interfaces, all with the same mac address - assigned to them.

Compare data from http://localhost:8080/api/dcim/mac-addresses/?device_id=1 and http://localhost:8080/api/dcim/interfaces/?device_id=1

BUG: Total interfaces(assigned_object) from dcim.mac_addresses endpoint (992) does not match total interfaces from dcim.interfaces endpoint (1000)

@jseifeddine commented on GitHub (Jul 22, 2025): In summary lots of interfaces, all with the same mac address - assigned to them. Compare data from `http://localhost:8080/api/dcim/mac-addresses/?device_id=1` and `http://localhost:8080/api/dcim/interfaces/?device_id=1` BUG: Total interfaces(assigned_object) from dcim.mac_addresses endpoint (992) does not match total interfaces from dcim.interfaces endpoint (1000)

adam commented 2025-12-29 21:44:45 +01:00

Author

Owner

Copy Link

@DanSheps commented on GitHub (Jul 24, 2025):

Hello @jseifeddine

I am unable to reproduce this. Here is what I have done:

1. Create a device ('Test-MacDevice')
2. Delete all existing interfaces (just to be sure) for that device
3. Create 1000 Interfaces
4. Create 1000 Mac Addresses Assigned to those interfaces (1:1)
5. Run a query against the interface API endpoint: interfaces = requests.get('http://127.0.0.1:8000/api/dcim/interfaces/', headers={'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}, params={'device_id': 565, 'limit':1001}).json().get('results')
6. Run a query against the mac address API endpoint: macs = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers={'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}, params={'device_id': 565, 'limit':1001}).json().get('results')
7. Count the mac addresses: len(macs) which gives me 1000
8. Count the interface mac addresses:

interface_macs = []
for interface in interfaces:
    interface_macs.extend(interface.get('mac_addresses'))
len(interface_macs)

Unfortunately, this is all consistent with what is expected.

In the future, please provide fully reproducable steps without the use of scripts and only the raw API queries required.

@DanSheps commented on GitHub (Jul 24, 2025): Hello @jseifeddine I am unable to reproduce this. Here is what I have done: 1. Create a device ('Test-MacDevice') 2. Delete all existing interfaces (just to be sure) for that device 3. Create 1000 Interfaces 4. Create 1000 Mac Addresses Assigned to those interfaces (1:1) 5. Run a query against the interface API endpoint: `interfaces = requests.get('http://127.0.0.1:8000/api/dcim/interfaces/', headers={'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}, params={'device_id': 565, 'limit':1001}).json().get('results')` 6. Run a query against the mac address API endpoint: `macs = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers={'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}, params={'device_id': 565, 'limit':1001}).json().get('results')` 7. Count the mac addresses: `len(macs)` which gives me 1000 8. Count the interface mac addresses: ``` interface_macs = [] for interface in interfaces: interface_macs.extend(interface.get('mac_addresses')) len(interface_macs) ``` Unfortunately, this is all consistent with what is expected. In the future, please provide fully reproducable steps without the use of scripts and only the raw API queries required.

adam commented 2025-12-29 21:44:45 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 25, 2025):

Thank you @DanSheps - I understand where you are coming from - however you didn't perform the steps to reproduce exactly, I think it all comes down to pagination.

You checked the interfaces' macs length < -- this was never in question, this is actually how I've worked around this bug
The len(macs) was also always showing correctly. Not in question.

What is wrong: The count of the macs 'assigned_object'

Try this, (PS. it's only reproducible when paginating - perhaps a vital clue to the issue)

#!/usr/bin/env python3

import requests

headers = {'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}

macs = []

macs_response = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers=headers, params={'device_id': 565, 'limit':100}).json()
macs.extend(macs_response.get('results')) # add first 100 macs to list

while macs_response.get('next'):
    print(macs_response.get('next'))
    macs_response = requests.get(macs_response.get('next'), headers=headers).json()
    macs.extend(macs_response.get('results')) # add next 100 macs to list

mac_interfaces = {}

for mac in macs:
    if mac.get('assigned_object') and mac.get('assigned_object').get('name'):
        if not mac_interfaces.get(mac.get('assigned_object').get('name')):
            mac_interfaces[mac.get('assigned_object').get('name')] = []
        mac_interfaces[mac.get('assigned_object').get('name')].append(mac)

total_macs = sum(len(macs) for macs in mac_interfaces.values())
print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces (macs assigned_object) from dcim.mac_addresses endpoint.")

Results:

./mac-test.py
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=100
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=200
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=300
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=400
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=500
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=600
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=700
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=800
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=900
Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.

In case your not convinced, all interfaces have primary_mac_address;
so you can conclude that ALL mac addresses are assigned, having assigned_object.

Interface List

GET /api/dcim/interfaces/?primary_mac_address=null

HTTP 200 OK
Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE
Content-Type: application/json
Vary: Accept

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}


Interface List

GET /api/dcim/interfaces/?limit=1000

HTTP 200 OK
Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE
Content-Type: application/json
Vary: Accept

{
    "count": 1000,
    "next": null,
    "previous": null,
    "results": [

@jseifeddine commented on GitHub (Jul 25, 2025): Thank you @DanSheps - I understand where you are coming from - however you didn't perform the steps to reproduce exactly, I think it all comes down to pagination. You checked the interfaces' macs length < -- this was never in question, this is actually how I've worked around this bug The `len(macs)` was also always showing correctly. Not in question. **What is wrong: The count of the macs 'assigned_object'** Try this, (PS. it's only reproducible when paginating - perhaps a vital clue to the issue) ```python #!/usr/bin/env python3 import requests headers = {'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'} macs = [] macs_response = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers=headers, params={'device_id': 565, 'limit':100}).json() macs.extend(macs_response.get('results')) # add first 100 macs to list while macs_response.get('next'): print(macs_response.get('next')) macs_response = requests.get(macs_response.get('next'), headers=headers).json() macs.extend(macs_response.get('results')) # add next 100 macs to list mac_interfaces = {} for mac in macs: if mac.get('assigned_object') and mac.get('assigned_object').get('name'): if not mac_interfaces.get(mac.get('assigned_object').get('name')): mac_interfaces[mac.get('assigned_object').get('name')] = [] mac_interfaces[mac.get('assigned_object').get('name')].append(mac) total_macs = sum(len(macs) for macs in mac_interfaces.values()) print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces (macs assigned_object) from dcim.mac_addresses endpoint.") ``` Results: ```bash ./mac-test.py http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=100 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=200 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=300 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=400 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=500 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=600 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=700 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=800 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=100&offset=900 Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. ``` In case your not convinced, all interfaces have primary_mac_address; so you can conclude that ALL mac addresses are assigned, having assigned_object. ``` Interface List GET /api/dcim/interfaces/?primary_mac_address=null HTTP 200 OK Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE Content-Type: application/json Vary: Accept { "count": 0, "next": null, "previous": null, "results": [] } Interface List GET /api/dcim/interfaces/?limit=1000 HTTP 200 OK Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE Content-Type: application/json Vary: Accept { "count": 1000, "next": null, "previous": null, "results": [ ```

adam commented 2025-12-29 21:44:45 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 25, 2025):

As we've just discovered;
It isn't reproducible IF NOT paginating

headers = {'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'}

macs  = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers=headers, params={'device_id': 565, 'limit':1000}).json().get('results')

mac_interfaces = {}

for mac in macs:
    if mac.get('assigned_object') and mac.get('assigned_object').get('name'):
        if not mac_interfaces.get(mac.get('assigned_object').get('name')):
            mac_interfaces[mac.get('assigned_object').get('name')] = []
        mac_interfaces[mac.get('assigned_object').get('name')].append(mac)

total_macs = sum(len(macs) for macs in mac_interfaces.values())
print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces (macs assigned_object) from dcim.mac_addresses endpoint.")

Found 1000 MAC addresses on 1000 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.

AND, lowering the limit per page, exasperates the issue

http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=940
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=950
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=960
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=970
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=980
http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=990
Found 1000 MAC addresses on 952 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.

@jseifeddine commented on GitHub (Jul 25, 2025): As we've just discovered; It isn't reproducible IF NOT paginating ```python headers = {'Authorization': 'Token 908d25811c1cced6fe893a34ca598026e8276946', 'Content-Type': 'application/json', 'Accept': 'application/json'} macs = requests.get('http://127.0.0.1:8000/api/dcim/mac-addresses/', headers=headers, params={'device_id': 565, 'limit':1000}).json().get('results') mac_interfaces = {} for mac in macs: if mac.get('assigned_object') and mac.get('assigned_object').get('name'): if not mac_interfaces.get(mac.get('assigned_object').get('name')): mac_interfaces[mac.get('assigned_object').get('name')] = [] mac_interfaces[mac.get('assigned_object').get('name')].append(mac) total_macs = sum(len(macs) for macs in mac_interfaces.values()) print(f"Found {total_macs} MAC addresses on {len(mac_interfaces)} interfaces (macs assigned_object) from dcim.mac_addresses endpoint.") ``` ```bash Found 1000 MAC addresses on 1000 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. ``` AND, lowering the limit per page, exasperates the issue ```bash http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=940 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=950 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=960 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=970 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=980 http://127.0.0.1:8000/api/dcim/mac-addresses/?device_id=565&limit=10&offset=990 Found 1000 MAC addresses on 952 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. ```

adam commented 2025-12-29 21:44:46 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 25, 2025):

if you actually look at that mac_interfaces (derived from assigned_object , from mac addresses endpoint)

for interface in mac_interfaces:
    if len(mac_interfaces[interface]) != 1:
        print(f"Interface {interface} has {len(mac_interfaces[interface])} MAC addresses")
        for mac in mac_interfaces[interface]:
            print(interface, mac.get('mac_address'))

./mac-test.py
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900
Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.
Interface dummy0 has 4 MAC addresses
dummy0 18:2A:D3:65:90:2E
dummy0 18:2A:D3:65:90:2E
dummy0 18:2A:D3:65:90:2E
dummy0 18:2A:D3:65:90:2E

@jseifeddine commented on GitHub (Jul 25, 2025): if you actually look at that `mac_interfaces` (derived from assigned_object , from mac addresses endpoint) ```python for interface in mac_interfaces: if len(mac_interfaces[interface]) != 1: print(f"Interface {interface} has {len(mac_interfaces[interface])} MAC addresses") for mac in mac_interfaces[interface]: print(interface, mac.get('mac_address')) ``` ```bash ./mac-test.py http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900 Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. Interface dummy0 has 4 MAC addresses dummy0 18:2A:D3:65:90:2E dummy0 18:2A:D3:65:90:2E dummy0 18:2A:D3:65:90:2E dummy0 18:2A:D3:65:90:2E ```

adam commented 2025-12-29 21:44:46 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 25, 2025):

it's a simple problem, but it takes a little bit of scripting to reproduce it.
its the nature of the bug - i cannot give you one API request that will reproduce this bug.
You must follow the steps that I shared initially.

Thanks

@jseifeddine commented on GitHub (Jul 25, 2025): it's a simple problem, but it takes a little bit of scripting to reproduce it. its the nature of the bug - i cannot give you one API request that will reproduce this bug. You must follow the steps that I shared initially. Thanks

adam commented 2025-12-29 21:44:46 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 25, 2025):

Scenario recap:
1000 Interfaces
Each interface assigned the SAME MAC address (eg. sub-interfaces each sharing the same (parent's interface MAC) MAC address)

I've forked and fixed the code responsible for this bug. However, unsure if the changes I've made follows your guidelines and if it will cause other issues.

d7f3fe8b0a

My tests pass with commit;

./test.py
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900
Found 1000 MAC addresses on 1000 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.

with netbox main branch it get this;

./mac-test.py
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800
http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900
Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint.

As per your community guidelines:
Please note that our contribution policy requires that a feature request or bug report be approved and assigned prior to opening a pull request

I will await your acknowledgement and approval....

Some notes;

Looking at the implementation of the MACAddressViewSet in netbox/dcim/api/views.py, the issue is that it's not properly handling pagination with the GenericRelation for MAC addresses.

The problem is that when filtering MAC addresses by device_id, the API is not correctly handling the GenericRelation between interfaces and MAC addresses, causing duplicate MAC addresses to appear across pagination boundaries.

class MACAddressViewSet(NetBoxModelViewSet):
    queryset = MACAddress.objects.all().distinct()  # Add distinct() to prevent duplicates
    serializer_class = serializers.MACAddressSerializer
    filterset_class = filtersets.MACAddressFilterSet

Also, it may be a problem is in the filter_device method of the MACAddressFilterSet class in netbox/dcim/filtersets.py

The issue is that when filtering by device_id, the code gets all interfaces for the device, then filters MAC addresses by those interfaces. However, it doesn't account for the fact that the same MAC address can be associated with multiple interfaces through the GenericRelation, causing duplicates in pagination.

def filter_device(self, queryset, name, value):
    devices = Device.objects.filter(**{f'{name}__in': value})
    if not devices.exists():
        return queryset.none()
    interface_ids = []
    for device in devices:
        interface_ids.extend(device.vc_interfaces().values_list('id', flat=True))
    return queryset.filter(
        interface__in=interface_ids
    ).distinct()  # Add distinct() to prevent duplicates

The issue was that when paginating through the API results without .distinct(), some MAC addresses were being returned multiple times across different pages. Here's what was likely happening:

1. Exactly 1000 interfaces, each with one MAC address
2. When paginating through the results without .distinct(), some MAC addresses appeared in multiple pages
3. This meant that some MAC addresses were counted multiple times in the total count
4. However, when grouped by interface name, the duplicates were assigned to the same interface
5. Giving 1000 total MAC addresses (including duplicates), but only 997 unique interfaces

To summarize:
Without .distinct(): 1000 MAC addresses (with some duplicates) on 997 unique interfaces.
With .distinct(): 1000 unique MAC addresses on 1000 unique interfaces.
The fix implemented ensures that the API behaves correctly and returns each MAC address exactly once, regardless of pagination.

@jseifeddine commented on GitHub (Jul 25, 2025): Scenario recap: 1000 Interfaces Each interface assigned the **SAME** MAC address (eg. sub-interfaces each sharing the same (parent's interface MAC) MAC address) I've forked and fixed the code responsible for this bug. However, unsure if the changes I've made follows your guidelines and if it will cause other issues. https://github.com/jseifeddine/netbox/commit/d7f3fe8b0a0a2895dd09988df3dd60d47ec34a7e My tests pass with commit; ```bash ./test.py http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900 Found 1000 MAC addresses on 1000 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. ``` with netbox main branch it get this; ```bash ./mac-test.py http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=100 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=200 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=300 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=400 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=500 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=600 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=700 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=800 http://127.0.0.1:8080/api/dcim/mac-addresses/?device_id=1&limit=100&offset=900 Found 1000 MAC addresses on 997 interfaces (macs assigned_object) from dcim.mac_addresses endpoint. ``` As per your community guidelines: `Please note that our contribution policy requires that a feature request or bug report be approved and assigned prior to opening a pull request` I will await your acknowledgement and approval.... Some notes; Looking at the implementation of the MACAddressViewSet in `netbox/dcim/api/views.py`, the issue is that it's not properly handling pagination with the GenericRelation for MAC addresses. The problem is that when filtering MAC addresses by device_id, the API is not correctly handling the GenericRelation between interfaces and MAC addresses, causing duplicate MAC addresses to appear across pagination boundaries. ```python class MACAddressViewSet(NetBoxModelViewSet): queryset = MACAddress.objects.all().distinct() # Add distinct() to prevent duplicates serializer_class = serializers.MACAddressSerializer filterset_class = filtersets.MACAddressFilterSet ``` Also, it may be a problem is in the filter_device method of the MACAddressFilterSet class in `netbox/dcim/filtersets.py` The issue is that when filtering by device_id, the code gets all interfaces for the device, then filters MAC addresses by those interfaces. However, it doesn't account for the fact that the same MAC address can be associated with multiple interfaces through the GenericRelation, causing duplicates in pagination. ```python def filter_device(self, queryset, name, value): devices = Device.objects.filter(**{f'{name}__in': value}) if not devices.exists(): return queryset.none() interface_ids = [] for device in devices: interface_ids.extend(device.vc_interfaces().values_list('id', flat=True)) return queryset.filter( interface__in=interface_ids ).distinct() # Add distinct() to prevent duplicates ``` The issue was that when paginating through the API results without .distinct(), some MAC addresses were being returned multiple times across different pages. Here's what was likely happening: 1. Exactly 1000 interfaces, each with one MAC address 2. When paginating through the results without .distinct(), some MAC addresses appeared in multiple pages 3. This meant that some MAC addresses were counted multiple times in the total count 4. However, when grouped by interface name, the duplicates were assigned to the same interface 5. Giving 1000 total MAC addresses (including duplicates), but only 997 unique interfaces To summarize: Without .distinct(): 1000 MAC addresses (with some duplicates) on 997 unique interfaces. With .distinct(): 1000 unique MAC addresses on 1000 unique interfaces. The fix implemented ensures that the API behaves correctly and returns each MAC address exactly once, regardless of pagination.

adam commented 2025-12-29 21:44:46 +01:00

Author

Owner

Copy Link

@DanSheps commented on GitHub (Jul 25, 2025):

Thank you for the detailed information @jseifeddine

Could you write a test to test the filterset for this condition (perhaps we don't need to use 1000 interfaces/mac addresses, just enough to encounter this issue)

I will need to validate this on my end.

@DanSheps commented on GitHub (Jul 25, 2025): Thank you for the detailed information @jseifeddine Could you write a test to test the filterset for this condition (perhaps we don't need to use 1000 interfaces/mac addresses, just enough to encounter this issue) I will need to validate this on my end.

adam commented 2025-12-29 21:44:47 +01:00

Author

Owner

Copy Link

@jseifeddine commented on GitHub (Jul 26, 2025):

OK,

here is my best effort, the tests corroborate my initial findings.

It would seem that I can only replicate the issue in the API, which would make sense - because I guess (without testing) the netbox front end component was showing consistent data.

043dc4cc96

python manage.py test dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct -v 2

...
System check identified no issues (0 silenced).
test_device_filter_pagination_distinct (dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct)
Test that filtering MAC addresses by device_id returns distinct results. ... ok
test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct)
Test that filtering MAC addresses by device_id via API returns distinct results ... FAIL

======================================================================
FAIL: test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct)
Test that filtering MAC addresses by device_id via API returns distinct results
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/netbox/netbox/dcim/tests/test_api.py", line 2754, in test_device_filter_pagination_distinct
    self.assertEqual(len(duplicates), 0,
AssertionError: 1 != 0 : Found duplicate MAC address IDs between pages: {512}

----------------------------------------------------------------------
Ran 2 tests in 3.084s

FAILED (failures=1)

the .distinct() that I applied earlier was addressing the symptoms and not root cause, I wasn't satisfied as you could imagine, so I investigated further, inspecting the SQL queries just to be sure.

Python 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> # Import the models
>>> from dcim.models import MACAddress, Device, Interface
>>> 
>>> # Get a device ID to test with
>>> device = Device.objects.first()
ing with device ID: {device.pk}")

# Test the basic queryset that the filterset uses
qs = MACAddress.objects.filter(interface__device_id=device.pk)
print("Basic query:")
print(qs.query)
print(f"Count: {qs.count()}")

# Test what happens when we add distinct
qs_distinct = MACAddress.objects.filter(interface__device_id=device.pk).distinct()
print("\nWith distinct:")
print(qs_distinct.query)
print(f"Count: {qs_distinct.count()}")

# Check if there's a difference in counts
print(f"\nDifference in counts: {qs.count() - qs_distinct.count()}")

# Look at the actual filter_device method
from dcim.filtersets import MACAddressFilterSet
filterset = MACAddressFilterSet({'device_id': [device.pk]}, queryset=MACAddress.objects.all())
filtered_qs = filterset.qs
print(f"\nFilterSet query:")
print(filtered_qs.query)
print(f"FilterSet count: {filtered_qs.count()}")>>> print(f"Testing with device ID: {device.pk}")
Testing with device ID: 1
>>> 
>>> # Test the basic queryset that the filterset uses
>>> qs = MACAddress.objects.filter(interface__device_id=device.pk)
>>> print("Basic query:")
Basic query:
>>> print(qs.query)
SELECT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."device_id" = 1 ORDER BY "dcim_macaddress"."mac_address" ASC
>>> print(f"Count: {qs.count()}")
Count: 1000
>>> 
>>> # Test what happens when we add distinct
>>> qs_distinct = MACAddress.objects.filter(interface__device_id=device.pk).distinct()
>>> print("\nWith distinct:")

With distinct:
>>> print(qs_distinct.query)
SELECT DISTINCT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."device_id" = 1 ORDER BY "dcim_macaddress"."mac_address" ASC
>>> print(f"Count: {qs_distinct.count()}")
Count: 1000
>>> 
>>> # Check if there's a difference in counts
>>> print(f"\nDifference in counts: {qs.count() - qs_distinct.count()}")

Difference in counts: 0
>>> 
>>> # Look at the actual filter_device method
>>> from dcim.filtersets import MACAddressFilterSet
>>> filterset = MACAddressFilterSet({'device_id': [device.pk]}, queryset=MACAddress.objects.all())
>>> filtered_qs = filterset.qs
>>> print(f"\nFilterSet query:")

FilterSet query:
>>> print(filtered_qs.query)
SELECT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000) ORDER BY "dcim_macaddress"."mac_address" ASC
>>> print(f"FilterSet count: {filtered_qs.count()}")

Key Findings:
Basic query vs FilterSet query are DIFFERENT:
Basic query: interface__device_id = 1 (simple join)
FilterSet query: interface__id IN (1, 2, 3, ..., 1000) (massive IN clause)
Both return 1000 results - so the duplication isn't at the SQL level
The FilterSet is using the filter_device method which:
Gets all interfaces for the device (device.vc_interfaces())
Creates a massive IN clause with all interface IDs
This is less efficient than a simple join

When you have many MAC addresses with the same value (like 18:2A:D3:65:90:2E), the ordering becomes non-deterministic for records with identical mac_address values. This can cause the same records to appear on multiple pages during pagination.

Lo and behold, we aren't ordering by id as we probably should be?

netbox/dcim/models/devices.py

#
# Addressing
#

class MACAddress(PrimaryModel):
...
    class Meta:
        ordering = ('mac_address',)
...

Proposed Fix:

bf38821efc

Instead of just adding .distinct(), we should fix the ordering to be deterministic:

Tests are passing now:

System check identified no issues (0 silenced).
test_device_filter_pagination_distinct (dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct)
Test that filtering MAC addresses by device_id returns distinct results. ... ok
test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct)
Test that filtering MAC addresses by device_id via API returns distinct results ... ok

----------------------------------------------------------------------
Ran 2 tests in 8.329s

OK

Including my initial way of testing,

Found 1000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint.
Found 1000 interfaces in NetBox for device 1 from dcim.interfaces endpoint.
Found 1000 MAC addresses on 1000 interfaces with from dcim.mac_addresses endpoint.
Found 1000 MAC addresses on 1000 interfaces from dcim.interfaces endpoint.

Test complete

I apologize for jumping in here guns blazing - not following this communities practices.
We were talking past each other... and to make amends, I went the extra mile - hope that it is accepted as a fix for this issue.

This is all a learning curve for me.

Thanks everyone 👍

@jseifeddine commented on GitHub (Jul 26, 2025): OK, here is my best effort, the tests corroborate my initial findings. It would seem that I can only replicate the issue in the API, which would make sense - because I guess (without testing) the netbox front end component was showing consistent data. https://github.com/jseifeddine/netbox/commit/043dc4cc9644f46a74d342eff1e2211bc0f52ed2 ```bash python manage.py test dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct -v 2 ... System check identified no issues (0 silenced). test_device_filter_pagination_distinct (dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct) Test that filtering MAC addresses by device_id returns distinct results. ... ok test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct) Test that filtering MAC addresses by device_id via API returns distinct results ... FAIL ====================================================================== FAIL: test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct) Test that filtering MAC addresses by device_id via API returns distinct results ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/netbox/netbox/dcim/tests/test_api.py", line 2754, in test_device_filter_pagination_distinct self.assertEqual(len(duplicates), 0, AssertionError: 1 != 0 : Found duplicate MAC address IDs between pages: {512} ---------------------------------------------------------------------- Ran 2 tests in 3.084s FAILED (failures=1) ``` the `.distinct()` that I applied earlier was addressing the symptoms and not root cause, I wasn't satisfied as you could imagine, so I investigated further, inspecting the SQL queries just to be sure. ```python Python 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> # Import the models >>> from dcim.models import MACAddress, Device, Interface >>> >>> # Get a device ID to test with >>> device = Device.objects.first() ing with device ID: {device.pk}") # Test the basic queryset that the filterset uses qs = MACAddress.objects.filter(interface__device_id=device.pk) print("Basic query:") print(qs.query) print(f"Count: {qs.count()}") # Test what happens when we add distinct qs_distinct = MACAddress.objects.filter(interface__device_id=device.pk).distinct() print("\nWith distinct:") print(qs_distinct.query) print(f"Count: {qs_distinct.count()}") # Check if there's a difference in counts print(f"\nDifference in counts: {qs.count() - qs_distinct.count()}") # Look at the actual filter_device method from dcim.filtersets import MACAddressFilterSet filterset = MACAddressFilterSet({'device_id': [device.pk]}, queryset=MACAddress.objects.all()) filtered_qs = filterset.qs print(f"\nFilterSet query:") print(filtered_qs.query) print(f"FilterSet count: {filtered_qs.count()}")>>> print(f"Testing with device ID: {device.pk}") Testing with device ID: 1 >>> >>> # Test the basic queryset that the filterset uses >>> qs = MACAddress.objects.filter(interface__device_id=device.pk) >>> print("Basic query:") Basic query: >>> print(qs.query) SELECT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."device_id" = 1 ORDER BY "dcim_macaddress"."mac_address" ASC >>> print(f"Count: {qs.count()}") Count: 1000 >>> >>> # Test what happens when we add distinct >>> qs_distinct = MACAddress.objects.filter(interface__device_id=device.pk).distinct() >>> print("\nWith distinct:") With distinct: >>> print(qs_distinct.query) SELECT DISTINCT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."device_id" = 1 ORDER BY "dcim_macaddress"."mac_address" ASC >>> print(f"Count: {qs_distinct.count()}") Count: 1000 >>> >>> # Check if there's a difference in counts >>> print(f"\nDifference in counts: {qs.count() - qs_distinct.count()}") Difference in counts: 0 >>> >>> # Look at the actual filter_device method >>> from dcim.filtersets import MACAddressFilterSet >>> filterset = MACAddressFilterSet({'device_id': [device.pk]}, queryset=MACAddress.objects.all()) >>> filtered_qs = filterset.qs >>> print(f"\nFilterSet query:") FilterSet query: >>> print(filtered_qs.query) SELECT "dcim_macaddress"."id", "dcim_macaddress"."created", "dcim_macaddress"."last_updated", "dcim_macaddress"."custom_field_data", "dcim_macaddress"."description", "dcim_macaddress"."comments", "dcim_macaddress"."mac_address", "dcim_macaddress"."assigned_object_type_id", "dcim_macaddress"."assigned_object_id" FROM "dcim_macaddress" INNER JOIN "dcim_interface" ON ("dcim_macaddress"."assigned_object_id" = "dcim_interface"."id" AND ("dcim_macaddress"."assigned_object_type_id" = 9)) WHERE "dcim_interface"."id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000) ORDER BY "dcim_macaddress"."mac_address" ASC >>> print(f"FilterSet count: {filtered_qs.count()}") ``` Key Findings: Basic query vs FilterSet query are DIFFERENT: Basic query: interface__device_id = 1 (simple join) FilterSet query: interface__id IN (1, 2, 3, ..., 1000) (massive IN clause) Both return 1000 results - so the duplication isn't at the SQL level The FilterSet is using the filter_device method which: Gets all interfaces for the device (device.vc_interfaces()) Creates a massive IN clause with all interface IDs This is less efficient than a simple join When you have many MAC addresses with the same value (like 18:2A:D3:65:90:2E), the ordering becomes non-deterministic for records with identical mac_address values. This can cause the same records to appear on multiple pages during pagination. Lo and behold, we aren't ordering by `id` as we probably should be? `netbox/dcim/models/devices.py` ```python # # Addressing # class MACAddress(PrimaryModel): ... class Meta: ordering = ('mac_address',) ... ``` **Proposed Fix:** https://github.com/jseifeddine/netbox/commit/bf38821efc31021fd35eb047e671e0168d7993de Instead of just adding .distinct(), we should fix the ordering to be deterministic: Tests are passing now: ```bash System check identified no issues (0 silenced). test_device_filter_pagination_distinct (dcim.tests.test_filtersets.MACAddressTestCase.test_device_filter_pagination_distinct) Test that filtering MAC addresses by device_id returns distinct results. ... ok test_device_filter_pagination_distinct (dcim.tests.test_api.MACAddressTest.test_device_filter_pagination_distinct) Test that filtering MAC addresses by device_id via API returns distinct results ... ok ---------------------------------------------------------------------- Ran 2 tests in 8.329s OK ``` Including my initial way of testing, ```bash Found 1000 MAC addresses in NetBox for device 1 from dcim.mac_addresses endpoint. Found 1000 interfaces in NetBox for device 1 from dcim.interfaces endpoint. Found 1000 MAC addresses on 1000 interfaces with from dcim.mac_addresses endpoint. Found 1000 MAC addresses on 1000 interfaces from dcim.interfaces endpoint. Test complete ``` I apologize for jumping in here guns blazing - not following this communities practices. We were talking past each other... and to make amends, I went the extra mile - hope that it is accepted as a fix for this issue. This is all a learning curve for me. Thanks everyone 👍

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

update-changelog-comments-docs

feature-removal-issue-type

20911-dropdown

20239-plugin-menu-classes-mutable-state

21097-graphql-id-lookups

feature

fix_module_substitution

20923-dcim-templates

20044-elevation-stuck-lightmode

feature-ip-prefix-link

v4.5-beta1-release

20068-import-moduletype-attrs

20766-fix-german-translation-code-literals

20378-del-script

7604-filter-modifiers-v3

circuit-swap

12318-case-insensitive-uniqueness

20637-improve-device-q-filter

20660-script-load

19724-graphql

20614-update-ruff

14884-script

02496-max-page

19720-macaddress-interface-generic-relation

19408-circuit-terminations-export-templates

20203-openapi-check

fix-19669-api-image-download

7604-filter-modifiers

19275-fixes-interface-bulk-edit

fix-17794-get_field_value_return_list

11507-show-aggregate-and-rir-on-api

9583-add_column_specific_search_field_to_tables

v4.5.0

v4.4.10

v4.4.9

v4.5.0-beta1

v4.4.8

v4.4.7

v4.4.6

v4.4.5

v4.4.4

v4.4.3

v4.4.2

v4.4.1

v4.4.0

v4.3.7

v4.4.0-beta1

v4.3.6

v4.3.5

v4.3.4

v4.3.3

v4.3.2

v4.3.1

v4.3.0

v4.2.9

v4.3.0-beta2

v4.2.8

v4.3.0-beta1

v4.2.7

v4.2.6

v4.2.5

v4.2.4

v4.2.3

v4.2.2

v4.2.1

v4.2.0

v4.1.11

v4.1.10

v4.1.9

v4.1.8

v4.2-beta1

v4.1.7

v4.1.6

v4.1.5

v4.1.4

v4.1.3

v4.1.2

v4.1.1

v4.1.0

v4.0.11

v4.0.10

v4.0.9

v4.1-beta1

v4.0.8

v4.0.7

v4.0.6

v4.0.5

v4.0.3

v4.0.2

v4.0.1

v4.0.0

v3.7.8

v3.7.7

v4.0-beta2

v3.7.6

v3.7.5

v4.0-beta1

v3.7.4

v3.7.3

v3.7.2

v3.7.1

v3.7.0

v3.6.9

v3.6.8

v3.6.7

v3.7-beta1

v3.6.6

v3.6.5

v3.6.4

v3.6.3

v3.6.2

v3.6.1

v3.6.0

v3.5.9

v3.6-beta2

v3.5.8

v3.6-beta1

v3.5.7

v3.5.6

v3.5.5

v3.5.4

v3.5.3

v3.5.2

v3.5.1

v3.5.0

v3.4.10

v3.4.9

v3.5-beta2

v3.4.8

v3.5-beta1

v3.4.7

v3.4.6

v3.4.5

v3.4.4

v3.4.3

v3.4.2

v3.4.1

v3.4.0

v3.3.10

v3.3.9

v3.4-beta1

v3.3.8

v3.3.7

v3.3.6

v3.3.5

v3.3.4

v3.3.3

v3.3.2

v3.3.1

v3.3.0

v3.2.9

v3.2.8

v3.3-beta2

v3.2.7

v3.3-beta1

v3.2.6

v3.2.5

v3.2.4

v3.2.3

v3.2.2

v3.2.1

v3.2.0

v3.1.11

v3.1.10

v3.2-beta2

v3.1.9

v3.2-beta1

v3.1.8

v3.1.7

v3.1.6

v3.1.5

v3.1.4

v3.1.3

v3.1.2

v3.1.1

v3.1.0

v3.0.12

v3.0.11

v3.0.10

v3.1-beta1

v3.0.9

v3.0.8

v3.0.7

v3.0.6

v3.0.5

v3.0.4

v3.0.3

v3.0.2

v3.0.1

v3.0.0

v2.11.12

v3.0-beta2

v2.11.11

v2.11.10

v3.0-beta1

v2.11.9

v2.11.8

v2.11.7

v2.11.6

v2.11.5

v2.11.4

v2.11.3

v2.11.2

v2.11.1

v2.11.0

v2.10.10

v2.10.9

v2.11-beta1

v2.10.8

v2.10.7

v2.10.6

v2.10.5

v2.10.4

v2.10.3

v2.10.2

v2.10.1

v2.10.0

v2.9.11

v2.10-beta2

v2.9.10

v2.10-beta1

v2.9.9

v2.9.8

v2.9.7

v2.9.6

v2.9.5

v2.9.4

v2.9.3

v2.9.2

v2.9.1

v2.9.0

v2.9-beta2

v2.8.9

v2.9-beta1

v2.8.8

v2.8.7

v2.8.6

v2.8.5

v2.8.4

v2.8.3

v2.8.2

v2.8.1

v2.8.0

v2.7.12

v2.7.11

v2.7.10

v2.7.9

v2.7.8

v2.7.7

v2.7.6

v2.7.5

v2.7.4

v2.7.3

v2.7.2

v2.7.1

v2.7.0

v2.6.12

v2.6.11

v2.6.10

v2.6.9

v2.7-beta1

Solcon-2020-01-06

v2.6.8

v2.6.7

v2.6.6

v2.6.5

v2.6.4

v2.6.3

v2.6.2

v2.6.1

v2.6.0

v2.5.13

v2.5.12

v2.6-beta1

v2.5.11

v2.5.10

v2.5.9

v2.5.8

v2.5.7

v2.5.6

v2.5.5

v2.5.4

v2.5.3

v2.5.2

v2.5.1

v2.5.0

v2.4.9

v2.5-beta2

v2.4.8

v2.5-beta1

v2.4.7

v2.4.6

v2.4.5

v2.4.4

v2.4.3

v2.4.2

v2.4.1

v2.4.0

v2.3.7

v2.4-beta1

v2.3.6

v2.3.5

v2.3.4

v2.3.3

v2.3.2

v2.3.1

v2.3.0

v2.2.10

v2.3-beta2

v2.2.9

v2.3-beta1

v2.2.8

v2.2.7

v2.2.6

v2.2.5

v2.2.4

v2.2.3

v2.2.2

v2.2.1

v2.2.0

v2.1.6

v2.2-beta2

v2.1.5

v2.2-beta1

v2.1.4

v2.1.3

v2.1.2

v2.1.1

v2.1.0

v2.0.10

v2.1-beta1

v2.0.9

v2.0.8

v2.0.7

v2.0.6

v2.0.5

v2.0.4

v2.0.3

v2.0.2

v2.0.1

v2.0.0

v2.0-beta3

v1.9.6

v1.9.5

v2.0-beta2

v1.9.4-r1

v1.9.3

v2.0-beta1

v1.9.2

v1.9.1

v1.9.0-r1

v1.8.4

v1.8.3

v1.8.2

v1.8.1

v1.8.0

v1.7.3

v1.7.2-r1

v1.7.1

v1.7.0

v1.6.3

v1.6.2-r1

v1.6.1-r1

1.6.1

v1.6.0

v1.5.2

v1.5.1

v1.5.0

v1.4.2

v1.4.1

v1.4.0

v1.3.2

v1.3.1

v1.3.0

v1.2.2

v1.2.1

v1.2.0

v1.1.0

v1.0.7-r1

v1.0.7

v1.0.6

v1.0.5

v1.0.4

v1.0.3-r1

v1.0.3

1.0.0

Labels

Clear labels

beta

breaking change

complexity: high

complexity: low

complexity: medium

needs milestone

netbox

pending closure

plugin candidate

pull-request

Mirrored from GitHub Pull Request

severity: high

severity: low

severity: medium

status: accepted

status: backlog

status: blocked

status: duplicate

status: needs owner

status: needs triage

status: revisions needed

status: under review

topic: GraphQL

topic: Internationalization

topic: OpenAPI

topic: UI/UX

topic: cabling

topic: event rules

topic: htmx navigation

topic: industrialization

topic: migrations

topic: plugins

topic: scripts

topic: templating

topic: testing

type: bug

type: deprecation

type: documentation

type: feature

type: housekeeping

type: translation

No Label severity: low status: accepted type: bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

adam

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/netbox#11400