starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Staged Changes: InventoryItem Fails Sync #8437

New Issue
Closed
opened 2025-12-29 20:36:41 +01:00 by adam · 8 comments
adam commented 2025-12-29 20:36:41 +01:00
Owner
Copy Link

Originally created by @minitriga on GitHub (Aug 9, 2023).

Originally assigned to: @jeremystretch, @minitriga on GitHub.

NetBox version

v3.5.1

Python version

3.10

Steps to Reproduce

Currently when working with staged changes there is a edge-case where the staged changes failed on the merge.

I believe this is because of the serialiser removing key parts of the data required to recreate the device when merge() is called.

f5a1f83f9f/netbox/netbox/staging.py (L119)

And when serialize_object is called it removes all MPTT model fields: f5a1f83f9f/netbox/utilities/utils.py (L157)
This causes the merge to fail because the MPTT fields are now null. In the staged changes table.

Error message:

>>> Branch.objects.all()[0].merge()
TEST
<RestrictedQuerySet [<StagedChange: Create dcim.manufacturer (182)>, <StagedChange: Create dcim.devicetype (177)>, <StagedChange: Create dcim.devicerole (120)>, <StagedChange: Update dcim.site (124)>, <StagedChange: Create dcim.device (9608)>, <StagedChange: Create dcim.device (9609)>, <StagedChange: Create dcim.inventoryitem (848)>, <StagedChange: Create dcim.inventoryitem (849)>]>
Create dcim.manufacturer (182)
Create dcim.devicetype (177)
Create dcim.devicerole (120)
Update dcim.site (124)
Create dcim.device (9608)
Create dcim.device (9609)
Create dcim.inventoryitem (848)
Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.NotNullViolation: null value in column "lft" of relation "dcim_inventoryitem" violates not-null constraint
DETAIL:  Failing row contains (2023-08-09 12:54:07.062+00, 2023-08-09 12:54:07.062+00, {}, 848, Inventory Item 1, , , , , , null, f, null, null, null, null, 9608, null, null, null, null, null).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/netbox/netbox/extras/models/staging.py", line 52, in merge
    change.apply()
  File "/opt/netbox/netbox/extras/models/staging.py", line 107, in apply
    instance.save()
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/serializers/base.py", line 288, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1006, in _save_table
    results = self._do_insert(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1047, in _do_insert
    return manager._insert(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1791, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "lft" of relation "dcim_inventoryitem" violates not-null constraint
DETAIL:  Failing row contains (2023-08-09 12:54:07.062+00, 2023-08-09 12:54:07.062+00, {}, 848, Inventory Item 1, , , , , , null, f, null, null, null, null, 9608, null, null, null, null, null).

Script to recreate:

from netbox.staging import checkout
from extras.models import Branch
from dcim.models import Device, VirtualChassis, DeviceType, Manufacturer, Site, DeviceRole, InventoryItem

Branch.objects.all().delete()
branch = Branch.objects.create(name='Branch 1')

Device.objects.filter(name="test device").delete()
Device.objects.filter(name="test device1").delete()
VirtualChassis.objects.filter(name='VC1').delete()
DeviceType.objects.filter(slug='device-type-1').delete()
Manufacturer.objects.filter(slug='manufacturer-1').delete()
DeviceRole.objects.filter(slug='device-role-1').delete()


with checkout(branch):

    manufacturer = Manufacturer.objects.create(
        name='Manufacturer', slug='manufacturer-1')
    device_type = DeviceType.objects.create(
        manufacturer=manufacturer, model='Device Type 1', slug='device-type-1'
    )
    device_role = DeviceRole.objects.create(
        name='Device Role', slug='device-role-1'
    )
    site, _ = Site.objects.update_or_create(
        defaults={'name': "test"}, slug="test")
    device, _ = Device.objects.update_or_create(defaults={
        "name": "test device", "device_role": device_role, "device_type": device_type, "site": site}, serial="123")
    device1, _ = Device.objects.update_or_create(defaults={
        "name": "test device1", "device_role": device_role, "device_type": device_type, "site": site}, serial="543")
    inv1 = InventoryItem.objects.create(device=device, name='Inventory Item 1'),
    inv2 = InventoryItem.objects.create(device=device1, name='Inventory Item 2'),



branch.merge()

print("DONE")

Expected Behavior

If there is a serializer or argument to pass to the serializer that does not remove key fields to recreate objects from the staged changes table.

Observed Behavior

The serialized object has key fields removed which are required to save the object.

Originally created by @minitriga on GitHub (Aug 9, 2023). Originally assigned to: @jeremystretch, @minitriga on GitHub. ### NetBox version v3.5.1 ### Python version 3.10 ### Steps to Reproduce Currently when working with staged changes there is a edge-case where the staged changes failed on the merge. I believe this is because of the serialiser removing key parts of the data required to recreate the device when `merge()` is called. https://github.com/netbox-community/netbox/blob/f5a1f83f9fa9d98c945d21eb0f7ccb8cd37fbf59/netbox/netbox/staging.py#L119 And when `serialize_object` is called it removes all MPTT model fields: https://github.com/netbox-community/netbox/blob/f5a1f83f9fa9d98c945d21eb0f7ccb8cd37fbf59/netbox/utilities/utils.py#L157 This causes the merge to fail because the MPTT fields are now null. In the staged changes table. Error message: ``` >>> Branch.objects.all()[0].merge() TEST <RestrictedQuerySet [<StagedChange: Create dcim.manufacturer (182)>, <StagedChange: Create dcim.devicetype (177)>, <StagedChange: Create dcim.devicerole (120)>, <StagedChange: Update dcim.site (124)>, <StagedChange: Create dcim.device (9608)>, <StagedChange: Create dcim.device (9609)>, <StagedChange: Create dcim.inventoryitem (848)>, <StagedChange: Create dcim.inventoryitem (849)>]> Create dcim.manufacturer (182) Create dcim.devicetype (177) Create dcim.devicerole (120) Update dcim.site (124) Create dcim.device (9608) Create dcim.device (9609) Create dcim.inventoryitem (848) Traceback (most recent call last): File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) psycopg2.errors.NotNullViolation: null value in column "lft" of relation "dcim_inventoryitem" violates not-null constraint DETAIL: Failing row contains (2023-08-09 12:54:07.062+00, 2023-08-09 12:54:07.062+00, {}, 848, Inventory Item 1, , , , , , null, f, null, null, null, null, 9608, null, null, null, null, null). The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<console>", line 1, in <module> File "/opt/netbox/netbox/extras/models/staging.py", line 52, in merge change.apply() File "/opt/netbox/netbox/extras/models/staging.py", line 107, in apply instance.save() File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/serializers/base.py", line 288, in save models.Model.save_base(self.object, using=using, raw=True, **kwargs) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base updated = self._save_table( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1006, in _save_table results = self._do_insert( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1047, in _do_insert return manager._insert( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1791, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql cursor.execute(sql, params) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 102, in execute return super().execute(sql, params) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute with self.db.wrap_database_errors: File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: null value in column "lft" of relation "dcim_inventoryitem" violates not-null constraint DETAIL: Failing row contains (2023-08-09 12:54:07.062+00, 2023-08-09 12:54:07.062+00, {}, 848, Inventory Item 1, , , , , , null, f, null, null, null, null, 9608, null, null, null, null, null). ``` Script to recreate: ```python from netbox.staging import checkout from extras.models import Branch from dcim.models import Device, VirtualChassis, DeviceType, Manufacturer, Site, DeviceRole, InventoryItem Branch.objects.all().delete() branch = Branch.objects.create(name='Branch 1') Device.objects.filter(name="test device").delete() Device.objects.filter(name="test device1").delete() VirtualChassis.objects.filter(name='VC1').delete() DeviceType.objects.filter(slug='device-type-1').delete() Manufacturer.objects.filter(slug='manufacturer-1').delete() DeviceRole.objects.filter(slug='device-role-1').delete() with checkout(branch): manufacturer = Manufacturer.objects.create( name='Manufacturer', slug='manufacturer-1') device_type = DeviceType.objects.create( manufacturer=manufacturer, model='Device Type 1', slug='device-type-1' ) device_role = DeviceRole.objects.create( name='Device Role', slug='device-role-1' ) site, _ = Site.objects.update_or_create( defaults={'name': "test"}, slug="test") device, _ = Device.objects.update_or_create(defaults={ "name": "test device", "device_role": device_role, "device_type": device_type, "site": site}, serial="123") device1, _ = Device.objects.update_or_create(defaults={ "name": "test device1", "device_role": device_role, "device_type": device_type, "site": site}, serial="543") inv1 = InventoryItem.objects.create(device=device, name='Inventory Item 1'), inv2 = InventoryItem.objects.create(device=device1, name='Inventory Item 2'), branch.merge() print("DONE") ``` ### Expected Behavior If there is a serializer or argument to pass to the serializer that does not remove key fields to recreate objects from the staged changes table. ### Observed Behavior The serialized object has key fields removed which are required to save the object.
adam added the type: bugstatus: acceptedseverity: medium labels 2025-12-29 20:36:41 +01:00
adam closed this issue 2025-12-29 20:36:41 +01:00
adam commented 2025-12-29 20:36:42 +01:00
Author
Owner
Copy Link

@minitriga commented on GitHub (Aug 30, 2023):

@jeremystretch Happy to do the work on this but what do you think the best approach to solve this is? Add a keyword argument mptt=True and then if mptt before the mptt stuff?

@minitriga commented on GitHub (Aug 30, 2023): @jeremystretch Happy to do the work on this but what do you think the best approach to solve this is? Add a keyword argument `mptt=True` and then `if mptt` before the mptt stuff?
adam commented 2025-12-29 20:36:42 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Sep 26, 2023):

@minitriga it would probably suffice to force an MPTT recalculation after the merge. Unfortunately, with all the competing bug reports currently open, I'm not able to dedicate any time to this. Would you like to take ownership of this issue?

@jeremystretch commented on GitHub (Sep 26, 2023): @minitriga it would probably suffice to force an MPTT recalculation after the merge. Unfortunately, with all the competing bug reports currently open, I'm not able to dedicate any time to this. Would you like to take ownership of this issue?
adam commented 2025-12-29 20:36:42 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Dec 26, 2023):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Dec 26, 2023): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. **Do not** attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).
adam commented 2025-12-29 20:36:42 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jan 26, 2024):

This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.

@github-actions[bot] commented on GitHub (Jan 26, 2024): This issue has been automatically closed due to lack of activity. In an effort to reduce noise, please do not comment any further. Note that the core maintainers may elect to reopen this issue at a later date if deemed necessary.
adam commented 2025-12-29 20:36:43 +01:00
Author
Owner
Copy Link

@minitriga commented on GitHub (Apr 24, 2024):

Reviving this as our customers need to sync models that are MPTT Models. Such as Location and Inventory Item. Recalculating didnt work so well as to rebuild the MPTT you have to have the items in the DB and the issue is adding them into the DB without the MPTT fields.

@minitriga commented on GitHub (Apr 24, 2024): Reviving this as our customers need to sync models that are MPTT Models. Such as Location and Inventory Item. Recalculating didnt work so well as to rebuild the MPTT you have to have the items in the DB and the issue is adding them into the DB without the MPTT fields.
adam commented 2025-12-29 20:36:43 +01:00
Author
Owner
Copy Link

@jeffgdotorg commented on GitHub (May 14, 2024):

Reopening and assigning since this remains valid and needed, and a PR is ready.

@jeffgdotorg commented on GitHub (May 14, 2024): Reopening and assigning since this remains valid and needed, and a PR is ready.
adam commented 2025-12-29 20:36:44 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 24, 2024):

Marking this as blocked by #16290, which when implemented should fully mostly resolve this bug. (Please also see my comment on the PR.)

@jeremystretch commented on GitHub (May 24, 2024): Marking this as blocked by #16290, which when implemented should ~fully~ mostly resolve this bug. (Please also see [my comment](https://github.com/netbox-community/netbox/pull/15830#issuecomment-2129818707) on the PR.)
adam commented 2025-12-29 20:36:44 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 29, 2024):

Now that #16290 has been completed, I can confirm that MPTT-enabled objects are created successfully upon merging the branch. However, they may not be ordered properly due to having outdated MPTT positional attributes. Calling merge() on a Branch should automatically rebuilt the MPTT tree (e.g. Region.objects.build()) for any relevant models.

@jeremystretch commented on GitHub (May 29, 2024): Now that #16290 has been completed, I can confirm that MPTT-enabled objects are created successfully upon merging the branch. However, they may not be ordered properly due to having outdated MPTT positional attributes. Calling `merge()` on a Branch should automatically rebuilt the MPTT tree (e.g. `Region.objects.build()`) for any relevant models.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label severity: medium status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8437
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?