starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Exception during migration from 2.9.11 to 2.10.2 when customfieldvalue references non-existent object #4434

New Issue
Closed
opened 2025-12-29 18:35:56 +01:00 by adam · 2 comments
adam commented 2025-12-29 18:35:56 +01:00
Owner
Copy Link

Originally created by @candlerb on GitHub (Jan 4, 2021).

Originally assigned to: @candlerb on GitHub.

Environment

  • Python version: 3.6.9
  • NetBox version: 2.9.11 -> 2.10.2

Steps to Reproduce

  1. Started with Netbox 2.9.11 where there is some custom field data (*)
  2. git pull and ./upgrade.sh

(*) specifics below

Expected Behavior

Data to be migrated

Observed Behavior

  Applying extras.0051_migrate_customfields...Traceback (most recent call last):
  File "netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues
    cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
TypeError: 'NoneType' object is not subscriptable

Analysis

After adding some debugging:

--- a/netbox/extras/migrations/0051_migrate_customfields.py
+++ b/netbox/extras/migrations/0051_migrate_customfields.py
@@ -66,6 +66,13 @@ def migrate_customfieldvalues(apps, schema_editor):
         # TODO: This can be done more efficiently once .update() is supported for JSON fields
         cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()
         try:
+            print("***")
+            print("cfv =", cfv)
+            print("cfv.obj_id =", cfv.obj_id)
+            print("cfv.field =", cfv.field)
+            print("cfv.serialized_value =", cfv.serialized_value)
+            print("cf_data =", cf_data)
+            print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data'))
             cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
         except ValueError as e:
             print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk})')

Result:

...
***
cfv = CustomFieldValue object (241)
cfv.obj_id = 125
cfv.field = CustomField object (9)
cfv.serialized_value = 40
cf_data = {'custom_field_data': {}}
cf_data.get('custom_field_data') = {}
***
cfv = CustomFieldValue object (20)
cfv.obj_id = 133
cfv.field = CustomField object (1)
cfv.serialized_value = 2
cf_data = None
Traceback (most recent call last):
...
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 75, in migrate_customfieldvalues
    print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data'))
AttributeError: 'NoneType' object has no attribute 'get'

So the problem is that cf_data is None. In turn:

netbox=# select * from extras_customfieldvalue where id=20;
 id | obj_id | serialized_value | field_id | obj_type_id
----+--------+------------------+----------+-------------
 20 |    133 | 2                |        1 |          23
(1 row)

netbox=# select * from django_content_type where id=23;
 id | app_label | model
----+-----------+--------
 23 | dcim      | device
(1 row)

netbox=# select id,name from dcim_device where id=133;
 id | name
----+------
(0 rows)

The problem occurs when there is some left-over customfieldvalue which references a non-existent object.

Proposed fix

Print a warning and skip:

--- a/netbox/extras/migrations/0051_migrate_customfields.py
+++ b/netbox/extras/migrations/0051_migrate_customfields.py
@@ -65,6 +65,9 @@ def migrate_customfieldvalues(apps, schema_editor):
         # Read and update custom field value for each instance
         # TODO: This can be done more efficiently once .update() is supported for JSON fields
         cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()
+        if cf_data is None:
+            print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk}): references non-existent {model.__name__} {cfv.obj_id}')
+            continue
         try:
             cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
         except ValueError as e:
Originally created by @candlerb on GitHub (Jan 4, 2021). Originally assigned to: @candlerb on GitHub. ### Environment * Python version: 3.6.9 * NetBox version: 2.9.11 -> 2.10.2 ### Steps to Reproduce 1. Started with Netbox 2.9.11 where there is some custom field data (*) 2. git pull and `./upgrade.sh` (*) specifics below ### Expected Behavior Data to be migrated ### Observed Behavior ``` Applying extras.0051_migrate_customfields...Traceback (most recent call last): File "netbox/manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line utility.execute() File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv self.execute(*args, **cmd_options) File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute output = self.handle(*args, **options) File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 85, in wrapped res = handle_func(*args, **kwargs) File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 245, in handle fake_initial=fake_initial, File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 227, in apply_migration state = migration.apply(state, schema_editor) File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 124, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards self.code(from_state.apps, schema_editor) File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value) TypeError: 'NoneType' object is not subscriptable ``` ### Analysis After adding some debugging: ``` --- a/netbox/extras/migrations/0051_migrate_customfields.py +++ b/netbox/extras/migrations/0051_migrate_customfields.py @@ -66,6 +66,13 @@ def migrate_customfieldvalues(apps, schema_editor): # TODO: This can be done more efficiently once .update() is supported for JSON fields cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first() try: + print("***") + print("cfv =", cfv) + print("cfv.obj_id =", cfv.obj_id) + print("cfv.field =", cfv.field) + print("cfv.serialized_value =", cfv.serialized_value) + print("cf_data =", cf_data) + print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data')) cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value) except ValueError as e: print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk})') ``` Result: ``` ... *** cfv = CustomFieldValue object (241) cfv.obj_id = 125 cfv.field = CustomField object (9) cfv.serialized_value = 40 cf_data = {'custom_field_data': {}} cf_data.get('custom_field_data') = {} *** cfv = CustomFieldValue object (20) cfv.obj_id = 133 cfv.field = CustomField object (1) cfv.serialized_value = 2 cf_data = None Traceback (most recent call last): ... File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 75, in migrate_customfieldvalues print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data')) AttributeError: 'NoneType' object has no attribute 'get' ``` So the problem is that `cf_data` is None. In turn: ``` netbox=# select * from extras_customfieldvalue where id=20; id | obj_id | serialized_value | field_id | obj_type_id ----+--------+------------------+----------+------------- 20 | 133 | 2 | 1 | 23 (1 row) netbox=# select * from django_content_type where id=23; id | app_label | model ----+-----------+-------- 23 | dcim | device (1 row) netbox=# select id,name from dcim_device where id=133; id | name ----+------ (0 rows) ``` The problem occurs when there is some left-over customfieldvalue which references a non-existent object. ### Proposed fix Print a warning and skip: ``` --- a/netbox/extras/migrations/0051_migrate_customfields.py +++ b/netbox/extras/migrations/0051_migrate_customfields.py @@ -65,6 +65,9 @@ def migrate_customfieldvalues(apps, schema_editor): # Read and update custom field value for each instance # TODO: This can be done more efficiently once .update() is supported for JSON fields cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first() + if cf_data is None: + print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk}): references non-existent {model.__name__} {cfv.obj_id}') + continue try: cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value) except ValueError as e: ```
adam added the type: bugstatus: accepted labels 2025-12-29 18:35:56 +01:00
adam closed this issue 2025-12-29 18:35:56 +01:00
adam commented 2025-12-29 18:35:57 +01:00
Author
Owner
Copy Link

@candlerb commented on GitHub (Jan 4, 2021):

Update: I have successfully updated the system with this fix.

I have also traced where the bad data came from. It originated way back (like 2017), in the days before Netbox had VirtualMachines, and I had modelled VMs as Devices.

When Netbox gained VirtualMachines I moved them from dcim_device to virtualization_virtualmachine (keeping the same id), but I forgot to update the related extras_customfieldvalue rows - I should have either updated the object_type_id, or deleted the row entirely. Hence these dangling values were still pointing at "devices" which no longer exist.

@candlerb commented on GitHub (Jan 4, 2021): Update: I have successfully updated the system with this fix. I have also traced where the bad data came from. It originated way back (like 2017), in the days before Netbox had VirtualMachines, and I had modelled VMs as Devices. When Netbox gained VirtualMachines I moved them from dcim_device to virtualization_virtualmachine (keeping the same id), but I forgot to update the related extras_customfieldvalue rows - I should have either updated the object_type_id, or deleted the row entirely. Hence these dangling values were still pointing at "devices" which no longer exist.
adam commented 2025-12-29 18:35:57 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Jan 5, 2021):

If you want to submit a PR for this that would be swell.

@DanSheps commented on GitHub (Jan 5, 2021): If you want to submit a PR for this that would be swell.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#4434
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?