starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Initial NetBox 3.4.x migration fails when a plugin using SearchIndex is installed #7388

New Issue
Closed
opened 2025-12-29 20:22:45 +01:00 by adam · 3 comments
adam commented 2025-12-29 20:22:45 +01:00
Owner
Copy Link

Originally created by @peteeckel on GitHub (Dec 19, 2022).

NetBox version

3.4.1

Python version

3.8

Steps to Reproduce

  1. Install NetBox 3.4.1
  2. Install any plugin that uses SearchIndex functionality (e.g. netbox-dns) and configure it in PLUGINS
  3. Run the initial migration

Expected Behavior

The migration succeeds

Observed Behavior

The migration fails with ProgrammingError exception:

Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, django_rq, extras, ipam, netbox_dns, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
  Applying extras.0083_search...Reindexing 67 models.
Clearing cached values... 0 entries deleted.
Indexing models
  netbox_dns.nameserver... Traceback (most recent call last):
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "netbox_dns_nameserver" does not exist
LINE 1: ...name", "netbox_dns_nameserver"."description" FROM "netbox_dn...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 96, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 349, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/migration.py", line 130, in apply
    operation.database_forwards(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/extras/migrations/0083_search.py", line 13, in reindex
    management.call_command('reindex')
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 198, in call_command
    return command.execute(*args, **defaults)
  File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/netbox/extras/management/commands/reindex.py", line 68, in handle
    i = search_backend.cache(model.objects.iterator(), remove_existing=False)
  File "/opt/netbox/netbox/netbox/search/backends.py", line 148, in cache
    for instance in instances:
  File "/opt/netbox/lib/python3.8/site-packages/django/db/models/query.py", line 512, in _iterator
    yield from iterable
  File "/opt/netbox/lib/python3.8/site-packages/django/db/models/query.py", line 87, in __iter__
    results = compiler.execute_sql(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 103, in execute
    return super().execute(sql, params)
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "netbox_dns_nameserver" does not exist
LINE 1: ...name", "netbox_dns_nameserver"."description" FROM "netbox_dn...

The netbox_dns_nameserver relation for the netbox_dns.models.NameServer model uses SearchIndex:

@register_search
class NameServerIndex(SearchIndex):
    model = NameServer
    fields = (
        ("name", 100),
        ("description", 500),
    )

That results in the NetBox migration extras.0083_search trying to reindex the plugin's relation, which does not exist yet at that point.

The obvious workaround is to disable the plugin, run the migration, re-enable the plugin and then run the migration again.

Originally created by @peteeckel on GitHub (Dec 19, 2022). ### NetBox version 3.4.1 ### Python version 3.8 ### Steps to Reproduce 1. Install NetBox 3.4.1 2. Install any plugin that uses SearchIndex functionality (e.g. netbox-dns) and configure it in PLUGINS 3. Run the initial migration ### Expected Behavior The migration succeeds ### Observed Behavior The migration fails with ProgrammingError exception: ``` Operations to perform: Apply all migrations: admin, auth, circuits, contenttypes, dcim, django_rq, extras, ipam, netbox_dns, sessions, social_django, taggit, tenancy, users, virtualization, wireless Running migrations: Applying extras.0083_search...Reindexing 67 models. Clearing cached values... 0 entries deleted. Indexing models netbox_dns.nameserver... Traceback (most recent call last): File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedTable: relation "netbox_dns_nameserver" does not exist LINE 1: ...name", "netbox_dns_nameserver"."description" FROM "netbox_dn... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/netbox/netbox/manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 96, in wrapped res = handle_func(*args, **kwargs) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 349, in handle post_migrate_state = executor.migrate( File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards state = self.apply_migration( File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration state = migration.apply(state, schema_editor) File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/migration.py", line 130, in apply operation.database_forwards( File "/opt/netbox/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards self.code(from_state.apps, schema_editor) File "/opt/netbox/netbox/extras/migrations/0083_search.py", line 13, in reindex management.call_command('reindex') File "/opt/netbox/lib/python3.8/site-packages/django/core/management/__init__.py", line 198, in call_command return command.execute(*args, **defaults) File "/opt/netbox/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/netbox/netbox/extras/management/commands/reindex.py", line 68, in handle i = search_backend.cache(model.objects.iterator(), remove_existing=False) File "/opt/netbox/netbox/netbox/search/backends.py", line 148, in cache for instance in instances: File "/opt/netbox/lib/python3.8/site-packages/django/db/models/query.py", line 512, in _iterator yield from iterable File "/opt/netbox/lib/python3.8/site-packages/django/db/models/query.py", line 87, in __iter__ results = compiler.execute_sql( File "/opt/netbox/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql cursor.execute(sql, params) File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 103, in execute return super().execute(sql, params) File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) File "/opt/netbox/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/netbox/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "netbox_dns_nameserver" does not exist LINE 1: ...name", "netbox_dns_nameserver"."description" FROM "netbox_dn... ``` The `netbox_dns_nameserver` relation for the `netbox_dns.models.NameServer` model uses `SearchIndex`: ``` @register_search class NameServerIndex(SearchIndex): model = NameServer fields = ( ("name", 100), ("description", 500), ) ``` That results in the NetBox migration `extras.0083_search` trying to reindex the plugin's relation, which does not exist yet at that point. The obvious workaround is to disable the plugin, run the migration, re-enable the plugin and then run the migration again.
adam added the type: bugstatus: accepted labels 2025-12-29 20:22:45 +01:00
adam closed this issue 2025-12-29 20:22:45 +01:00
adam commented 2025-12-29 20:22:46 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Dec 22, 2022):

We can tweak the migration to simply skip any indexers whose tables don't exist yet, which should address the problem cited here. (This should be non-impacting, as if the table doesn't exist there's no data to index anyway.) However, there's also a scenario where indexing gets skipped if a plugin adds a new field to an existing table, which would require that the user manually re-generate the search cache for the plugin model.

@jeremystretch commented on GitHub (Dec 22, 2022): We can tweak the migration to simply skip any indexers whose tables don't exist yet, which should address the problem cited here. (This should be non-impacting, as if the table doesn't exist there's no data to index anyway.) However, there's also a scenario where indexing gets skipped if a plugin adds a new field to an existing table, which would require that the user manually re-generate the search cache for the plugin model.
adam commented 2025-12-29 20:22:46 +01:00
Author
Owner
Copy Link

@peteeckel commented on GitHub (Dec 22, 2022):

However, there's also a scenario where indexing gets skipped if a plugin adds a new field to an existing table, which would require that the user manually re-generate the search cache for the plugin model.

Exactly that happens with my plugin ... but I don't think it's a major issue, I can add the reindexing code to the migration for the plugin.

@peteeckel commented on GitHub (Dec 22, 2022): > However, there's also a scenario where indexing gets skipped if a plugin adds a new field to an existing table, which would require that the user manually re-generate the search cache for the plugin model. Exactly that happens with my plugin ... but I don't think it's a major issue, I can add the reindexing code to the migration for the plugin.
adam commented 2025-12-29 20:22:47 +01:00
Author
Owner
Copy Link

@abhi1693 commented on GitHub (Dec 22, 2022):

Can the reindex command be added as a post-migration signal call? With that, it runs only after the migration for the plugin has finished. Although, the side-effect of this approach that I can think of is the overhead of calling the command several times for big migrations.

@abhi1693 commented on GitHub (Dec 22, 2022): Can the `reindex` command be added as a post-migration signal call? With that, it runs only after the migration for the plugin has finished. Although, the side-effect of this approach that I can think of is the overhead of calling the command several times for big migrations.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#7388
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?