starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Background Tasks UI Broken when using Redis Sentinel #9072

New Issue
Closed
opened 2025-12-29 20:45:03 +01:00 by adam · 6 comments
adam commented 2025-12-29 20:45:03 +01:00
Owner
Copy Link

Originally created by @tagur87 on GitHub (Jan 9, 2024).

Deployment Type

Self-hosted

NetBox Version

v3.7.0

Python Version

3.11

Steps to Reproduce

  1. Deploy Redis Sentinel instance
  2. Deploy netbox with the following configuration for the REDIS_KEY
REDIS = {
  'tasks': {
      'SENTINELS': [('<sentinel>', <port>)],
      'SENTINEL_SERVICE': 'mymaster',
      'SENTINEL_TIMEOUT': 10,
      'PASSWORD': '',
      'DATABASE': 0,
      'SSL': False,
      'INSECURE_SKIP_TLS_VERIFY': False,
  },
  'caching': {
      'SENTINELS': [('<sentinel>', <port>)],
      'SENTINEL_SERVICE': 'mymaster',
      'SENTINEL_TIMEOUT': 10,
      'PASSWORD': '',
      'DATABASE': 1,
      'SSL': False,
      'INSECURE_SKIP_TLS_VERIFY': False,
  },
}
  1. Browse to http://..../admin/background-tasks/
  2. See error message

Expected Behavior

Background tasks page should work as normal when a regular redis instance is used.

Observed Behavior

Following error is returned from debug output:

Environment:


Request Method: GET
Request URL: https://admin-jdt-nsot.o11n-dev.jw-cd-lab-1.10aws.org/admin/background-tasks/

Django Version: 4.2.8
Python Version: 3.11.4
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.forms',
 'corsheaders',
 'debug_toolbar',
 'graphiql_debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'graphene_django',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'core',
 'account',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'vpn',
 'wireless',
 'django_rq',
 'drf_spectacular',
 'drf_spectacular_sidecar',
 'netbox_ip_reservations.IPReservationConfig']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'netbox.middleware.RemoteUserMiddleware',
 'netbox.middleware.CoreMiddleware',
 'netbox.middleware.MaintenanceModeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapper_view_func
    response = view_func(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/views.py", line 37, in stats
    **get_scheduler_statistics(),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/utils.py", line 106, in get_scheduler_statistics
    conn_key = f"{connection_kwargs['host']}:{connection_kwargs['port']}/{connection_kwargs.get('db', 0)}"
                  ^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: KeyError at /admin/background-tasks/
Exception Value: 'host'
Originally created by @tagur87 on GitHub (Jan 9, 2024). ### Deployment Type Self-hosted ### NetBox Version v3.7.0 ### Python Version 3.11 ### Steps to Reproduce 1. Deploy Redis Sentinel instance 2. Deploy netbox with the following configuration for the `REDIS_KEY` ``` REDIS = { 'tasks': { 'SENTINELS': [('<sentinel>', <port>)], 'SENTINEL_SERVICE': 'mymaster', 'SENTINEL_TIMEOUT': 10, 'PASSWORD': '', 'DATABASE': 0, 'SSL': False, 'INSECURE_SKIP_TLS_VERIFY': False, }, 'caching': { 'SENTINELS': [('<sentinel>', <port>)], 'SENTINEL_SERVICE': 'mymaster', 'SENTINEL_TIMEOUT': 10, 'PASSWORD': '', 'DATABASE': 1, 'SSL': False, 'INSECURE_SKIP_TLS_VERIFY': False, }, } ``` 3. Browse to http://..../admin/background-tasks/ 4. See error message ### Expected Behavior Background tasks page should work as normal when a regular redis instance is used. ### Observed Behavior Following error is returned from debug output: ``` Environment: Request Method: GET Request URL: https://admin-jdt-nsot.o11n-dev.jw-cd-lab-1.10aws.org/admin/background-tasks/ Django Version: 4.2.8 Python Version: 3.11.4 Installed Applications: ['django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'django.forms', 'corsheaders', 'debug_toolbar', 'graphiql_debug_toolbar', 'django_filters', 'django_tables2', 'django_prometheus', 'graphene_django', 'mptt', 'rest_framework', 'social_django', 'taggit', 'timezone_field', 'core', 'account', 'circuits', 'dcim', 'ipam', 'extras', 'tenancy', 'users', 'utilities', 'virtualization', 'vpn', 'wireless', 'django_rq', 'drf_spectacular', 'drf_spectacular_sidecar', 'netbox_ip_reservations.IPReservationConfig'] Installed Middleware: ['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware', 'django_prometheus.middleware.PrometheusBeforeMiddleware', 'corsheaders.middleware.CorsMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'netbox.middleware.RemoteUserMiddleware', 'netbox.middleware.CoreMiddleware', 'netbox.middleware.MaintenanceModeMiddleware', 'django_prometheus.middleware.PrometheusAfterMiddleware'] Traceback (most recent call last): File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapper_view_func response = view_func(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view return view_func(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/views.py", line 37, in stats **get_scheduler_statistics(), ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/utils.py", line 106, in get_scheduler_statistics conn_key = f"{connection_kwargs['host']}:{connection_kwargs['port']}/{connection_kwargs.get('db', 0)}" ^^^^^^^^^^^^^^^^^^^^^^^^^ Exception Type: KeyError at /admin/background-tasks/ Exception Value: 'host' ```
adam added the type: bugstatus: under reviewnetboxseverity: medium labels 2025-12-29 20:45:03 +01:00
adam closed this issue 2025-12-29 20:45:04 +01:00
adam commented 2025-12-29 20:45:04 +01:00
Author
Owner
Copy Link

@moonrail commented on GitHub (Jan 10, 2024):

We run into this issue as well - it seems to have been introduced in django-rq 2.9.0 via this commit:
df3271341a

Not completely sure, but seeing our own settings, your settings and the examples of django-rq in their README.md it seems they did not test with Sentinel and simply broke support with this change.

Edit: We've downgraded django-rq to 2.8.1 and there the error does not occur. If someone else might verify this as well, we should open an issue at django-rq´s repository.

@moonrail commented on GitHub (Jan 10, 2024): We run into this issue as well - it seems to have been introduced in django-rq 2.9.0 via this commit: https://github.com/rq/django-rq/commit/df3271341a3668717844dcdc9a4a77e64a6316d0 Not completely sure, but seeing our own settings, your settings and the examples of django-rq in their README.md it seems they did not test with Sentinel and simply broke support with this change. Edit: We've downgraded django-rq to 2.8.1 and there the error does not occur. If someone else might verify this as well, we should open an issue at django-rq´s repository.
adam commented 2025-12-29 20:45:04 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 16, 2024):

@tagur87 @moonrail could one of you please raise a bug report with django-rq to confirm that this is an uptsream issue?

@jeremystretch commented on GitHub (Jan 16, 2024): @tagur87 @moonrail could one of you please raise a bug report with [django-rq](https://github.com/rq/django-rq) to confirm that this is an uptsream issue?
adam commented 2025-12-29 20:45:04 +01:00
Author
Owner
Copy Link

@arthanson commented on GitHub (Jan 24, 2024):

There is a ticket in django-rq: https://github.com/rq/django-rq/issues/642

@arthanson commented on GitHub (Jan 24, 2024): There is a ticket in django-rq: https://github.com/rq/django-rq/issues/642
adam commented 2025-12-29 20:45:05 +01:00
Author
Owner
Copy Link

@moonrail commented on GitHub (Oct 11, 2024):

For NetBox < 4.0.0 this issue is relevant, as the Django Admin UI for Background Tasks is affected.

Since NetBox 4.0.0 this Admin UI is not integrated anymore and a Custom UI was built directly into NetBox. This custom implementation does not seem to call the function django_rq.utils.get_scheduler_statistics which contains the bug.

Therefore NetBox >= 4.0.0 currently does not hit this django-rq bug.

The underlying bug at django-rq has been fixed via 77bc4ff477 but is not released yet (current version is 2.10.2).

@moonrail commented on GitHub (Oct 11, 2024): For NetBox < 4.0.0 this issue is relevant, as the Django Admin UI for Background Tasks is affected. Since NetBox 4.0.0 this Admin UI is not integrated anymore and a Custom UI was built directly into NetBox. This custom implementation does not seem to call the function `django_rq.utils.get_scheduler_statistics` which contains the bug. Therefore NetBox >= 4.0.0 currently does not hit this django-rq bug. The underlying bug at django-rq has been fixed via https://github.com/rq/django-rq/commit/77bc4ff477f19bbf22cdd46ecf76dc5dee4f6fc8 but is not released yet (current version is 2.10.2).
adam commented 2025-12-29 20:45:05 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Dec 26, 2024):

NetBox v4.1.7 upgraded django-rq to v3.0, presumably resolving the upstream issue with Sentinel support. @moonrail @tagur87 are either of you able to confirm this?

@jeremystretch commented on GitHub (Dec 26, 2024): NetBox v4.1.7 upgraded django-rq to v3.0, presumably resolving the upstream issue with Sentinel support. @moonrail @tagur87 are either of you able to confirm this?
adam commented 2025-12-29 20:45:05 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Apr 22, 2025):

Going to close this out as there's been no further updates in four months.

@jeremystretch commented on GitHub (Apr 22, 2025): Going to close this out as there's been no further updates in four months.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label netbox severity: medium status: under review type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#9072
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?