starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

AWS_STORAGE_BUCKET_NAME env variable ignored when using S3 storage (using MinIO) #11535

New Issue
Closed
opened 2025-12-29 21:46:29 +01:00 by adam · 1 comment
adam commented 2025-12-29 21:46:29 +01:00
Owner
Copy Link

Originally created by @mx-db on GitHub (Aug 27, 2025).

Deployment Type

Self-hosted

NetBox Version

v4.3.6

Python Version

3.12

Steps to Reproduce

  1. Configure storage values for helm deployment
...
dbWaitDebug: true
storages:
  default:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: "netbox-data"
      location: "images"
  staticfiles:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: "netbox-data"
      location: "static"
  scripts:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: "netbox-data"
      location: "scripts"

extraEnvVarsSecret: "netbox-s3"
...
  1. Create the corresponding env secret "netbox-s3" (with all imaginable alternative variable names) (see https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#)
data:
  AWS_S3_ACCESS_KEY_ID: "{{ app_netbox_s3_data_access_key | b64encode }}"
  AWS_S3_SECRET_ACCESS_KEY: "{{ app_netbox_s3_data_secret_key | b64encode }}"
  AWS_STORAGE_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}"
  AWS_S3_STORAGE_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}"
  AWS_S3_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}"
  AWS_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}"
  AWS_S3_ENDPOINT_URL: "{{ app_netbox_s3_data_url | b64encode }}"
  AWS_S3_ADDRESSING_STYLE: "{{ 'path' | b64encode }}"
  1. Deploy Netbox via Helm with specified values

Expected Behavior

Netbox gets the storage options from the specified environment variables and successfully connects to the s3 storage.

Observed Behavior

Netbox fails to initialize because bucket name is missing:

⏳ Waiting on DB... (3s / 30s)
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/netbox/netbox/core/models/__init__.py", line 6, in <module>
    from .jobs import *
  File "/opt/netbox/netbox/core/models/jobs.py", line 18, in <module>
    from core.signals import job_end, job_start
  File "/opt/netbox/netbox/core/signals.py", line 15, in <module>
    from extras.events import enqueue_event
  File "/opt/netbox/netbox/extras/events.py", line 20, in <module>
    from .models import EventRule
  File "/opt/netbox/netbox/extras/models/__init__.py", line 2, in <module>
    from .customfields import *
  File "/opt/netbox/netbox/extras/models/customfields.py", line 27, in <module>
    from utilities.forms.fields import (
  File "/opt/netbox/netbox/utilities/forms/fields/__init__.py", line 6, in <module>
    from .fields import *
  File "/opt/netbox/netbox/utilities/forms/fields/fields.py", line 24, in <module>
    class CommentField(forms.CharField):
  File "/opt/netbox/netbox/utilities/forms/fields/fields.py", line 33, in CommentField
    ).format(url=static('docs/reference/markdown/'))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/templatetags/static.py", line 179, in static
    return StaticNode.handle_simple(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/templatetags/static.py", line 129, in handle_simple
    return staticfiles_storage.url(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/storages/backends/s3.py", line 691, in url
    params["Bucket"] = self.bucket.name
                       ^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/storages/backends/s3.py", line 515, in bucket
    self._bucket = self.connection.Bucket(self.bucket_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/boto3/resources/factory.py", line 528, in create_resource
    return partial(
           ^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/boto3/resources/base.py", line 123, in __init__
    raise ValueError(f'Required parameter {identifier} not set')
ValueError: Required parameter name not set

Checking the env inside the container shows the variables as expected:

$ printenv | grep BUCKET
      AWS_S3_BUCKET_NAME=my-bucket
      AWS_BUCKET_NAME=my-bucket
      AWS_STORAGE_BUCKET_NAME=my-bucket
      AWS_S3_STORAGE_BUCKET_NAME=my-bucket

However, when adding the bucket name in the config, everything works as expected (or to be more specific results in #19615), so all the other storage related variables are being used correctly from the environment variables:

 storages:
  default:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: my-bucket
      location: "images"
  staticfiles:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: my-bucket
      location: "static"
  scripts:
    BACKEND: "storages.backends.s3.S3Storage"
    OPTIONS:
      bucket_name: my-bucket
      location: "scripts"

extraEnvVarsSecret: "netbox-s3"
Originally created by @mx-db on GitHub (Aug 27, 2025). ### Deployment Type Self-hosted ### NetBox Version v4.3.6 ### Python Version 3.12 ### Steps to Reproduce 1. Configure storage values for helm deployment ``` ... dbWaitDebug: true storages: default: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: "netbox-data" location: "images" staticfiles: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: "netbox-data" location: "static" scripts: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: "netbox-data" location: "scripts" extraEnvVarsSecret: "netbox-s3" ... ``` 2. Create the corresponding env secret "netbox-s3" (with all imaginable alternative variable names) (see https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#) ``` data: AWS_S3_ACCESS_KEY_ID: "{{ app_netbox_s3_data_access_key | b64encode }}" AWS_S3_SECRET_ACCESS_KEY: "{{ app_netbox_s3_data_secret_key | b64encode }}" AWS_STORAGE_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}" AWS_S3_STORAGE_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}" AWS_S3_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}" AWS_BUCKET_NAME: "{{ app_netbox_s3_data_bucket | b64encode }}" AWS_S3_ENDPOINT_URL: "{{ app_netbox_s3_data_url | b64encode }}" AWS_S3_ADDRESSING_STYLE: "{{ 'path' | b64encode }}" ``` 3. Deploy Netbox via Helm with specified values ### Expected Behavior Netbox gets the storage options from the specified environment variables and successfully connects to the s3 storage. ### Observed Behavior Netbox fails to initialize because bucket name is missing: ``` ⏳ Waiting on DB... (3s / 30s) 🧬 loaded config '/etc/netbox/config/configuration.py' 🧬 loaded config '/etc/netbox/config/extra.py' 🧬 loaded config '/etc/netbox/config/logging.py' 🧬 loaded config '/etc/netbox/config/plugins.py' Traceback (most recent call last): File "/opt/netbox/netbox/./manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line utility.execute() File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute django.setup() File "/opt/netbox/venv/lib/python3.12/site-packages/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/netbox/venv/lib/python3.12/site-packages/django/apps/registry.py", line 116, in populate app_config.import_models() File "/opt/netbox/venv/lib/python3.12/site-packages/django/apps/config.py", line 269, in import_models self.models_module = import_module(models_module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 995, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/opt/netbox/netbox/core/models/__init__.py", line 6, in <module> from .jobs import * File "/opt/netbox/netbox/core/models/jobs.py", line 18, in <module> from core.signals import job_end, job_start File "/opt/netbox/netbox/core/signals.py", line 15, in <module> from extras.events import enqueue_event File "/opt/netbox/netbox/extras/events.py", line 20, in <module> from .models import EventRule File "/opt/netbox/netbox/extras/models/__init__.py", line 2, in <module> from .customfields import * File "/opt/netbox/netbox/extras/models/customfields.py", line 27, in <module> from utilities.forms.fields import ( File "/opt/netbox/netbox/utilities/forms/fields/__init__.py", line 6, in <module> from .fields import * File "/opt/netbox/netbox/utilities/forms/fields/fields.py", line 24, in <module> class CommentField(forms.CharField): File "/opt/netbox/netbox/utilities/forms/fields/fields.py", line 33, in CommentField ).format(url=static('docs/reference/markdown/')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/django/templatetags/static.py", line 179, in static return StaticNode.handle_simple(path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/django/templatetags/static.py", line 129, in handle_simple return staticfiles_storage.url(path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/storages/backends/s3.py", line 691, in url params["Bucket"] = self.bucket.name ^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/storages/backends/s3.py", line 515, in bucket self._bucket = self.connection.Bucket(self.bucket_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/boto3/resources/factory.py", line 528, in create_resource return partial( ^^^^^^^^ File "/opt/netbox/venv/lib/python3.12/site-packages/boto3/resources/base.py", line 123, in __init__ raise ValueError(f'Required parameter {identifier} not set') ValueError: Required parameter name not set ``` Checking the env inside the container shows the variables as expected: ``` $ printenv | grep BUCKET AWS_S3_BUCKET_NAME=my-bucket AWS_BUCKET_NAME=my-bucket AWS_STORAGE_BUCKET_NAME=my-bucket AWS_S3_STORAGE_BUCKET_NAME=my-bucket ``` However, when adding the bucket name in the config, everything works as expected (or to be more specific results in #19615), so all the other storage related variables are being used correctly from the environment variables: ``` storages: default: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: my-bucket location: "images" staticfiles: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: my-bucket location: "static" scripts: BACKEND: "storages.backends.s3.S3Storage" OPTIONS: bucket_name: my-bucket location: "scripts" extraEnvVarsSecret: "netbox-s3" ```
adam added the type: bugstatus: under review labels 2025-12-29 21:46:29 +01:00
adam closed this issue 2025-12-29 21:46:29 +01:00
adam commented 2025-12-29 21:46:30 +01:00
Author
Owner
Copy Link

@jnovinger commented on GitHub (Aug 27, 2025):

Thank you for the bug report, @mx-db .

For future reference, bug reports must include reproduction steps that work on a standard NetBox installation (running in a virtualenv without Docker/Helm/Kubernetes dependencies). Since this issue requires a complex deployment stack to reproduce as originally reported, it is better suited for the project that it relies on for those dependencies--possibly netbox-community/netbox-chart for Helm-specific configuration issues or netbox-community/netbox-docker for Docker-related configuration patterns--or GitHub Discussions for deployment and configuration assistance.

This behavior is due to how NetBox handles django-storages configuration rather than a bug. NetBox processes storage configuration in a way that prevents django-storages from automatically detecting environment variables like AWS_STORAGE_BUCKET_NAME. When you configure S3Storage without explicit bucket_name in OPTIONS, the environment variable detection that's documented in django-storages doesn't work in NetBox's context.

You can achieve the desired behavior by explicitly reading environment variables in your NetBox configuration.py (if this works in your deployment scenario):

import os
STORAGES = {
    'staticfiles': {
        'BACKEND': 'storages.backends.s3.S3Storage',
        'OPTIONS': {
            'bucket_name': os.environ.get('AWS_STORAGE_BUCKET_NAME'),
            'location': 'static/',
        }
    }
}

This works because the environment variable gets resolved during NetBox's configuration processing.

I'll be opening a feature request to improve NetBox's documentation to clarify this behavior and provide better examples. If you need further deployment assistance, GitHub Discussions would be the best place to continue the conversation.

@jnovinger commented on GitHub (Aug 27, 2025): Thank you for the bug report, @mx-db . For future reference, bug reports must include reproduction steps that work on a standard NetBox installation (running in a virtualenv without Docker/Helm/Kubernetes dependencies). Since this issue requires a complex deployment stack to reproduce as originally reported, it is better suited for the project that it relies on for those dependencies--possibly [netbox-community/netbox-chart](https://github.com/netbox-community/netbox-chart) for Helm-specific configuration issues or [netbox-community/netbox-docker](https://github.com/netbox-community/netbox-docker) for Docker-related configuration patterns--or GitHub Discussions for deployment and configuration assistance. This behavior is due to how NetBox handles `django-storages` configuration rather than a bug. NetBox processes storage configuration in a way that prevents `django-storages` from automatically detecting environment variables like `AWS_STORAGE_BUCKET_NAME`. When you configure S3Storage without explicit `bucket_name` in `OPTIONS`, the environment variable detection that's documented in `django-storages` doesn't work in NetBox's context. You can achieve the desired behavior by explicitly reading environment variables in your NetBox `configuration.py` (if this works in your deployment scenario): ```py import os STORAGES = { 'staticfiles': { 'BACKEND': 'storages.backends.s3.S3Storage', 'OPTIONS': { 'bucket_name': os.environ.get('AWS_STORAGE_BUCKET_NAME'), 'location': 'static/', } } } ``` This works because the environment variable gets resolved during NetBox's configuration processing. I'll be opening a feature request to improve NetBox's documentation to clarify this behavior and provide better examples. If you need further deployment assistance, GitHub Discussions would be the best place to continue the conversation.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: under review type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#11535
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?