starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Netbox fails to connect to read-only postgresql #11289

New Issue
Closed
opened 2025-12-29 21:43:00 +01:00 by adam · 10 comments
adam commented 2025-12-29 21:43:00 +01:00
Owner
Copy Link

Originally created by @fumped on GitHub (Jun 17, 2025).

Deployment Type

Self-hosted

NetBox Version

v4.3.1

Python Version

3.10

Steps to Reproduce

Netbox is supposed to be able to run towards read only replicas of Postgres.
https://github.com/netbox-community/netbox/pull/12490
https://github.com/netbox-community/netbox/issues/11233

It fails to start.

We're running the official helm-chart but after input from Benjamin Reed it seems like the issue is in core netbox and not specific to the containerized version.

We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2.

Image

Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines.
C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO.

The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW.

C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases.
So Yes, the Database is set to the Replica in C2 and to the VIP in C1.

Values:

Chart netbox C1

netbox:
  externalDatabase:
    host: IP-FOR-C1-VIP
    port: 5432
    database: netbox

Chart netbox C2

netbox:
  externalDatabase:
    host: IP-FOR-C2-IP
    port: 5432
    database: netbox

Expected Behavior

Netbox should successfully launch.

Observed Behavior

"django.db.utils.OperationalError: connection failed: connection to server at "ip-address-here", port 5432 failed: session is read-only
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]
:hourglass_flowing_sand: Waiting on DB... (3s / 30s)"
Originally created by @fumped on GitHub (Jun 17, 2025). ### Deployment Type Self-hosted ### NetBox Version v4.3.1 ### Python Version 3.10 ### Steps to Reproduce Netbox is supposed to be able to run towards read only replicas of Postgres. https://github.com/netbox-community/netbox/pull/12490 https://github.com/netbox-community/netbox/issues/11233 It fails to start. We're running the official helm-chart but after input from Benjamin Reed it seems like the issue is in core netbox and not specific to the containerized version. We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2. ![Image](https://github.com/user-attachments/assets/783b0042-64d5-4d3b-b1eb-5f655998c852) Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines. C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO. The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW. C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases. So Yes, the Database is set to the Replica in C2 and to the VIP in C1. Values: # Chart netbox C1 ``` netbox: externalDatabase: host: IP-FOR-C1-VIP port: 5432 database: netbox ``` # Chart netbox C2 ``` netbox: externalDatabase: host: IP-FOR-C2-IP port: 5432 database: netbox ``` ### Expected Behavior Netbox should successfully launch. ### Observed Behavior ``` "django.db.utils.OperationalError: connection failed: connection to server at "ip-address-here", port 5432 failed: session is read-only [ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ] :hourglass_flowing_sand: Waiting on DB... (3s / 30s)" ```
adam added the netbox label 2025-12-29 21:43:00 +01:00
adam closed this issue 2025-12-29 21:43:00 +01:00
adam commented 2025-12-29 21:43:01 +01:00
Author
Owner
Copy Link

@arthanson commented on GitHub (Jun 18, 2025):

@fumped Thank you for opening a bug report. Unfortunately, the information you have provided is not sufficient for someone else to attempt to reproduce the reported behavior. Remember, each bug report must include detailed steps that someone else can follow on a clean, empty NetBox installation to reproduce the exact problem you're experiencing. These instructions should include the creation of any involved objects, any configuration changes, and complete accounting of the actions being taken. Also be sure that your report does not reference data on the public NetBox demo, as that is subject to change at any time by an outside party and cannot be relied upon for bug reports.

Can you please add repro steps, you don't put anything there about how you are configuring anything - are you setting DATABASES to something or?

@arthanson commented on GitHub (Jun 18, 2025): @fumped Thank you for opening a bug report. Unfortunately, the information you have provided is not sufficient for someone else to attempt to reproduce the reported behavior. Remember, each bug report must include detailed steps that someone else can follow on a clean, empty NetBox installation to reproduce the exact problem you're experiencing. These instructions should include the creation of any involved objects, any configuration changes, and complete accounting of the actions being taken. Also be sure that your report does not reference data on the public NetBox demo, as that is subject to change at any time by an outside party and cannot be relied upon for bug reports. Can you please add repro steps, you don't put anything there about how you are configuring anything - are you setting DATABASES to something or?
adam commented 2025-12-29 21:43:02 +01:00
Author
Owner
Copy Link

@fumped commented on GitHub (Jun 19, 2025):

Hi,

We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2.

Image

Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines.
C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO.

The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW.

C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases.
So Yes, the Database is set to the Replica in C2 and to the VIP in C1.

Values:

# Chart netbox C1
netbox:
  externalDatabase:
    host: IP-FOR-C1-VIP
    port: 5432
    database: netbox
    
# Chart netbox C2
netbox:
  externalDatabase:
    host: IP-FOR-C2-IP
    port: 5432
    database: netbox

If this is not supported, please let me know :).

@fumped commented on GitHub (Jun 19, 2025): Hi, We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2. ![Image](https://github.com/user-attachments/assets/842cd18f-4a94-4d28-9929-bff394b2ae56) Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines. C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO. The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW. C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases. So Yes, the Database is set to the Replica in C2 and to the VIP in C1. Values: ``` # Chart netbox C1 netbox: externalDatabase: host: IP-FOR-C1-VIP port: 5432 database: netbox # Chart netbox C2 netbox: externalDatabase: host: IP-FOR-C2-IP port: 5432 database: netbox ``` If this is not supported, please let me know :).
adam commented 2025-12-29 21:43:02 +01:00
Author
Owner
Copy Link

@pycolas commented on GitHub (Jun 19, 2025):

Hi, do you set MAINTENANCE_MODE on C2 ?

@pycolas commented on GitHub (Jun 19, 2025): Hi, do you set [MAINTENANCE_MODE](https://netboxlabs.com/docs/netbox/configuration/miscellaneous/#maintenance_mode) on C2 ?
adam commented 2025-12-29 21:43:02 +01:00
Author
Owner
Copy Link

@fumped commented on GitHub (Jun 19, 2025):

Hi, do you set MAINTENANCE_MODE on C2 ?

Yes, it's set to maintenance mode, using this as value to the chart.

netbox:
  maintenanceMode: true

I tried setting this in C1 too and it got applied correctly with the banner.

@fumped commented on GitHub (Jun 19, 2025): > Hi, do you set [MAINTENANCE_MODE](https://netboxlabs.com/docs/netbox/configuration/miscellaneous/#maintenance_mode) on C2 ? Yes, it's set to maintenance mode, using this as value to the chart. ``` netbox: maintenanceMode: true ``` I tried setting this in C1 too and it got applied correctly with the banner.
adam commented 2025-12-29 21:43:03 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jun 27, 2025):

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

@github-actions[bot] commented on GitHub (Jun 27, 2025): This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.
adam commented 2025-12-29 21:43:03 +01:00
Author
Owner
Copy Link

@fumped commented on GitHub (Jul 1, 2025):

Got notice it will be auto-closed, I updates the main post. Hopefully that was it :)

@fumped commented on GitHub (Jul 1, 2025): Got notice it will be auto-closed, I updates the main post. Hopefully that was it :)
adam commented 2025-12-29 21:43:03 +01:00
Author
Owner
Copy Link

@RangerRick commented on GitHub (Jul 2, 2025):

Looking at the original links about RO mode, the work was done in NetBox 3.x. I wonder if the move to Django 5 in nb4.x broke something?

The pod startup is running ./manage.py showmigrations to check if the DB is available, and that's where the above error comes from. Just the act of showing migrations shouldn't, in theory, require read-write access to the database.

@RangerRick commented on GitHub (Jul 2, 2025): Looking at the original links about RO mode, the work was done in NetBox 3.x. I wonder if the move to Django 5 in nb4.x broke something? The pod startup is running `./manage.py showmigrations` to check if the DB is available, and that's where the above error comes from. Just the act of showing migrations shouldn't, in theory, require read-write access to the database.
adam commented 2025-12-29 21:43:04 +01:00
Author
Owner
Copy Link

@cruse1977 commented on GitHub (Jul 2, 2025):

@fumped - is this relevant ? - https://github.com/netbox-community/netbox/discussions/17026

@cruse1977 commented on GitHub (Jul 2, 2025): @fumped - is this relevant ? - https://github.com/netbox-community/netbox/discussions/17026
adam commented 2025-12-29 21:43:04 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jul 10, 2025):

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

@github-actions[bot] commented on GitHub (Jul 10, 2025): This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.
adam commented 2025-12-29 21:43:04 +01:00
Author
Owner
Copy Link

@fumped commented on GitHub (Jul 10, 2025):

@fumped - is this relevant ? - #17026

Hi, thanks for the link.
It does not seem to be the same issue, i don't see this user trying to connect to a read-only replica and my pods don't get to the traceback stage. It just keeps sending the same messages.
Seems like the solution for this ticket is not in the chart anymore but I tried setting it anyway, it made no difference.

externalDatabase:
targetSessionAttrs: any

@fumped commented on GitHub (Jul 10, 2025): > [@fumped](https://github.com/fumped) - is this relevant ? - [#17026](https://github.com/netbox-community/netbox/discussions/17026) Hi, thanks for the link. It does not seem to be the same issue, i don't see this user trying to connect to a read-only replica and my pods don't get to the traceback stage. It just keeps sending the same messages. Seems like the solution for this ticket is not in the chart anymore but I tried setting it anyway, it made no difference. externalDatabase: targetSessionAttrs: any
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label netbox
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#11289
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?