The contrib systemd unit file can cause first NetBox WSGI start to fail #3572

Closed
opened 2025-12-29 18:29:57 +01:00 by adam · 8 comments
Owner

Originally created by @bluikko on GitHub (Apr 15, 2020).

Originally assigned to: @DanSheps on GitHub.

Environment

  • Python version: 3.6.8
  • NetBox version: 2.8.0

Steps to Reproduce

  1. Configure systemd with contrib/netbox.service.
  2. Make sure Redis takes a few seconds to start.
  3. systemd attempts to start NetBox before Redis has completed starting up and loading data and NetBox will error.

Expected Behavior

NetBox starts only after Redis is ready.

Observed Behavior

NetBox systemd unit file does not depend on Redis so it will not wait Redis to signal to systemd that it has completed startup.

Originally created by @bluikko on GitHub (Apr 15, 2020). Originally assigned to: @DanSheps on GitHub. <!-- NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED. This form is only for reproducible bugs. If you need assistance with NetBox installation, or if you have a general question, DO NOT open an issue. Instead, post to our mailing list: https://groups.google.com/forum/#!forum/netbox-discuss Please describe the environment in which you are running NetBox. Be sure that you are running an unmodified instance of the latest stable release before submitting a bug report, and that any plugins have been disabled. --> ### Environment * Python version: 3.6.8 * NetBox version: 2.8.0 <!-- Describe in detail the exact steps that someone else can take to reproduce this bug using the current stable release of NetBox. Begin with the creation of any necessary database objects and call out every operation being performed explicitly. If reporting a bug in the REST API, be sure to reconstruct the raw HTTP request(s) being made: Don't rely on a client library such as pynetbox. --> ### Steps to Reproduce 1. Configure systemd with `contrib/netbox.service`. 2. Make sure Redis takes a few seconds to start. 3. systemd attempts to start NetBox before Redis has completed starting up and loading data and NetBox will error. <!-- What did you expect to happen? --> ### Expected Behavior NetBox starts only after Redis is ready. <!-- What happened instead? --> ### Observed Behavior NetBox systemd unit file does not depend on Redis so it will not wait Redis to signal to systemd that it has completed startup.
adam closed this issue 2025-12-29 18:29:57 +01:00
Author
Owner

@kobayashi commented on GitHub (Apr 19, 2020):

Could you share your environment and error message to reproduce?

@kobayashi commented on GitHub (Apr 19, 2020): Could you share your environment and error message to reproduce?
Author
Owner

@bluikko commented on GitHub (Apr 20, 2020):

If Netbox is started before Redis is ready the error is seen:

Apr 15 10:00:00 netbox gunicorn[2497]: redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

This should be mitigated (possibly not completely fixed in older Redis, there is a bug up to at least Redis 5.0.6 that signal readiness too early) by starting after Redis and not the generic target. redis.conf needs supervised systemd option or alternatively on EL7 the stock redis.service unit file already starts Redis with --supervised systemd.

@bluikko commented on GitHub (Apr 20, 2020): If Netbox is started before Redis is ready the error is seen: ``` Apr 15 10:00:00 netbox gunicorn[2497]: redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory ``` This should be mitigated (possibly not completely fixed in older Redis, there is a bug up to at least Redis 5.0.6 that signal readiness too early) by starting after Redis and not the generic target. `redis.conf` needs `supervised systemd` option or alternatively on EL7 the stock `redis.service` unit file already starts Redis with `--supervised systemd`.
Author
Owner

@jeremystretch commented on GitHub (Apr 21, 2020):

Why are we concerned with Redis specifically and not, for example, PostgreSQL or network connectivity? PostgreSQL is obviously crucial, and any queued webhooks should not be sent until NetBox has network connectivity.

@jeremystretch commented on GitHub (Apr 21, 2020): Why are we concerned with Redis specifically and not, for example, PostgreSQL or network connectivity? PostgreSQL is obviously crucial, and any queued webhooks should not be sent until NetBox has network connectivity.
Author
Owner

@bluikko commented on GitHub (Apr 22, 2020):

I have not had problem with PostgreSQL or network connectivity. I do not know if for example pgsql can signal its readiness to systemd.
I do have the problem listed in the error message above where Redis takes its time to start.

@bluikko commented on GitHub (Apr 22, 2020): I have not had problem with PostgreSQL or network connectivity. I do not know if for example pgsql can signal its readiness to systemd. I do have the problem listed in the error message above where Redis takes its time to start.
Author
Owner

@DanSheps commented on GitHub (Apr 22, 2020):

I don't know if this is something we want to do, as redis does not have to be on the local NetBox server. Same with postgresql.

@DanSheps commented on GitHub (Apr 22, 2020): I don't know if this is something we want to do, as redis does not have to be on the local NetBox server. Same with postgresql.
Author
Owner

@jeremystretch commented on GitHub (Apr 22, 2020):

@DanSheps that's a very good point. IMO we should leave things as-is, and leave any modifications up to the discretion of the user.

@jeremystretch commented on GitHub (Apr 22, 2020): @DanSheps that's a very good point. IMO we should leave things as-is, and leave any modifications up to the discretion of the user.
Author
Owner

@DanSheps commented on GitHub (Apr 22, 2020):

IMO changing it to Wants= isn't going to break things, Wants= doesn't actually do anything if the wanted service doesn't start.

You need to use the Requires= for that and I don't think we want to get into that.

@DanSheps commented on GitHub (Apr 22, 2020): IMO changing it to Wants= isn't going to break things, Wants= doesn't actually do anything if the wanted service doesn't start. You need to use the Requires= for that and I don't think we want to get into that.
Author
Owner

@DanSheps commented on GitHub (Apr 27, 2020):

Given that this would not only require changing Wants to Requires, but also redis.conf changes to get the desired effect, we want to leave this as is and not make any changes.

If someone wants to make these changes on their own system that is not a problem but we should keep things simple here. network-online.target works for most of the cases.

@DanSheps commented on GitHub (Apr 27, 2020): Given that this would not only require changing Wants to Requires, but also redis.conf changes to get the desired effect, we want to leave this as is and not make any changes. If someone wants to make these changes on their own system that is not a problem but we should keep things simple here. network-online.target works for most of the cases.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#3572