starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Add "die-on-term" to fix uWSGI shutdown #9629

New Issue
Closed
opened 2025-12-29 21:20:02 +01:00 by adam · 6 comments
adam commented 2025-12-29 21:20:02 +01:00
Owner
Copy Link

Originally created by @markkuleinio on GitHub (May 8, 2024).

Originally assigned to: @markkuleinio on GitHub.

Change Type

Addition

Area

Installation/upgrade

Proposed Changes

When using "systemctl stop netbox", systemd sends SIGTERM to the process (see https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillSignal=).

However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means:

brutally reload all the workers and the master process

= it is not a "stop" signal for uWSGI.

(... "this bad choice has been fixed in uWSGI 2.1" as mentioned by the maintainer in https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html, but it is apparently not happening any time soon.)

Thus, I'm proposing adding die-on-term = true to be added in the NetBox-default uwsgi.ini.

That makes the default systemctl behavior to work with uWSGI as well, instead of changing the kill signal in netbox.service, to keep it simplier for also Gunicorn use.

With that addition NetBox shuts down correctly with "systemctl stop netbox":

May 08 18:42:53 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service...
May 08 18:42:53 netbox-lab uwsgi[572422]: SIGINT/SIGTERM received...killing workers...
May 08 18:42:54 netbox-lab uwsgi[572422]: worker 1 buried after 1 seconds
May 08 18:42:54 netbox-lab uwsgi[572422]: goodbye to uWSGI.
May 08 18:42:54 netbox-lab systemd[1]: netbox.service: Deactivated successfully.
May 08 18:42:54 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service.

Note: With the current default configuration, "systemctl stop netbox" just happens to currently work, because:

May 08 18:33:06 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service...
May 08 18:33:06 netbox-lab uwsgi[572209]: ...brutally killing workers...
May 08 18:33:07 netbox-lab uwsgi[572209]: worker 1 buried after 1 seconds
May 08 18:33:07 netbox-lab uwsgi[572209]: binary reloading uWSGI...
May 08 18:33:07 netbox-lab uwsgi[572209]: chdir() to /opt/netbox
May 08 18:33:07 netbox-lab uwsgi[572209]: closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
May 08 18:33:07 netbox-lab uwsgi[572209]: found fd 3 mapped to socket 0 (127.0.0.1:8001)
May 08 18:33:07 netbox-lab uwsgi[572209]: running /usr/bin/python3.11
May 08 18:33:07 netbox-lab uwsgi[572209]: Traceback (most recent call last):
May 08 18:33:07 netbox-lab uwsgi[572209]:   File "/opt/netbox/venv/bin/uwsgi", line 5, in <module>
May 08 18:33:07 netbox-lab uwsgi[572209]:     from pyuwsgi import run
May 08 18:33:07 netbox-lab uwsgi[572209]: ModuleNotFoundError: No module named 'pyuwsgi'
May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Main process exited, code=exited, status=1/FAILURE
May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Failed with result 'exit-code'.
May 08 18:33:07 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service.

= it tries to reload uWSGI but a module import error prevents that happening.

I don't currently know why that happens, I haven't used pyuwsgi before, and this module import problem does not occur with uwsgi package. (But the "reload vs. stop" problem is still valid with uwsgi as well, so die-on-term = true is needed anyway.)

Originally created by @markkuleinio on GitHub (May 8, 2024). Originally assigned to: @markkuleinio on GitHub. ### Change Type Addition ### Area Installation/upgrade ### Proposed Changes When using "`systemctl stop netbox`", systemd sends **SIGTERM** to the process (see https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillSignal=). However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means: > brutally **reload** all the workers and the master process = it is not a "stop" signal for uWSGI. (_... "this bad choice has been fixed in uWSGI 2.1" as mentioned by the maintainer in https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html, but it is apparently not happening any time soon._) Thus, I'm proposing adding `die-on-term = true` to be added in the NetBox-default `uwsgi.ini`. That makes the default `systemctl` behavior to work with uWSGI as well, instead of changing the kill signal in `netbox.service`, to keep it simplier for also Gunicorn use. With that addition NetBox shuts down correctly with "`systemctl stop netbox`": ``` May 08 18:42:53 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service... May 08 18:42:53 netbox-lab uwsgi[572422]: SIGINT/SIGTERM received...killing workers... May 08 18:42:54 netbox-lab uwsgi[572422]: worker 1 buried after 1 seconds May 08 18:42:54 netbox-lab uwsgi[572422]: goodbye to uWSGI. May 08 18:42:54 netbox-lab systemd[1]: netbox.service: Deactivated successfully. May 08 18:42:54 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service. ``` --- **Note:** With the current default configuration, "systemctl stop netbox" _just happens to currently work_, because: ``` May 08 18:33:06 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service... May 08 18:33:06 netbox-lab uwsgi[572209]: ...brutally killing workers... May 08 18:33:07 netbox-lab uwsgi[572209]: worker 1 buried after 1 seconds May 08 18:33:07 netbox-lab uwsgi[572209]: binary reloading uWSGI... May 08 18:33:07 netbox-lab uwsgi[572209]: chdir() to /opt/netbox May 08 18:33:07 netbox-lab uwsgi[572209]: closing all non-uwsgi socket fds > 2 (max_fd = 1024)... May 08 18:33:07 netbox-lab uwsgi[572209]: found fd 3 mapped to socket 0 (127.0.0.1:8001) May 08 18:33:07 netbox-lab uwsgi[572209]: running /usr/bin/python3.11 May 08 18:33:07 netbox-lab uwsgi[572209]: Traceback (most recent call last): May 08 18:33:07 netbox-lab uwsgi[572209]: File "/opt/netbox/venv/bin/uwsgi", line 5, in <module> May 08 18:33:07 netbox-lab uwsgi[572209]: from pyuwsgi import run May 08 18:33:07 netbox-lab uwsgi[572209]: ModuleNotFoundError: No module named 'pyuwsgi' May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Main process exited, code=exited, status=1/FAILURE May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Failed with result 'exit-code'. May 08 18:33:07 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service. ``` = it tries to reload uWSGI but a module import error prevents that happening. I don't currently know why that happens, I haven't used `pyuwsgi` before, and this module import problem does not occur with `uwsgi` package. (But the "reload vs. stop" problem is still valid with `uwsgi` as well, so `die-on-term = true` is needed anyway.)
adam added the status: acceptedtype: documentation labels 2025-12-29 21:20:02 +01:00
adam closed this issue 2025-12-29 21:20:02 +01:00
adam commented 2025-12-29 21:20:03 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (May 8, 2024):

I'm willing to provide the PR for this change.

@markkuleinio commented on GitHub (May 8, 2024): I'm willing to provide the PR for this change.
adam commented 2025-12-29 21:20:03 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (May 8, 2024):

Writing this down here:

With pyuwsgi:

May 08 18:55:49 netbox-lab uwsgi[572660]: *** Starting uWSGI 2.0.23 (64bit) on [Wed May  8 18:55:49 2024] ***
May 08 18:55:49 netbox-lab uwsgi[572660]: detected binary path: /usr/bin/python3.11

With uwsgi:

May 08 19:02:59 netbox-lab uwsgi[573360]: *** Starting uWSGI 2.0.25.1 (64bit) on [Wed May  8 19:02:59 2024] ***
May 08 19:02:59 netbox-lab uwsgi[573360]: detected binary path: /opt/netbox/venv/bin/uwsgi

Looks like pyuwsgi somehow finds itself to be the system-default Python binary, instead of discovering the virtual environment, and maybe thus the module import fails whenever a reload is attempted.

I didn't spot any relevant commit diffs between uWSGI 2.0.23 (which latest pyuwsgi still uses) and 2.0.25.1.

@markkuleinio commented on GitHub (May 8, 2024): Writing this down here: With `pyuwsgi`: ``` May 08 18:55:49 netbox-lab uwsgi[572660]: *** Starting uWSGI 2.0.23 (64bit) on [Wed May 8 18:55:49 2024] *** May 08 18:55:49 netbox-lab uwsgi[572660]: detected binary path: /usr/bin/python3.11 ``` With `uwsgi`: ``` May 08 19:02:59 netbox-lab uwsgi[573360]: *** Starting uWSGI 2.0.25.1 (64bit) on [Wed May 8 19:02:59 2024] *** May 08 19:02:59 netbox-lab uwsgi[573360]: detected binary path: /opt/netbox/venv/bin/uwsgi ``` Looks like `pyuwsgi` somehow finds itself to be the system-default Python binary, instead of discovering the virtual environment, and maybe thus the module import fails whenever a reload is attempted. I didn't spot any relevant commit diffs between uWSGI 2.0.23 (which latest `pyuwsgi` still uses) and 2.0.25.1.
adam commented 2025-12-29 21:20:04 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (May 8, 2024):

However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means:

Can I just say how utterly stupid that is since SIGTERM is also the default signal from kill...

@DanSheps commented on GitHub (May 8, 2024): > However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means: Can I just say how utterly stupid that is since SIGTERM is also the default signal from `kill`...
adam commented 2025-12-29 21:20:04 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (May 8, 2024):

Can I just say how utterly stupid that is since SIGTERM is also the default signal from kill...

--> Very clever indeed: you can reload uWSGI by just using kill <pid>, not needing any options!

😉

@markkuleinio commented on GitHub (May 8, 2024): > Can I just say how utterly stupid that is since SIGTERM is also the default signal from `kill`... --> Very clever indeed: you can reload uWSGI by just using `kill <pid>`, not needing any options! 😉
adam commented 2025-12-29 21:20:05 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (May 8, 2024):

Fun stuff. All yours @markkuleinio. Thanks for staying on top of these uWSGI items!

@jeremystretch commented on GitHub (May 8, 2024): Fun stuff. All yours @markkuleinio. Thanks for staying on top of these uWSGI items!
adam commented 2025-12-29 21:20:05 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (May 9, 2024):

FYI, https://github.com/lincolnloop/pyuwsgi-wheels/issues/23 opened about the pyuwsgi reload problem.

@markkuleinio commented on GitHub (May 9, 2024): FYI, https://github.com/lincolnloop/pyuwsgi-wheels/issues/23 opened about the `pyuwsgi` reload problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: accepted type: documentation
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#9629
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?