starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Error 502 after gunicorn worker auto-restart #9039

New Issue
Closed
opened 2025-12-29 20:44:35 +01:00 by adam · 18 comments
adam commented 2025-12-29 20:44:35 +01:00
Owner
Copy Link

Originally created by @v0tti on GitHub (Jan 4, 2024).

Originally assigned to: @arthanson on GitHub.

Deployment Type

Self-hosted

NetBox Version

v3.7.0

Python Version

3.9

Steps to Reproduce

This is a know issue with gunicorn that was introduced with the update of the gunicorn version in NetBox v3.5.7. Steps to reproduce can be found at https://github.com/benoitc/gunicorn/issues/3038. Also some NetBox users are reporting these errors since v3.5.7 in that issue.

Expected Behavior

I do not expect to get an error 502 for common requests without an actual fault.

Observed Behavior

I get an error 502 when multiple sequential requests are sent to gunicorn with the default gunicorn version and configuration.

As there is not much happening with the gunicorn issue and pull request, I would like to suggest to downgrade gunicorn to version 20.1.0.

Originally created by @v0tti on GitHub (Jan 4, 2024). Originally assigned to: @arthanson on GitHub. ### Deployment Type Self-hosted ### NetBox Version v3.7.0 ### Python Version 3.9 ### Steps to Reproduce This is a know issue with gunicorn that was introduced with the update of the gunicorn version in NetBox v3.5.7. Steps to reproduce can be found at https://github.com/benoitc/gunicorn/issues/3038. Also some NetBox users are reporting these errors since v3.5.7 in that issue. ### Expected Behavior I do not expect to get an error 502 for common requests without an actual fault. ### Observed Behavior I get an error 502 when multiple sequential requests are sent to gunicorn with the default gunicorn version and configuration. As there is not much happening with the gunicorn issue and pull request, I would like to suggest to downgrade gunicorn to version 20.1.0.
adam added the type: bugstatus: acceptedseverity: low labels 2025-12-29 20:44:35 +01:00
adam closed this issue 2025-12-29 20:44:36 +01:00
adam commented 2025-12-29 20:44:36 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 4, 2024):

There are a few points to consider here:

  1. AFAICT this is a bug entirely contained in gunicorn itself, and not something that we can fix with a change to NetBox code.
  2. NetBox currently prescribes the installation of gunicorn v21.2.0, which is marked as the current stable release.
  3. The proposed previous release (20.1.0) is more than two years older than the current release, and would give up significant features and bug fixes, in addition (apparently) to Python 3.11 support.
  4. The current gunicorn version has been in place for roughly six months, and this is the first time the issue has surfaced as a bug. (I'll readily admit it's likely been discussed elsewhere by NetBox users and just not been on my radar.)
  5. Users are free to install whichever version of gunicorn they want, regardless of what NetBox prescribes.
  6. Downgrading gunicorn to an older release might break things (like Python 3.11 support).

All this is to say I'm not sure where I stand on the proposal. I think my biggest issue with it is that forcing a downgrade of gunicorn on the next NetBox release could potentially cause more problems for some users than the bug at issue here. This puts us in a difficult position.

@jeremystretch commented on GitHub (Jan 4, 2024): There are a few points to consider here: 1. AFAICT this is a bug entirely contained in gunicorn itself, and not something that we can fix with a change to NetBox code. 2. NetBox currently prescribes the installation of gunicorn v21.2.0, which is marked as the current stable release. 3. The proposed previous release (20.1.0) is more than two years older than the current release, and would give up significant features and bug fixes, in addition (apparently) to Python 3.11 support. 4. The current gunicorn version has been in place for roughly six months, and this is the first time the issue has surfaced as a bug. (I'll readily admit it's likely been discussed elsewhere by NetBox users and just not been on my radar.) 5. Users are free to install whichever version of gunicorn they want, regardless of what NetBox prescribes. 6. Downgrading gunicorn to an older release might break things (like Python 3.11 support). All this is to say I'm not sure where I stand on the proposal. I think my biggest issue with it is that forcing a downgrade of gunicorn on the next NetBox release could potentially cause more problems for some users than the bug at issue here. This puts us in a difficult position.
adam commented 2025-12-29 20:44:36 +01:00
Author
Owner
Copy Link

@v0tti commented on GitHub (Jan 4, 2024):

I fully understand that downgrading may not be an option, especially if we lose Python 3.11 support.
Maybe the existence of this problem could be mentioned in the docs?
I hope gunicorn will fix this bug soon, and if so, I think the dependency in NetBox should be updated.

@v0tti commented on GitHub (Jan 4, 2024): I fully understand that downgrading may not be an option, especially if we lose Python 3.11 support. Maybe the existence of this problem could be mentioned in the docs? I hope gunicorn will fix this bug soon, and if so, I think the dependency in NetBox should be updated.
adam commented 2025-12-29 20:44:36 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 4, 2024):

I suppose it's worth explicitly calling out the workaround for anyone who happens upon this issue: Gunicorn can be downgraded to v20.1.0 for an existing NetBox deployment using pip.

$ pip install gunicorn==20.1.0
Collecting gunicorn==20.1.0
  Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB)
Requirement already satisfied: setuptools>=3.0 in ./venv/lib/python3.10/site-packages (from gunicorn==20.1.0) (59.6.0)
Installing collected packages: gunicorn
  Attempting uninstall: gunicorn
    Found existing installation: gunicorn 21.2.0
    Uninstalling gunicorn-21.2.0:
      Successfully uninstalled gunicorn-21.2.0
Successfully installed gunicorn-20.1.0
@jeremystretch commented on GitHub (Jan 4, 2024): I suppose it's worth explicitly calling out the workaround for anyone who happens upon this issue: Gunicorn can be downgraded to v20.1.0 for an existing NetBox deployment using `pip`. ``` $ pip install gunicorn==20.1.0 Collecting gunicorn==20.1.0 Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB) Requirement already satisfied: setuptools>=3.0 in ./venv/lib/python3.10/site-packages (from gunicorn==20.1.0) (59.6.0) Installing collected packages: gunicorn Attempting uninstall: gunicorn Found existing installation: gunicorn 21.2.0 Uninstalling gunicorn-21.2.0: Successfully uninstalled gunicorn-21.2.0 Successfully installed gunicorn-20.1.0 ```
adam commented 2025-12-29 20:44:37 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (Jan 5, 2024):

Debian 12 (on which I plan to run all my NetBox 3.7.x setups) has Python 3.11, so I'm really interested in this issue. Currently trying to find out what the "Python 3.11 support" mentioned in Gunicorn 21.x actually means.

@markkuleinio commented on GitHub (Jan 5, 2024): Debian 12 (on which I plan to run all my NetBox 3.7.x setups) has Python 3.11, so I'm really interested in this issue. Currently trying to find out what the "Python 3.11 support" mentioned in Gunicorn 21.x actually means.
adam commented 2025-12-29 20:44:37 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 5, 2024):

FWIW I don't expect that earlier versions of gunicorn won't work on Python 3.11, just that they aren't specifically supported (or tested).

@jeremystretch commented on GitHub (Jan 5, 2024): FWIW I don't expect that earlier versions of gunicorn won't _work_ on Python 3.11, just that they aren't specifically supported (or tested).
adam commented 2025-12-29 20:44:37 +01:00
Author
Owner
Copy Link

@nomad-cyanide commented on GitHub (Jan 8, 2024):

I suppose it's worth explicitly calling out the workaround for anyone who happens upon this issue: Gunicorn can be downgraded to v20.1.0 for an existing NetBox deployment using pip.

$ pip install gunicorn==20.1.0
Collecting gunicorn==20.1.0
  Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB)
Requirement already satisfied: setuptools>=3.0 in ./venv/lib/python3.10/site-packages (from gunicorn==20.1.0) (59.6.0)
Installing collected packages: gunicorn
  Attempting uninstall: gunicorn
    Found existing installation: gunicorn 21.2.0
    Uninstalling gunicorn-21.2.0:
      Successfully uninstalled gunicorn-21.2.0
Successfully installed gunicorn-20.1.0

Would this work-around be a way forward for me, in this problem I have described here: https://github.com/netbox-community/netbox/discussions/14730 ?

@nomad-cyanide commented on GitHub (Jan 8, 2024): > I suppose it's worth explicitly calling out the workaround for anyone who happens upon this issue: Gunicorn can be downgraded to v20.1.0 for an existing NetBox deployment using `pip`. > > ``` > $ pip install gunicorn==20.1.0 > Collecting gunicorn==20.1.0 > Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB) > Requirement already satisfied: setuptools>=3.0 in ./venv/lib/python3.10/site-packages (from gunicorn==20.1.0) (59.6.0) > Installing collected packages: gunicorn > Attempting uninstall: gunicorn > Found existing installation: gunicorn 21.2.0 > Uninstalling gunicorn-21.2.0: > Successfully uninstalled gunicorn-21.2.0 > Successfully installed gunicorn-20.1.0 > ``` Would this work-around be a way forward for me, in this problem I have described here: https://github.com/netbox-community/netbox/discussions/14730 ?
adam commented 2025-12-29 20:44:37 +01:00
Author
Owner
Copy Link

@v0tti commented on GitHub (Jan 8, 2024):

Would this work-around be a way forward for me, in this problem I have described here: #14730 ?

Maybe. You should look into your gunicorn logs in systemd and check if you see something like

Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Autorestarting worker after current request.
Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Worker exiting (pid: 579373)
Aug 10 13:00:07 hostname gunicorn[648814]: [2023-08-10 13:00:07 +0000] [648814] [INFO] Booting worker with pid: 648814

there.

@v0tti commented on GitHub (Jan 8, 2024): > Would this work-around be a way forward for me, in this problem I have described here: #14730 ? Maybe. You should look into your `gunicorn` logs in systemd and check if you see something like ``` Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Autorestarting worker after current request. Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Worker exiting (pid: 579373) Aug 10 13:00:07 hostname gunicorn[648814]: [2023-08-10 13:00:07 +0000] [648814] [INFO] Booting worker with pid: 648814 ``` there.
adam commented 2025-12-29 20:44:37 +01:00
Author
Owner
Copy Link

@nomad-cyanide commented on GitHub (Jan 8, 2024):

Would this work-around be a way forward for me, in this problem I have described here: #14730 ?

Maybe. You should look into your gunicorn logs in systemd and check if you see something like

Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Autorestarting worker after current request.
Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Worker exiting (pid: 579373)
Aug 10 13:00:07 hostname gunicorn[648814]: [2023-08-10 13:00:07 +0000] [648814] [INFO] Booting worker with pid: 648814

there.

And I do.

Jan 08 14:39:25 30001vmnb02-prod gunicorn[1129991]: [2024-01-08 14:39:25 +0100] [1129991] [INFO] Autorestarting worker after current request.
Jan 08 14:39:25 30001vmnb02-prod gunicorn[1129991]: [2024-01-08 14:39:25 +0100] [1129991] [INFO] Worker exiting (pid: 1129991)
Jan 08 14:39:26 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:39:26 +0100] [1139845] [INFO] Booting worker with pid: 1139845
Jan 08 14:44:11 30001vmnb02-prod gunicorn[1129962]: [2024-01-08 14:44:11 +0100] [1129962] [INFO] Autorestarting worker after current request.
Jan 08 14:44:11 30001vmnb02-prod gunicorn[1129962]: [2024-01-08 14:44:11 +0100] [1129962] [INFO] Worker exiting (pid: 1129962)
Jan 08 14:44:11 30001vmnb02-prod gunicorn[1139926]: [2024-01-08 14:44:11 +0100] [1139926] [INFO] Booting worker with pid: 1139926
Jan 08 14:46:09 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:46:09 +0100] [1139845] [INFO] Autorestarting worker after current request.
Jan 08 14:46:09 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:46:09 +0100] [1139845] [INFO] Worker exiting (pid: 1139845)
Jan 08 14:46:10 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:46:10 +0100] [1139967] [INFO] Booting worker with pid: 1139967
Jan 08 14:50:27 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:50:27 +0100] [1139967] [INFO] Autorestarting worker after current request.
Jan 08 14:50:27 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:50:27 +0100] [1139967] [INFO] Worker exiting (pid: 1139967)
Jan 08 14:50:28 30001vmnb02-prod gunicorn[1140032]: [2024-01-08 14:50:28 +0100] [1140032] [INFO] Booting worker with pid: 1140032
Jan 08 14:50:45 30001vmnb02-prod gunicorn[1129800]: [2024-01-08 14:50:45 +0100] [1129800] [INFO] Autorestarting worker after current request.
Jan 08 14:50:45 30001vmnb02-prod gunicorn[1129800]: [2024-01-08 14:50:45 +0100] [1129800] [INFO] Worker exiting (pid: 1129800)
Jan 08 14:50:46 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:50:46 +0100] [1140042] [INFO] Booting worker with pid: 1140042
Jan 08 14:55:20 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:55:20 +0100] [1140042] [INFO] Autorestarting worker after current request.
Jan 08 14:55:20 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:55:20 +0100] [1140042] [INFO] Worker exiting (pid: 1140042)
Jan 08 14:55:21 30001vmnb02-prod gunicorn[1140117]: [2024-01-08 14:55:21 +0100] [1140117] [INFO] Booting worker with pid: 1140117
Jan 08 14:56:39 30001vmnb02-prod gunicorn[1107507]: [2024-01-08 14:56:39 +0100] [1107507] [INFO] Autorestarting worker after current request.
Jan 08 14:56:39 30001vmnb02-prod gunicorn[1107507]: [2024-01-08 14:56:39 +0100] [1107507] [INFO] Worker exiting (pid: 1107507)
Jan 08 14:56:40 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 14:56:40 +0100] [1140142] [INFO] Booting worker with pid: 1140142
Jan 08 15:01:20 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 15:01:20 +0100] [1140142] [INFO] Autorestarting worker after current request.
Jan 08 15:01:21 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 15:01:21 +0100] [1140142] [INFO] Worker exiting (pid: 1140142)
Jan 08 15:01:21 30001vmnb02-prod gunicorn[1140221]: [2024-01-08 15:01:21 +0100] [1140221] [INFO] Booting worker with pid: 1140221

If I'm honest, gunicorn 21.2.0 doesn't really seem production ready, if this is the behavior we can expect.

@nomad-cyanide commented on GitHub (Jan 8, 2024): > > Would this work-around be a way forward for me, in this problem I have described here: #14730 ? > > Maybe. You should look into your `gunicorn` logs in systemd and check if you see something like > > ``` > Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Autorestarting worker after current request. > Aug 10 13:00:06 hostname gunicorn[579373]: [2023-08-10 13:00:06 +0000] [579373] [INFO] Worker exiting (pid: 579373) > Aug 10 13:00:07 hostname gunicorn[648814]: [2023-08-10 13:00:07 +0000] [648814] [INFO] Booting worker with pid: 648814 > ``` > > there. And I do. > Jan 08 14:39:25 30001vmnb02-prod gunicorn[1129991]: [2024-01-08 14:39:25 +0100] [1129991] [INFO] Autorestarting worker after current request. Jan 08 14:39:25 30001vmnb02-prod gunicorn[1129991]: [2024-01-08 14:39:25 +0100] [1129991] [INFO] Worker exiting (pid: 1129991) Jan 08 14:39:26 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:39:26 +0100] [1139845] [INFO] Booting worker with pid: 1139845 Jan 08 14:44:11 30001vmnb02-prod gunicorn[1129962]: [2024-01-08 14:44:11 +0100] [1129962] [INFO] Autorestarting worker after current request. Jan 08 14:44:11 30001vmnb02-prod gunicorn[1129962]: [2024-01-08 14:44:11 +0100] [1129962] [INFO] Worker exiting (pid: 1129962) Jan 08 14:44:11 30001vmnb02-prod gunicorn[1139926]: [2024-01-08 14:44:11 +0100] [1139926] [INFO] Booting worker with pid: 1139926 Jan 08 14:46:09 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:46:09 +0100] [1139845] [INFO] Autorestarting worker after current request. Jan 08 14:46:09 30001vmnb02-prod gunicorn[1139845]: [2024-01-08 14:46:09 +0100] [1139845] [INFO] Worker exiting (pid: 1139845) Jan 08 14:46:10 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:46:10 +0100] [1139967] [INFO] Booting worker with pid: 1139967 Jan 08 14:50:27 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:50:27 +0100] [1139967] [INFO] Autorestarting worker after current request. Jan 08 14:50:27 30001vmnb02-prod gunicorn[1139967]: [2024-01-08 14:50:27 +0100] [1139967] [INFO] Worker exiting (pid: 1139967) Jan 08 14:50:28 30001vmnb02-prod gunicorn[1140032]: [2024-01-08 14:50:28 +0100] [1140032] [INFO] Booting worker with pid: 1140032 Jan 08 14:50:45 30001vmnb02-prod gunicorn[1129800]: [2024-01-08 14:50:45 +0100] [1129800] [INFO] Autorestarting worker after current request. Jan 08 14:50:45 30001vmnb02-prod gunicorn[1129800]: [2024-01-08 14:50:45 +0100] [1129800] [INFO] Worker exiting (pid: 1129800) Jan 08 14:50:46 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:50:46 +0100] [1140042] [INFO] Booting worker with pid: 1140042 Jan 08 14:55:20 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:55:20 +0100] [1140042] [INFO] Autorestarting worker after current request. Jan 08 14:55:20 30001vmnb02-prod gunicorn[1140042]: [2024-01-08 14:55:20 +0100] [1140042] [INFO] Worker exiting (pid: 1140042) Jan 08 14:55:21 30001vmnb02-prod gunicorn[1140117]: [2024-01-08 14:55:21 +0100] [1140117] [INFO] Booting worker with pid: 1140117 Jan 08 14:56:39 30001vmnb02-prod gunicorn[1107507]: [2024-01-08 14:56:39 +0100] [1107507] [INFO] Autorestarting worker after current request. Jan 08 14:56:39 30001vmnb02-prod gunicorn[1107507]: [2024-01-08 14:56:39 +0100] [1107507] [INFO] Worker exiting (pid: 1107507) Jan 08 14:56:40 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 14:56:40 +0100] [1140142] [INFO] Booting worker with pid: 1140142 Jan 08 15:01:20 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 15:01:20 +0100] [1140142] [INFO] Autorestarting worker after current request. Jan 08 15:01:21 30001vmnb02-prod gunicorn[1140142]: [2024-01-08 15:01:21 +0100] [1140142] [INFO] Worker exiting (pid: 1140142) Jan 08 15:01:21 30001vmnb02-prod gunicorn[1140221]: [2024-01-08 15:01:21 +0100] [1140221] [INFO] Booting worker with pid: 1140221 If I'm honest, gunicorn 21.2.0 doesn't really seem production ready, if this is the behavior we can expect.
adam commented 2025-12-29 20:44:38 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (Jan 9, 2024):

Note that gunicorn's autorestart itself is expected and desired (as investigated in earlier NetBox issues, search here for max_requests), but it behaves differently in gunicorn 21.2.0, causing resets in connections.

In any case, supporting the issue at https://github.com/benoitc/gunicorn/issues/3038 is probably the way to affect gunicorn.

@markkuleinio commented on GitHub (Jan 9, 2024): Note that gunicorn's autorestart itself is expected and desired (as investigated in earlier NetBox issues, search here for max_requests), but it behaves differently in gunicorn 21.2.0, causing resets in connections. In any case, supporting the issue at https://github.com/benoitc/gunicorn/issues/3038 is probably the way to affect gunicorn.
adam commented 2025-12-29 20:44:38 +01:00
Author
Owner
Copy Link

@v0tti commented on GitHub (Jan 9, 2024):

Note that gunicorn's autorestart itself is expected and desired (as investigated in earlier NetBox issues, search here for max_requests), but it behaves differently in gunicorn 21.2.0, causing resets in connections.

Sure, I should have added that these autorestarts should correlate with the 502s that are observed.

@v0tti commented on GitHub (Jan 9, 2024): > Note that gunicorn's autorestart itself is expected and desired (as investigated in earlier NetBox issues, search here for max_requests), but it behaves differently in gunicorn 21.2.0, causing resets in connections. Sure, I should have added that these autorestarts should correlate with the 502s that are observed.
adam commented 2025-12-29 20:44:38 +01:00
Author
Owner
Copy Link

@nomad-cyanide commented on GitHub (Jan 9, 2024):

I have been testing a few things and this is my finding.
It doesn't help if I decrease the max_requests setting. It just fails sooner.
it doesn't help if I increase the max_requests setting either. It still fails at some point.
When it fails a log-entry about restarting shows up simultaneously:

Example:

an 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Autorestarting worker after current request.
Jan 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Worker exiting (pid: 106082)
Jan 09 13:20:08 30001vmnb02-test gunicorn[106256]: [2024-01-09 13:20:08 +0100] [106256] [INFO] Booting worker with pid: 106256

If I set the max_requests to 0, thus disabling it, my scripts work. without error. But is this preferable to having gunicorn processes restart regularly. I suppose it would start consuming memory, if it has memory-leak errors that is.

Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?

@nomad-cyanide commented on GitHub (Jan 9, 2024): I have been testing a few things and this is my finding. It doesn't help if I decrease the max_requests setting. It just fails sooner. it doesn't help if I increase the max_requests setting either. It still fails at some point. When it fails a log-entry about restarting shows up simultaneously: Example: > an 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Autorestarting worker after current request. Jan 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Worker exiting (pid: 106082) Jan 09 13:20:08 30001vmnb02-test gunicorn[106256]: [2024-01-09 13:20:08 +0100] [106256] [INFO] Booting worker with pid: 106256 If I set the max_requests to 0, thus disabling it, my scripts work. without error. But is this preferable to having gunicorn processes restart regularly. I suppose it would start consuming memory, if it has memory-leak errors that is. Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?
adam commented 2025-12-29 20:44:38 +01:00
Author
Owner
Copy Link

@nomad-cyanide commented on GitHub (Jan 11, 2024):

If I set the max_requests to 0, thus disabling it, my scripts work. without error. But is this preferable to having gunicorn processes restart regularly. I suppose it would start consuming memory, if it has memory-leak errors that is.

Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?

I have come to the conclusion, that rather than downgrade gunicorn of maybe loose some necessary features, I will go ahead with max_requests set i 0 and if memory usage becomes an issue on the server I will set up a scheduled job that restarts the worker processes with this command:

ps -aux | grep venv/bin/gunicorn | grep Ss | awk '{ system("kill -HUP " $2 )}'
@nomad-cyanide commented on GitHub (Jan 11, 2024): > If I set the max_requests to 0, thus disabling it, my scripts work. without error. But is this preferable to having gunicorn processes restart regularly. I suppose it would start consuming memory, if it has memory-leak errors that is. > > Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick? I have come to the conclusion, that rather than downgrade gunicorn of maybe loose some necessary features, I will go ahead with max_requests set i 0 and if memory usage becomes an issue on the server I will set up a scheduled job that restarts the worker processes with this command: ``` ps -aux | grep venv/bin/gunicorn | grep Ss | awk '{ system("kill -HUP " $2 )}' ```
adam commented 2025-12-29 20:44:39 +01:00
Author
Owner
Copy Link

@MANT5149 commented on GitHub (Jan 11, 2024):

We ran into this issue as well. We have several scripts using the NetBox API, and they began having intermittent issues after the gunicorn upgrade to 21.2.0. We've been downgrading to 20.1.0 which works flawlessly. Recently did an in-place upgrade from Debian 11 to 12 and everything is still working smooth with 20.1.0 and NB 3.7.0.

@MANT5149 commented on GitHub (Jan 11, 2024): We ran into this issue as well. We have several scripts using the NetBox API, and they began having intermittent issues after the gunicorn upgrade to 21.2.0. We've been downgrading to 20.1.0 which works flawlessly. Recently did an in-place upgrade from Debian 11 to 12 and everything is still working smooth with 20.1.0 and NB 3.7.0.
adam commented 2025-12-29 20:44:39 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (Jan 11, 2024):

I'd appreciate observations:

  • commenting out max_requests in gunicorn.py (= no restarts) + restarting netbox+netbox-rq
  • observing the memory consumption of the gunicorn processes over time with NetBox 3.7.0.

Unfortunately my current prod NetBoxes are 3.3.x so its no use doing this there, and my 3.7.0 test boxes don't get nearly enough of legitime use to tell if the gunicorn instances are still leaking memory or not.

@markkuleinio commented on GitHub (Jan 11, 2024): I'd appreciate observations: - commenting out `max_requests` in gunicorn.py (= no restarts) + restarting netbox+netbox-rq - observing the memory consumption of the gunicorn processes over time with NetBox 3.7.0. Unfortunately my current prod NetBoxes are 3.3.x so its no use doing this there, and my 3.7.0 test boxes don't get nearly enough of legitime use to tell if the gunicorn instances are still leaking memory or not.
adam commented 2025-12-29 20:44:39 +01:00
Author
Owner
Copy Link

@tyler-8 commented on GitHub (Jan 11, 2024):

I'll just add that I've been running NetBox using uwsgi for years now and it's worked without any noticeable issue.

@tyler-8 commented on GitHub (Jan 11, 2024): I'll just add that I've been running NetBox using `uwsgi` for years now and it's worked without any noticeable issue.
adam commented 2025-12-29 20:44:39 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (Jan 11, 2024):

I see NetBox maintainers also mentioned uWSGI in their meeting: #14777

Do you want to share your uWSGI configs and relevant systemd (etc) files? It's totally new area for me, even though I know how gunicorn+systemd combo works.

@markkuleinio commented on GitHub (Jan 11, 2024): I see NetBox maintainers also mentioned uWSGI in their meeting: #14777 Do you want to share your uWSGI configs and relevant systemd (etc) files? It's totally new area for me, even though I know how gunicorn+systemd combo works.
adam commented 2025-12-29 20:44:39 +01:00
Author
Owner
Copy Link

@markkuleinio commented on GitHub (Jan 12, 2024):

Discussion #14780 opened about uWSGI configuration

@markkuleinio commented on GitHub (Jan 12, 2024): Discussion #14780 opened about uWSGI configuration
adam commented 2025-12-29 20:44:40 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Jan 25, 2024):

We've added a note in the installation docs for now. Will keep an eye on this; hoping for a new gunicorn release soon. 🤞

@jeremystretch commented on GitHub (Jan 25, 2024): We've added a note in the installation docs for now. Will keep an eye on this; hoping for a new gunicorn release soon. :crossed_fingers:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label severity: low status: accepted type: bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#9039
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?