starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Netbox Consuming 100% of CPU over time with LDAP authentication enabled #4150

New Issue
Closed
opened 2025-12-29 18:33:28 +01:00 by adam · 9 comments
adam commented 2025-12-29 18:33:28 +01:00
Owner
Copy Link

Originally created by @nniehoff on GitHub (Sep 29, 2020).

Environment

  • Python version: 3.7.6
  • NetBox version: 2.9.4

Steps to Reproduce

  1. Configure LDAP authentication, following the documentation here using a Microsoft Active Directory server as the LDAP server. Some key configurations to note:
    1a. Our AUTH_LDAP_SERVER_URI is an ldaps endpoint.
    1b. AUTH_LDAP_GROUP_TYPE is set to NestedActiveDirectoryGroupType()
  2. Log on/Log off several times over a several hour period

Expected Behavior

Netbox should run with no noticeable increase in CPU usage over time.

Observed Behavior

Over the course of several (appx 8 in our case) hours the CPU usage of uwsgi serving netbox increases gradually over time until it is using 100% of the CPU.

Originally created by @nniehoff on GitHub (Sep 29, 2020). <!-- NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED. This form is only for reproducible bugs. If you need assistance with NetBox installation, or if you have a general question, DO NOT open an issue. Instead, post to our mailing list: https://groups.google.com/forum/#!forum/netbox-discuss Please describe the environment in which you are running NetBox. Be sure that you are running an unmodified instance of the latest stable release before submitting a bug report, and that any plugins have been disabled. --> ### Environment * Python version: 3.7.6 * NetBox version: 2.9.4 <!-- Describe in detail the exact steps that someone else can take to reproduce this bug using the current stable release of NetBox. Begin with the creation of any necessary database objects and call out every operation being performed explicitly. If reporting a bug in the REST API, be sure to reconstruct the raw HTTP request(s) being made: Don't rely on a client library such as pynetbox. --> ### Steps to Reproduce 1. Configure LDAP authentication, following the documentation [here](https://netbox.readthedocs.io/en/stable/installation/6-ldap/) using a Microsoft Active Directory server as the LDAP server. Some key configurations to note: 1a. Our `AUTH_LDAP_SERVER_URI` is an ldaps endpoint. 1b. `AUTH_LDAP_GROUP_TYPE` is set to `NestedActiveDirectoryGroupType()` 2. Log on/Log off several times over a several hour period <!-- What did you expect to happen? --> ### Expected Behavior Netbox should run with no noticeable increase in CPU usage over time. <!-- What happened instead? --> ### Observed Behavior Over the course of several (appx 8 in our case) hours the CPU usage of uwsgi serving netbox increases gradually over time until it is using 100% of the CPU.
adam added the status: duplicate label 2025-12-29 18:33:28 +01:00
adam closed this issue 2025-12-29 18:33:28 +01:00
adam commented 2025-12-29 18:33:28 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Sep 29, 2020):

  1. This appears to be a duplicate of #5192.
  2. If it's not, you'll need to provide far more detail than "Configure LDAP authentication." Sharing your complete configuration and noting what you're using for LDAP authentication would be a good start.
@jeremystretch commented on GitHub (Sep 29, 2020): 1. This appears to be a duplicate of #5192. 2. If it's not, you'll need to provide _far_ more detail than "Configure LDAP authentication." Sharing your complete configuration and noting what you're using for LDAP authentication would be a good start.
adam commented 2025-12-29 18:33:28 +01:00
Author
Owner
Copy Link

@nniehoff commented on GitHub (Sep 29, 2020):

To be more explicit about our LDAP configuration the entire contents are here with some redactions:

import os
import logging

from django_auth_ldap.config import LDAPGroupQuery, LDAPSearch, NestedActiveDirectoryGroupType
import ldap

# Instantiate logger
my_logger = logging.getLogger('django_auth_ldap')


# Server URI
# AUTH_LDAP_SERVER_URI = "ldaps://ad1-ldaps.example.com"
AUTH_LDAP_SERVER_URI = "ldaps://10.10.10.10"

# The following may be needed if you are binding to Active Directory.
AUTH_LDAP_CONNECTION_OPTIONS = {ldap.OPT_REFERRALS: 0}

# Set the DN and password for the NetBox service account.
AUTH_LDAP_BIND_DN = f"{os.environ.get('LDAP_USERNAME', '')}@example.com"
AUTH_LDAP_BIND_PASSWORD = os.environ.get("LDAP_PASSWORD", "")

# Include this setting if you want to ignore certificate errors. This might be needed to accept a self-signed cert.
# Note that this is a NetBox-specific setting which sets:
#     ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
LDAP_IGNORE_CERT_ERRORS = True

# This search matches users with the sAMAccountName equal to the provided username. This is required if the user's
# username is not in their DN (Active Directory).
AUTH_LDAP_USER_SEARCH = LDAPSearch(
    "dc=example,dc=com", ldap.SCOPE_SUBTREE, "(&(objectClass=user)(sAMAccountName=%(user)s))"
)

# If a user's DN is producible from their username, we don't need to search.
# AUTH_LDAP_USER_DN_TEMPLATE = "sAMAccountName=%(user)s,dc=example,dc=com"

# You can map user attributes to Django attributes as so.
AUTH_LDAP_USER_ATTR_MAP = {
    "first_name": "givenName",
    "last_name": "sn",
    "email": "mail",
}

# This search ought to return all groups to which the user belongs. django_auth_ldap uses this to determine group
# hierarchy.
AUTH_LDAP_GROUP_SEARCH = LDAPSearch(
    "dc=example,dc=com", ldap.SCOPE_SUBTREE, "(objectClass=group)"
)
AUTH_LDAP_GROUP_TYPE = NestedActiveDirectoryGroupType()

# Define a group required to login.
# AUTH_LDAP_REQUIRE_GROUP = "CN=Netbox-ReadOnly,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure"

# Mirror LDAP group assignments.
AUTH_LDAP_MIRROR_GROUPS = True

# Define special user types using groups. Exercise great caution when assigning superuser status.
AUTH_LDAP_USER_FLAGS_BY_GROUP = {
    "is_active": (
        LDAPGroupQuery("CN=Netbox-ReadOnly,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com") |
        LDAPGroupQuery("CN=Netbox-StandardAccess,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com") |
        LDAPGroupQuery("CN=Netbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com")
    ),
    "is_staff": "CN=Netbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com",
    "is_superuser": "CN=DNetbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com",
}

# For more granular permissions, we can map LDAP groups to Django groups.
AUTH_LDAP_FIND_GROUP_PERMS = True

# Cache groups for one hour to reduce LDAP traffic
AUTH_LDAP_CACHE_TIMEOUT = 3600
@nniehoff commented on GitHub (Sep 29, 2020): To be more explicit about our LDAP configuration the entire contents are here with some redactions: ``` import os import logging from django_auth_ldap.config import LDAPGroupQuery, LDAPSearch, NestedActiveDirectoryGroupType import ldap # Instantiate logger my_logger = logging.getLogger('django_auth_ldap') # Server URI # AUTH_LDAP_SERVER_URI = "ldaps://ad1-ldaps.example.com" AUTH_LDAP_SERVER_URI = "ldaps://10.10.10.10" # The following may be needed if you are binding to Active Directory. AUTH_LDAP_CONNECTION_OPTIONS = {ldap.OPT_REFERRALS: 0} # Set the DN and password for the NetBox service account. AUTH_LDAP_BIND_DN = f"{os.environ.get('LDAP_USERNAME', '')}@example.com" AUTH_LDAP_BIND_PASSWORD = os.environ.get("LDAP_PASSWORD", "") # Include this setting if you want to ignore certificate errors. This might be needed to accept a self-signed cert. # Note that this is a NetBox-specific setting which sets: # ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER) LDAP_IGNORE_CERT_ERRORS = True # This search matches users with the sAMAccountName equal to the provided username. This is required if the user's # username is not in their DN (Active Directory). AUTH_LDAP_USER_SEARCH = LDAPSearch( "dc=example,dc=com", ldap.SCOPE_SUBTREE, "(&(objectClass=user)(sAMAccountName=%(user)s))" ) # If a user's DN is producible from their username, we don't need to search. # AUTH_LDAP_USER_DN_TEMPLATE = "sAMAccountName=%(user)s,dc=example,dc=com" # You can map user attributes to Django attributes as so. AUTH_LDAP_USER_ATTR_MAP = { "first_name": "givenName", "last_name": "sn", "email": "mail", } # This search ought to return all groups to which the user belongs. django_auth_ldap uses this to determine group # hierarchy. AUTH_LDAP_GROUP_SEARCH = LDAPSearch( "dc=example,dc=com", ldap.SCOPE_SUBTREE, "(objectClass=group)" ) AUTH_LDAP_GROUP_TYPE = NestedActiveDirectoryGroupType() # Define a group required to login. # AUTH_LDAP_REQUIRE_GROUP = "CN=Netbox-ReadOnly,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure" # Mirror LDAP group assignments. AUTH_LDAP_MIRROR_GROUPS = True # Define special user types using groups. Exercise great caution when assigning superuser status. AUTH_LDAP_USER_FLAGS_BY_GROUP = { "is_active": ( LDAPGroupQuery("CN=Netbox-ReadOnly,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com") | LDAPGroupQuery("CN=Netbox-StandardAccess,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com") | LDAPGroupQuery("CN=Netbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com") ), "is_staff": "CN=Netbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com", "is_superuser": "CN=DNetbox-Admins,OU=Resource_Groups,OU=Support - Netbox,OU=Infrastructure Services,DC=example,DC=com", } # For more granular permissions, we can map LDAP groups to Django groups. AUTH_LDAP_FIND_GROUP_PERMS = True # Cache groups for one hour to reduce LDAP traffic AUTH_LDAP_CACHE_TIMEOUT = 3600 ```
adam commented 2025-12-29 18:33:28 +01:00
Author
Owner
Copy Link

@nniehoff commented on GitHub (Sep 29, 2020):

I created several flamegraphs to watch instrument the uwsgi processes which were running during the high CPU usage. I found a lot of time was being spent in authentication.py:177. I'm not sure why logging handlers are being created and added to the logger with every new instance of this class, it seems like the LOGGING configuration should be handled by the LOGGING configuration parameter in configuration.py. I did find removing the additional logging configuration in my environment has drastically helped my issue. I'm still monitoring for more data.

@nniehoff commented on GitHub (Sep 29, 2020): I created several flamegraphs to watch instrument the uwsgi processes which were running during the high CPU usage. I found a lot of time was being spent in authentication.py:177. I'm not sure why logging handlers are being created and added to the logger with every new instance of this class, it seems like the LOGGING configuration should be handled by the [LOGGING](https://netbox.readthedocs.io/en/stable/configuration/optional-settings/#logging) configuration parameter in configuration.py. I did find removing the additional logging configuration in my environment has drastically helped my issue. I'm still monitoring for more data.
adam commented 2025-12-29 18:33:28 +01:00
Author
Owner
Copy Link

@nniehoff commented on GitHub (Sep 29, 2020):

  1. This appears to be a duplicate of #5192.

I'm not sure this is a duplicate. I don't see any occurrence of "Initiating TLS" in my logs as is mentioned in the other issue. I've enabled debug logging in my environment with my changes from my PR. I'll continue to monitor and note any TLS related log entries.

  1. If it's not, you'll need to provide far more detail than "Configure LDAP authentication." Sharing your complete configuration and noting what you're using for LDAP authentication would be a good start.

I hope the above provides enough detail about the configuration.

@nniehoff commented on GitHub (Sep 29, 2020): > 1. This appears to be a duplicate of #5192. I'm not sure this is a duplicate. I don't see any occurrence of "Initiating TLS" in my logs as is mentioned in the other issue. I've enabled debug logging in my environment with my changes from my PR. I'll continue to monitor and note any TLS related log entries. > 2. If it's not, you'll need to provide _far_ more detail than "Configure LDAP authentication." Sharing your complete configuration and noting what you're using for LDAP authentication would be a good start. I hope the above provides enough detail about the configuration.
adam commented 2025-12-29 18:33:29 +01:00
Author
Owner
Copy Link

@jeremystretch commented on GitHub (Sep 29, 2020):

I'm not sure why logging handlers are being created and added to the logger with every new instance of this class, it seems like the LOGGING configuration should be handled by the LOGGING configuration parameter in configuration.py.

I seem to recall some discussion way back when LDAP support was first added, where it was decided to intentionally force LDAP logging on the console because so many people experience problems with it, though I can't say for certain. (Note that the current LDAPBackend class was a complete copy & paste from the prior implementation within settings.py, so the logging code was simply copied over.)

I'm okay with removing the logging code if we clearly communicate the change to users, so that they are aware LDAP logging will need to be manually configured moving forward. However, I'd also like to be more certain that this is the underlying cause of the reported bug before doing so. If the logging code is the culprit, it seems unlikely that this bug would manifest in some LDAP-enabled instance but not others.

@jeremystretch commented on GitHub (Sep 29, 2020): > I'm not sure why logging handlers are being created and added to the logger with every new instance of this class, it seems like the LOGGING configuration should be handled by the LOGGING configuration parameter in configuration.py. I seem to recall some discussion _way_ back when LDAP support was first added, where it was decided to intentionally force LDAP logging on the console because so many people experience problems with it, though I can't say for certain. (Note that the current `LDAPBackend` class was a complete copy & paste from the prior implementation within `settings.py`, so the logging code was simply copied over.) I'm okay with removing the logging code if we clearly communicate the change to users, so that they are aware LDAP logging will need to be manually configured moving forward. However, I'd also like to be more certain that this is the underlying cause of the reported bug before doing so. If the logging code is the culprit, it seems unlikely that this bug would manifest in some LDAP-enabled instance but not others.
adam commented 2025-12-29 18:33:29 +01:00
Author
Owner
Copy Link

@nniehoff commented on GitHub (Sep 29, 2020):

That's interesting about the logging from settings.py. I have another environment (my production environment) that is currently running 2.8.8. In that environment, I don't see the CPU usage but I do see some high memory usage occasionally which I'm still troubleshooting, I might have to look into this as well. I'll attach a flamegraph that is a good example of CPU time being spent at authentication.py:177. From the flamegraph you can see 87.8% of the time was spent here. This is what lead me down this path. I'm thinking the change in 2.9 we are now creating more instances of the LDAPBackend class which in turn is calling new more frequently than in 2.8. The flamegraph was captured using 50 samples/sec for 60 seconds.
flamegraph.zip

@nniehoff commented on GitHub (Sep 29, 2020): That's interesting about the logging from settings.py. I have another environment (my production environment) that is currently running 2.8.8. In that environment, I don't see the CPU usage but I do see some high memory usage occasionally which I'm still troubleshooting, I might have to look into this as well. I'll attach a flamegraph that is a good example of CPU time being spent at authentication.py:177. From the flamegraph you can see 87.8% of the time was spent here. This is what lead me down this path. I'm thinking the change in 2.9 we are now creating more instances of the LDAPBackend class which in turn is calling __new__ more frequently than in 2.8. The flamegraph was captured using 50 samples/sec for 60 seconds. [flamegraph.zip](https://github.com/netbox-community/netbox/files/5301004/flamegraph.zip)
adam commented 2025-12-29 18:33:29 +01:00
Author
Owner
Copy Link

@nniehoff commented on GitHub (Sep 30, 2020):

We have not seen any re-occurrence of high CPU usage in 48 hours after removing the logging configuration from authentication.py. I have DEBUG logging enabled for LDAP and I also see no occurrences of "initiating TLS" in the logs, I suspect this is due to LDAP_IGNORE_CERT_ERRORS = True in this environment. I'm thinking this is not a duplicate of #5192

@nniehoff commented on GitHub (Sep 30, 2020): We have not seen any re-occurrence of high CPU usage in 48 hours after removing the logging configuration from authentication.py. I have DEBUG logging enabled for LDAP and I also see no occurrences of "initiating TLS" in the logs, I suspect this is due to `LDAP_IGNORE_CERT_ERRORS = True` in this environment. I'm thinking this is not a duplicate of #5192
adam commented 2025-12-29 18:33:29 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Oct 1, 2020):

I am going to mark this as a duplicate of #5199 then @nniehoff

@DanSheps commented on GitHub (Oct 1, 2020): I am going to mark this as a duplicate of #5199 then @nniehoff
adam commented 2025-12-29 18:33:29 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Oct 1, 2020):

Duplicate of #5199

@DanSheps commented on GitHub (Oct 1, 2020): Duplicate of #5199
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label status: duplicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#4150
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?