starred/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2026-01-11 21:10:29 +01:00
Code Issues 273 Packages Projects Releases 10 Wiki Activity

Handshake Timeouted for Netbox Worker #8260

New Issue
Closed
opened 2025-12-29 20:34:27 +01:00 by adam · 1 comment
adam commented 2025-12-29 20:34:27 +01:00
Owner
Copy Link

Originally created by @estein9825 on GitHub (Jun 28, 2023).

NetBox version

v3.5.3

Python version

3.10

Steps to Reproduce

  1. Install redis 6.0.5-bloom separately into kubernetes cluster using helm chart
  2. Install netbox 4.1.0 helm chart into kubernetes (which installs netbox 3.5.3)
  3. Have +2000 interfaces and/or +2000 ip addresses in release

Expected Behavior

Worker should not keep restarting with ssl failure

Observed Behavior

Every 10 min, Error thrown and worker restarts:

Traceback (most recent call last):
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 699, in connect
sock = self.retry.call_with_retry(
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
raise error
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 700, in
lambda: self._connect(), lambda error: self.disconnect(error)
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 1122, in _connect
sslsock = context.wrap_socket(sock, server_hostname=self.host)
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
TimeoutError: _ssl.c:980: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/netbox/netbox/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/init.py", line 446, in execute_from_command_line
utility.execute()
File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/init.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
File "/opt/netbox/netbox/core/management/commands/rqworker.py", line 28, in handle
super().handle(*args, **options)
File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/management/commands/rqworker.py", line 123, in handle
w = get_worker(*args, **worker_kwargs)
File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/workers.py", line 50, in get_worker
return worker_class(
File "/opt/netbox/venv/lib/python3.10/site-packages/rq/worker.py", line 204, in init
connection.client_setname(self.name)
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/commands/core.py", line 707, in client_setname
return self.execute_command("CLIENT SETNAME", name, **kwargs)
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/client.py", line 1266, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 1457, in get_connection
connection.connect()
File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 703, in connect
raise TimeoutError("Timeout connecting to server")
redis.exceptions.TimeoutError: Timeout connecting to server

I have tried increasing the RQ_DEFAULT_TIMEOUT but it didn't seem to affect anything. I also tried overriding the redis config to change the SOCKET_TIMEOUT and CONNECTION_KARGS socket_connection_timeout as well. But it didn't change anything.

Originally created by @estein9825 on GitHub (Jun 28, 2023). ### NetBox version v3.5.3 ### Python version 3.10 ### Steps to Reproduce 1. Install redis 6.0.5-bloom separately into kubernetes cluster using helm chart 3. Install netbox 4.1.0 helm chart into kubernetes (which installs netbox 3.5.3) 2. Have +2000 interfaces and/or +2000 ip addresses in release ### Expected Behavior Worker should not keep restarting with ssl failure ### Observed Behavior Every 10 min, Error thrown and worker restarts: Traceback (most recent call last): File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 699, in connect sock = self.retry.call_with_retry( File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry raise error File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry return do() File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 700, in <lambda> lambda: self._connect(), lambda error: self.disconnect(error) File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 1122, in _connect sslsock = context.wrap_socket(sock, server_hostname=self.host) File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket return self.sslsocket_class._create( File "/usr/lib/python3.10/ssl.py", line 1071, in _create self.do_handshake() File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake self._sslobj.do_handshake() TimeoutError: _ssl.c:980: The handshake operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/netbox/netbox/manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/netbox/netbox/core/management/commands/rqworker.py", line 28, in handle super().handle(*args, **options) File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/management/commands/rqworker.py", line 123, in handle w = get_worker(*args, **worker_kwargs) File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/workers.py", line 50, in get_worker return worker_class( File "/opt/netbox/venv/lib/python3.10/site-packages/rq/worker.py", line 204, in __init__ connection.client_setname(self.name) File "/opt/netbox/venv/lib/python3.10/site-packages/redis/commands/core.py", line 707, in client_setname return self.execute_command("CLIENT SETNAME", name, **kwargs) File "/opt/netbox/venv/lib/python3.10/site-packages/redis/client.py", line 1266, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 1457, in get_connection connection.connect() File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 703, in connect raise TimeoutError("Timeout connecting to server") redis.exceptions.TimeoutError: Timeout connecting to server I have tried increasing the RQ_DEFAULT_TIMEOUT but it didn't seem to affect anything. I also tried overriding the redis config to change the SOCKET_TIMEOUT and CONNECTION_KARGS socket_connection_timeout as well. But it didn't change anything.
adam closed this issue 2025-12-29 20:34:27 +01:00
adam commented 2025-12-29 20:34:28 +01:00
Author
Owner
Copy Link

@DanSheps commented on GitHub (Jun 29, 2023):

It seems more likely that there is something wrong with your redis server, which is causing you to get timeout errors in NetBox and causing the NetBox worker to restart.

I do see anything wrong with NetBox here.

@DanSheps commented on GitHub (Jun 29, 2023): It seems more likely that there is something wrong with your redis server, which is causing you to get timeout errors in NetBox and causing the NetBox worker to restart. I do see anything wrong with NetBox here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
beta
breaking change
complexity: high
complexity: low
complexity: medium
needs milestone
netbox
pending closure
plugin candidate
pull-request
Mirrored from GitHub Pull Request
 severity: high
severity: low
severity: medium
status: accepted
status: backlog
status: blocked
status: duplicate
status: needs owner
status: needs triage
status: revisions needed
status: under review
topic: GraphQL
topic: Internationalization
topic: OpenAPI
topic: UI/UX
topic: cabling
topic: event rules
topic: htmx navigation
topic: industrialization
topic: migrations
topic: plugins
topic: scripts
topic: templating
topic: testing
type: bug
type: deprecation
type: documentation
type: feature
type: housekeeping
type: translation
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/netbox#8260
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?