starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] panic in serveLongPoll after ephemeral node deletion ("invalid memory address or nil pointer dereference") #1076

New Issue
Closed
opened 2025-12-29 02:28:08 +01:00 by adam · 3 comments
adam commented 2025-12-29 02:28:08 +01:00
Owner
Copy Link

Originally created by @vsychov on GitHub (Aug 4, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Hello,

I encountering occasional panics in (*mapSession).serveLongPoll after ephemeral nodes are deleted. The panic appears to happen when Headscale attempts to access a node in the database that no longer exists, resulting in a nil pointer dereference.

2025-07-31T16:37:02Z DBG deleted ephemeral node node.id=58
2025-07-31T16:37:02Z ERR go/src/headscale/hscontrol/poll.go:595 > Could not get machine from db error="record not found" node=xxxxx node.id=58 omitPeers=false readOnly=false stream=true
2025/07/31 16:37:02 http2: panic serving 10.61.15.17:37448: runtime error: invalid memory address or nil pointer dereference
goroutine 21012 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	/go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2426 +0x13e
panic({0x1f6cdc0?, 0x3d16800?})
	/usr/local/go/src/runtime/panic.go:792 +0x132
github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll.func1()
	/go/src/headscale/hscontrol/poll.go:182 +0xaa
github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0xc00096e000)
	/go/src/headscale/hscontrol/poll.go:275 +0x14e5
github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0xc000f063f0, {0x2b20ba0, 0xc0010c6130}, 0xc0004b8280)
	/go/src/headscale/hscontrol/noise.go:229 +0x258
net/http.HandlerFunc.ServeHTTP(0xc0014520c0?, {0x2b20ba0?, 0xc0010c6130?}, 0x7670a9791a78?)
	/usr/local/go/src/net/http/server.go:2294 +0x29
github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x2b20ba0, 0xc0010c6130}, 0xc0004b8280)
	/go/src/headscale/hscontrol/metrics.go:82 +0x18e
net/http.HandlerFunc.ServeHTTP(0xc0004b8140?, {0x2b20ba0?, 0xc0010c6130?}, 0x5b?)
	/usr/local/go/src/net/http/server.go:2294 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0006c29c0, {0x2b20ba0, 0xc0010c6130}, 0xc0013603c0)
	/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
golang.org/x/net/http2.(*serverConn).runHandler(0x44d2d2?, 0x0?, 0x0?, 0xc000830960?)
	/go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2433 +0xf5
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 21009
	/go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2367 +0x21d

This issue occurs spontaneously and is not reliably reproducible. It happens intermittently without any specific external trigger.

Expected Behavior

Working without panic.

Steps To Reproduce

I haven't found a way to reproduce it deterministically, but it occurs occasionally under minimal load (~10-20 nodes).

Environment

- OS: Debian
- Headscale version: v0.26.1
- Tailscale version: 1.84.1

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

Originally created by @vsychov on GitHub (Aug 4, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Hello, I encountering occasional panics in `(*mapSession).serveLongPoll` after ephemeral nodes are deleted. The panic appears to happen when Headscale attempts to access a node in the database that no longer exists, resulting in a `nil` pointer dereference. ``` 2025-07-31T16:37:02Z DBG deleted ephemeral node node.id=58 2025-07-31T16:37:02Z ERR go/src/headscale/hscontrol/poll.go:595 > Could not get machine from db error="record not found" node=xxxxx node.id=58 omitPeers=false readOnly=false stream=true 2025/07/31 16:37:02 http2: panic serving 10.61.15.17:37448: runtime error: invalid memory address or nil pointer dereference goroutine 21012 [running]: golang.org/x/net/http2.(*serverConn).runHandler.func1() /go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2426 +0x13e panic({0x1f6cdc0?, 0x3d16800?}) /usr/local/go/src/runtime/panic.go:792 +0x132 github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll.func1() /go/src/headscale/hscontrol/poll.go:182 +0xaa github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0xc00096e000) /go/src/headscale/hscontrol/poll.go:275 +0x14e5 github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0xc000f063f0, {0x2b20ba0, 0xc0010c6130}, 0xc0004b8280) /go/src/headscale/hscontrol/noise.go:229 +0x258 net/http.HandlerFunc.ServeHTTP(0xc0014520c0?, {0x2b20ba0?, 0xc0010c6130?}, 0x7670a9791a78?) /usr/local/go/src/net/http/server.go:2294 +0x29 github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x2b20ba0, 0xc0010c6130}, 0xc0004b8280) /go/src/headscale/hscontrol/metrics.go:82 +0x18e net/http.HandlerFunc.ServeHTTP(0xc0004b8140?, {0x2b20ba0?, 0xc0010c6130?}, 0x5b?) /usr/local/go/src/net/http/server.go:2294 +0x29 github.com/gorilla/mux.(*Router).ServeHTTP(0xc0006c29c0, {0x2b20ba0, 0xc0010c6130}, 0xc0013603c0) /go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 golang.org/x/net/http2.(*serverConn).runHandler(0x44d2d2?, 0x0?, 0x0?, 0xc000830960?) /go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2433 +0xf5 created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 21009 /go/pkg/mod/golang.org/x/net@v0.41.0/http2/server.go:2367 +0x21d ``` This issue occurs spontaneously and is not reliably reproducible. It happens intermittently without any specific external trigger. ### Expected Behavior Working without panic. ### Steps To Reproduce I haven't found a way to reproduce it deterministically, but it occurs occasionally under minimal load (~10-20 nodes). ### Environment ```markdown - OS: Debian - Headscale version: v0.26.1 - Tailscale version: 1.84.1 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information ---
adam added the bug label 2025-12-29 02:28:08 +01:00
adam closed this issue 2025-12-29 02:28:08 +01:00
adam commented 2025-12-29 02:28:08 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 10, 2025):

Investigating this, are you sure this was on v0.26.1 ? I am trying to track the line numbers in the stack trace to that tag and I suspect they might be off: https://github.com/juanfont/headscale/blob/v0.26.1/hscontrol/poll.go#L182

@kradalby commented on GitHub (Sep 10, 2025): Investigating this, are you sure this was on `v0.26.1` ? I am trying to track the line numbers in the stack trace to that tag and I suspect they might be off: https://github.com/juanfont/headscale/blob/v0.26.1/hscontrol/poll.go#L182
adam commented 2025-12-29 02:28:08 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 10, 2025):

I suspect this is the same as #2698 and #2697, which should be fixed in #2670

@kradalby commented on GitHub (Sep 10, 2025): I suspect this is the same as #2698 and #2697, which should be fixed in #2670
adam commented 2025-12-29 02:28:09 +01:00
Author
Owner
Copy Link

@vsychov commented on GitHub (Sep 10, 2025):

are you sure this was on v0.26.1

No, not sure anymore. The version was this one:

root@headscale-0:/# headscale version
855c48a

Thanks for the fix!

@vsychov commented on GitHub (Sep 10, 2025): > are you sure this was on `v0.26.1` No, not sure anymore. The version was [this one](https://github.com/juanfont/headscale/commit/855c48a): ``` root@headscale-0:/# headscale version 855c48a ``` Thanks for the fix!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1076
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?