starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

headscale crashes after what seems a successful start - latest master, alpha6,7+8 #692

New Issue
Closed
opened 2025-12-29 02:22:08 +01:00 by adam · 8 comments
adam commented 2025-12-29 02:22:08 +01:00
Owner
Copy Link

Originally created by @mannp on GitHub (Apr 17, 2024).

Bug description

Using the latest master, alpha6 and 7 I get the same oops.

Environment

nix unstable flake

  • OS: nix unstable
  • Headscale version: latest master 40953727cf / alpha6 / 7
  • Tailscale version: N/A

Apologies but I do not wish to hand over my telephone number to discord, to post there.

To Reproduce

Afpha6 didn't go so well for me, so I deleted by data dir and started again.

Then I updated to alpha7 with a clean start.

This error remained with the versions I tried.

I can add a preshared key and when adding that to the nodes they get an error stating that key does not exist. The server logs a failed attempt with the preshared key.

Logs and attachments

Apr 17 14:21:55 server headscale-start[40624]: 2024/04/17 14:21:55 http2: panic serving 127.0.0.1:52794: runtime error: invalid memory address or nil pointer dereference
Apr 17 14:21:55 server headscale-start[40624]: goroutine 72 [running]:
Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net/http2.(*serverConn).runHandler.func1()
Apr 17 14:21:55 server headscale-start[40624]:         golang.org/x/net@v0.22.0/http2/server.go:2361 +0x145
Apr 17 14:21:55 server headscale-start[40624]: panic({0x1c45e20?, 0x3224aa0?})
Apr 17 14:21:55 server headscale-start[40624]:         runtime/panic.go:770 +0x132
Apr 17 14:21:55 server headscale-start[40624]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0xc0000c85a0, {0x22a96c0, 0xc000124030}, 0xc00047fd40)
Apr 17 14:21:55 server headscale-start[40624]:         github.com/juanfont/headscale/hscontrol/noise.go:228 +0x72d
Apr 17 14:21:55 server headscale-start[40624]: net/http.HandlerFunc.ServeHTTP(0xc00047eea0?, {0x22a96c0?, 0xc000124030?}, 0x10?)
Apr 17 14:21:55 server headscale-start[40624]:         net/http/server.go:2166 +0x29
Apr 17 14:21:55 server headscale-start[40624]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000180, {0x22a96c0, 0xc000124030}, 0xc00047e5a0)
Apr 17 14:21:55 server headscale-start[40624]:         github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net/http2.(*serverConn).runHandler(0xc0005ca000?, 0xc0005d9fd0?, 0x8afbe5?, 0xc0003a9040?)
Apr 17 14:21:55 server headscale-start[40624]:         golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb
Apr 17 14:21:55 server headscale-start[40624]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 40
Apr 17 14:21:55 server headscale-start[40624]:         golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d

-->

Originally created by @mannp on GitHub (Apr 17, 2024). <!-- Using headscale with the nix flake. --> ## Bug description Using the latest master, alpha6 and 7 I get the same oops. ## Environment nix unstable flake - OS: nix unstable - Headscale version: latest master 40953727cfb9fa9861becd7d34d44591d18528d5 / alpha6 / 7 - Tailscale version: N/A Apologies but I do not wish to hand over my telephone number to discord, to post there. ## To Reproduce Afpha6 didn't go so well for me, so I deleted by data dir and started again. Then I updated to alpha7 with a clean start. This error remained with the versions I tried. I can add a preshared key and when adding that to the nodes they get an error stating that key does not exist. The server logs a failed attempt with the preshared key. ## Logs and attachments ``` Apr 17 14:21:55 server headscale-start[40624]: 2024/04/17 14:21:55 http2: panic serving 127.0.0.1:52794: runtime error: invalid memory address or nil pointer dereference Apr 17 14:21:55 server headscale-start[40624]: goroutine 72 [running]: Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net/http2.(*serverConn).runHandler.func1() Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net@v0.22.0/http2/server.go:2361 +0x145 Apr 17 14:21:55 server headscale-start[40624]: panic({0x1c45e20?, 0x3224aa0?}) Apr 17 14:21:55 server headscale-start[40624]: runtime/panic.go:770 +0x132 Apr 17 14:21:55 server headscale-start[40624]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0xc0000c85a0, {0x22a96c0, 0xc000124030}, 0xc00047fd40) Apr 17 14:21:55 server headscale-start[40624]: github.com/juanfont/headscale/hscontrol/noise.go:228 +0x72d Apr 17 14:21:55 server headscale-start[40624]: net/http.HandlerFunc.ServeHTTP(0xc00047eea0?, {0x22a96c0?, 0xc000124030?}, 0x10?) Apr 17 14:21:55 server headscale-start[40624]: net/http/server.go:2166 +0x29 Apr 17 14:21:55 server headscale-start[40624]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000180, {0x22a96c0, 0xc000124030}, 0xc00047e5a0) Apr 17 14:21:55 server headscale-start[40624]: github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net/http2.(*serverConn).runHandler(0xc0005ca000?, 0xc0005d9fd0?, 0x8afbe5?, 0xc0003a9040?) Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb Apr 17 14:21:55 server headscale-start[40624]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 40 Apr 17 14:21:55 server headscale-start[40624]: golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d ``` -->
adam added the bug label 2025-12-29 02:22:08 +01:00
adam closed this issue 2025-12-29 02:22:09 +01:00
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@Nickiel12 commented on GitHub (Apr 18, 2024):

Can confirm this in alpha8 as well

@Nickiel12 commented on GitHub (Apr 18, 2024): Can confirm this in alpha8 as well
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@Nickiel12 commented on GitHub (Apr 18, 2024):

If you add your headscale nodes anyways, this message doesn't appear, but even after restarting headscale, tailscale nodes are not given IP addresses

@Nickiel12 commented on GitHub (Apr 18, 2024): If you add your headscale nodes anyways, this message doesn't appear, but even after restarting headscale, tailscale nodes are not given IP addresses
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@alirezamirsepassi commented on GitHub (Apr 18, 2024):

I can confirm this is happening to me as well on the alpha8

@alirezamirsepassi commented on GitHub (Apr 18, 2024): I can confirm this is happening to me as well on the alpha8
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@SuperSandro2000 commented on GitHub (Apr 19, 2024):

I needed to restore the headscale database after rolling back due to a missing column (ip_addresses). Creating the column again by hand with alter table nodes add COLUMN ip_addresses; didn't help much, as everything would have been needed to be re-registered or the table would need to be refilled by hand with IPs.

@SuperSandro2000 commented on GitHub (Apr 19, 2024): I needed to restore the headscale database after rolling back due to a missing column (`ip_addresses`). Creating the column again by hand with ``alter table nodes add COLUMN ip_addresses;`` didn't help much, as everything would have been needed to be re-registered or the table would need to be refilled by hand with IPs.
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@mannp commented on GitHub (Apr 19, 2024):

I tried rolling back to my known working rev for me and still had this issue. I am currently building the latest master with a clean setup to see if I can get up and running again.

So wiping /var/lib/headscale and starting from today's master rev c906aaf927 headscale started fine.

I of course needed to start from scratch reconnecting nodes, but it's worked around for me.

It appears a dB migration issue? This can be closed for me, but will leave open for others who may not be able to wipe out everything and start again.

@mannp commented on GitHub (Apr 19, 2024): I tried rolling back to my known working rev for me and still had this issue. I am currently building the latest master with a clean setup to see if I can get up and running again. So wiping /var/lib/headscale and starting from today's master rev c906aaf9278fd5f9df8551600dfaa57ff557733d headscale started fine. I of course needed to start from scratch reconnecting nodes, but it's worked around for me. It appears a dB migration issue? This can be closed for me, but will leave open for others who may not be able to wipe out everything and start again.
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@Nickiel12 commented on GitHub (Apr 21, 2024):

Can confirm the same issue on alpha9

Apr 21 14:40:22 headscale-start[2384417]:         github.com/juanfont/headscale/hscontrol/metrics.go:72 +0x143
Apr 21 14:40:22 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0xc000444fc0?, {0x22b5020?, 0xc0001246b>
Apr 21 14:40:22 headscale-start[2384417]:         net/http/server.go:2166 +0x29
Apr 21 14:40:22 headscale-start[2384417]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000176180, {0x22b5020, 0>
Apr 21 14:40:22 headscale-start[2384417]:         github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
Apr 21 14:40:22 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler(0x44b87d?, 0xc0004f291>
Apr 21 14:40:22 headscale-start[2384417]:         golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb
Apr 21 14:40:22 headscale-start[2384417]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in gor>
Apr 21 14:40:22 headscale-start[2384417]:         golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d
Apr 21 14:40:38 headscale-start[2384417]: 2024/04/21 14:40:38 http2: panic serving 127.0.0.1:34652: runtime erro>
Apr 21 14:40:38 headscale-start[2384417]: goroutine 145 [running]:
Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler.func1()
Apr 21 14:40:38 headscale-start[2384417]:         golang.org/x/net@v0.22.0/http2/server.go:2361 +0x145
Apr 21 14:40:38 headscale-start[2384417]: panic({0x1c4f160?, 0x3238b60?})
Apr 21 14:40:38 headscale-start[2384417]:         runtime/panic.go:770 +0x132
Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMap>
Apr 21 14:40:38 headscale-start[2384417]:         github.com/juanfont/headscale/hscontrol/noise.go:229 +0x79e
Apr 21 14:40:38 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0x7f822b5d3108?, {0x22b5020?, 0xc000000>
Apr 21 14:40:38 headscale-start[2384417]:         net/http/server.go:2166 +0x29
Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x>
Apr 21 14:40:38 headscale-start[2384417]:         github.com/juanfont/headscale/hscontrol/metrics.go:72 +0x143
Apr 21 14:40:38 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0xc000445440?, {0x22b5020?, 0xc00000014>
Apr 21 14:40:38 headscale-start[2384417]:         net/http/server.go:2166 +0x29
Apr 21 14:40:38 headscale-start[2384417]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000176180, {0x22b5020, 0>
Apr 21 14:40:38 headscale-start[2384417]:         github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler(0x44b87d?, 0xc0003945a>
Apr 21 14:40:38 headscale-start[2384417]:         golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb
Apr 21 14:40:38 headscale-start[2384417]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in gor>
Apr 21 14:40:38 headscale-start[2384417]:         golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d
@Nickiel12 commented on GitHub (Apr 21, 2024): Can confirm the same issue on alpha9 ``` Apr 21 14:40:22 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol/metrics.go:72 +0x143 Apr 21 14:40:22 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0xc000444fc0?, {0x22b5020?, 0xc0001246b> Apr 21 14:40:22 headscale-start[2384417]: net/http/server.go:2166 +0x29 Apr 21 14:40:22 headscale-start[2384417]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000176180, {0x22b5020, 0> Apr 21 14:40:22 headscale-start[2384417]: github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 Apr 21 14:40:22 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler(0x44b87d?, 0xc0004f291> Apr 21 14:40:22 headscale-start[2384417]: golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb Apr 21 14:40:22 headscale-start[2384417]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in gor> Apr 21 14:40:22 headscale-start[2384417]: golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d Apr 21 14:40:38 headscale-start[2384417]: 2024/04/21 14:40:38 http2: panic serving 127.0.0.1:34652: runtime erro> Apr 21 14:40:38 headscale-start[2384417]: goroutine 145 [running]: Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler.func1() Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net@v0.22.0/http2/server.go:2361 +0x145 Apr 21 14:40:38 headscale-start[2384417]: panic({0x1c4f160?, 0x3238b60?}) Apr 21 14:40:38 headscale-start[2384417]: runtime/panic.go:770 +0x132 Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMap> Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol/noise.go:229 +0x79e Apr 21 14:40:38 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0x7f822b5d3108?, {0x22b5020?, 0xc000000> Apr 21 14:40:38 headscale-start[2384417]: net/http/server.go:2166 +0x29 Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x> Apr 21 14:40:38 headscale-start[2384417]: github.com/juanfont/headscale/hscontrol/metrics.go:72 +0x143 Apr 21 14:40:38 headscale-start[2384417]: net/http.HandlerFunc.ServeHTTP(0xc000445440?, {0x22b5020?, 0xc00000014> Apr 21 14:40:38 headscale-start[2384417]: net/http/server.go:2166 +0x29 Apr 21 14:40:38 headscale-start[2384417]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000176180, {0x22b5020, 0> Apr 21 14:40:38 headscale-start[2384417]: github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net/http2.(*serverConn).runHandler(0x44b87d?, 0xc0003945a> Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net@v0.22.0/http2/server.go:2368 +0xbb Apr 21 14:40:38 headscale-start[2384417]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in gor> Apr 21 14:40:38 headscale-start[2384417]: golang.org/x/net@v0.22.0/http2/server.go:2303 +0x21d ```
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@kfkawalec commented on GitHub (Apr 22, 2024):

This is a big blocker. Is there any way to fix it?
"alter table nodes add COLUMN ip_addresses" does not work on PostgreSQL and I do not want to experiment because I do not know what type of column it should be.

@kfkawalec commented on GitHub (Apr 22, 2024): This is a big blocker. Is there any way to fix it? "_alter table nodes add COLUMN ip_addresses_" does not work on PostgreSQL and I do not want to experiment because I do not know what type of column it should be.
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@SuperSandro2000 commented on GitHub (Apr 23, 2024):

I didn't verify this but could it be that we are trying to display the node id of the node we failed to fetch? so just dropping line 229 should help?

https://github.com/juanfont/headscale/blob/main/hscontrol/noise.go#L229-L230

@SuperSandro2000 commented on GitHub (Apr 23, 2024): I didn't verify this but could it be that we are trying to display the node id of the node we failed to fetch? so just dropping line 229 should help? https://github.com/juanfont/headscale/blob/main/hscontrol/noise.go#L229-L230
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#692
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?