starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] unaligned 64-bit atomic operation #990

New Issue
Closed
opened 2025-12-29 02:27:06 +01:00 by adam · 6 comments
adam commented 2025-12-29 02:27:06 +01:00
Owner
Copy Link

Originally created by @trackmafia on GitHub (Mar 30, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

fresh install of headscale server, TLS-ALPN-01 SSL enabled. Using Raspberry PI 3, 32bit arm7 install as server. Opnsense 25.1.4 router as client.

tailscale client on opnsense router. node registers but hangs at client. tailscale client never logs in. On headscale server, node is approved but never comes online.

Expected Behavior

after authorizing node, client would log in and join the tailnet

Steps To Reproduce

  1. install fresh server, configure for SSL
  2. install client and try to connect.

Environment

- OS: Raspbian 12 32bit client. Opnsense router on x86 hardware
- Headscale version: 0.25.1 and 0.24.3
- Tailscale version: 1.82

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

here are the logs:

Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0x44244b8, chan: 0x4501080 node=J>
Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0x44244b8, chan: 0x4501080 nod>
Mar 30 16:15:35 headscale[16010]: 2025/03/30 16:15:35 http2: panic serving 1.2.3.4:21191: unaligned 64-bit atomic operation
Mar 30 16:15:35 headscale[16010]: goroutine 1294 [running]:
Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler.func1()
Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x184
Mar 30 16:15:35 headscale[16010]: panic({0x15eeb80, 0x1b66b18})
Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0xfc
Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.panicUnaligned()
Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/unaligned.go:8 +0x24
Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.Xadd64(0x461d7b4, 0x1)
Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/atomic_arm.s:265 +0x14
Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).marshalMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...)
Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:331 +0x3c
Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).FullMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...)
Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:197 +0xfc
Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0x44244b8)
Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/poll.go:289 +0x1238
Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0x46bc100, {0x1b75960, 0x49524a0}, 0x47b86e8)
Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/noise.go:225 +0x250
Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x47480c8, {0x1b75960, 0x49524a0}, 0x47b86e8)
Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34
Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x1b75960, 0x49524a0}, 0x47b86e8)
Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/metrics.go:82 +0x244
Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x4aa8700, {0x1b75960, 0x49524a0}, 0x47b86e8)
Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34
Mar 30 16:15:35 headscale[16010]: github.com/gorilla/mux.(*Router).ServeHTTP(0x46fa0c0, {0x1b75960, 0x49524a0}, 0x499c638)
Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1dc
Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler(0x479e788, 0x49524a0, 0x499c638, 0x4911800)
Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xd8
Mar 30 16:15:35 headscale[16010]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 216
Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x248

Originally created by @trackmafia on GitHub (Mar 30, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior fresh install of headscale server, TLS-ALPN-01 SSL enabled. Using Raspberry PI 3, 32bit arm7 install as server. Opnsense 25.1.4 router as client. tailscale client on opnsense router. node registers but hangs at client. tailscale client never logs in. On headscale server, node is approved but never comes online. ### Expected Behavior after authorizing node, client would log in and join the tailnet ### Steps To Reproduce 1. install fresh server, configure for SSL 2. install client and try to connect. ### Environment ```markdown - OS: Raspbian 12 32bit client. Opnsense router on x86 hardware - Headscale version: 0.25.1 and 0.24.3 - Tailscale version: 1.82 ``` ### Runtime environment - [ ] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Debug information here are the logs: Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0x44244b8, chan: 0x4501080 node=J> Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0x44244b8, chan: 0x4501080 nod> Mar 30 16:15:35 headscale[16010]: 2025/03/30 16:15:35 http2: panic serving 1.2.3.4:21191: unaligned 64-bit atomic operation Mar 30 16:15:35 headscale[16010]: goroutine 1294 [running]: Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler.func1() Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x184 Mar 30 16:15:35 headscale[16010]: panic({0x15eeb80, 0x1b66b18}) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0xfc Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.panicUnaligned() Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/unaligned.go:8 +0x24 Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.Xadd64(0x461d7b4, 0x1) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/atomic_arm.s:265 +0x14 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).marshalMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:331 +0x3c Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).FullMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:197 +0xfc Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0x44244b8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/poll.go:289 +0x1238 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0x46bc100, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/noise.go:225 +0x250 Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x47480c8, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/metrics.go:82 +0x244 Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x4aa8700, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34 Mar 30 16:15:35 headscale[16010]: github.com/gorilla/mux.(*Router).ServeHTTP(0x46fa0c0, {0x1b75960, 0x49524a0}, 0x499c638) Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1dc Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler(0x479e788, 0x49524a0, 0x499c638, 0x4911800) Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xd8 Mar 30 16:15:35 headscale[16010]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 216 Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x248
adam added the bug label 2025-12-29 02:27:06 +01:00
adam closed this issue 2025-12-29 02:27:06 +01:00
adam commented 2025-12-29 02:27:07 +01:00
Author
Owner
Copy Link

@muru commented on GitHub (Apr 12, 2025):

I'm seeing the same thing on 32-bit Arch Linux ARM (armv7l). I have tried:

I'm running Headscale behind an nginx reverse proxy (however it is using stream + ssl_preread, so the traffic is pretty much untouched by nginx).

@muru commented on GitHub (Apr 12, 2025): I'm seeing the same thing on 32-bit Arch Linux ARM (`armv7l`). I have tried: - The package from Arch Linux ARM (`headscale-0.25.1-1`) - The ARMv7 binary from the GitHub release for `0.25.1` (https://github.com/juanfont/headscale/releases/download/v0.25.1/headscale_0.25.1_linux_armv7) - The ARMv7 binary from the GitHub release for `0.24.3` (https://github.com/juanfont/headscale/releases/download/v0.24.3/headscale_0.24.3_linux_armv7) I'm running Headscale behind an nginx reverse proxy (however it is using [`stream` + `ssl_preread`](https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html), so the traffic is pretty much untouched by nginx).
adam commented 2025-12-29 02:27:07 +01:00
Author
Owner
Copy Link

@habnabit commented on GitHub (Apr 14, 2025):

same issue; same context. trying with armv6 now.

e: nope, same issue

@habnabit commented on GitHub (Apr 14, 2025): same issue; same context. trying with `armv6` now. e: nope, same issue
adam commented 2025-12-29 02:27:07 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jul 14, 2025):

This issue is stale because it has been open for 90 days with no activity.

@github-actions[bot] commented on GitHub (Jul 14, 2025): This issue is stale because it has been open for 90 days with no activity.
adam commented 2025-12-29 02:27:07 +01:00
Author
Owner
Copy Link

@habnabit commented on GitHub (Jul 14, 2025):

has this bug been identified or patched? i haven’t checked if there’s any newer releases
Message ID: @.***>

@habnabit commented on GitHub (Jul 14, 2025): has this bug been identified or patched? i haven’t checked if there’s any newer releases Message ID: ***@***.***>
adam commented 2025-12-29 02:27:08 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Jul 15, 2025):

No, and I think realistically we will not put any effort into solving this. I have no 32 bit hardware and its a runtime error so we wont be able to catch it in tests.

Happy to take contributions as long as they fit within maintainability and so on, but it will be hard for us to guarantee that we dont break it again in the future.

@kradalby commented on GitHub (Jul 15, 2025): No, and I think realistically we will not put any effort into solving this. I have no 32 bit hardware and its a runtime error so we wont be able to catch it in tests. Happy to take contributions as long as they fit within maintainability and so on, but it will be hard for us to guarantee that we dont break it again in the future.
adam commented 2025-12-29 02:27:08 +01:00
Author
Owner
Copy Link

@Duncaen commented on GitHub (Oct 30, 2025):

The issue was that atomic.AddUint64[0] was used on a uint64 struct member, which happened to be unaliged because of the struct layout.

This code was already removed and in other places the where atomic are used the bug is fixed because the atomic.Int64[1] type is used, which provide the alignment.

@Duncaen commented on GitHub (Oct 30, 2025): The issue was that `atomic.AddUint64`[0] was used on a uint64 struct member, which happened to be unaliged because of the struct layout. This code was already removed and in other places the where atomic are used the bug is fixed because the `atomic.Int64`[1] type is used, which provide the alignment. - [0] https://github.com/juanfont/headscale/blob/v0.25.1/hscontrol/mapper/mapper.go#L331 - [1] https://github.com/juanfont/headscale/blob/0a43aab8f5c876935f84ab9725f0e8b47dffe809/hscontrol/mapper/batcher_lockfree.go#L37
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#990
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?