starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] "Unexpected fault address" when opening SQLite db under armv7 #828

New Issue
Closed
opened 2025-12-29 02:24:32 +01:00 by adam · 11 comments
adam commented 2025-12-29 02:24:32 +01:00
Owner
Copy Link

Originally created by @m-kloeckner on GitHub (Oct 11, 2024).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I am running headscale on a armv7 SOC (Cubieboard2) in a docker container since quite a while now. After switching to version 0.23 I'm running in an error.

When trying to start headscale using the official headscale/headscale docker image for armv7 the process fails with the following error after trying to open the SQLite db.

headscale | 2024-10-11T08:13:30Z WRN
headscale | WARN: The "dns.use_username_in_magic_dns" configuration key is deprecated and has been removed. Please see the changelog for more details.
headscale |
headscale | 2024-10-11T08:13:30Z INF No private key file at path, creating... path=/var/lib/headscale/noise_private.key
headscale | 2024-10-11T08:13:30Z INF Opening database database=sqlite3 path=/etc/headscale/db.sqlite
headscale | unexpected fault address 0x769affff
headscale | fatal error: fault
headscale | [signal SIGBUS: bus error code=0x2 addr=0x769affff pc=0x979e58]
headscale |
headscale | goroutine 1 gp=0x3402128 m=4 mp=0x3461408 [running]:
headscale | runtime.throw({0x175ca21, 0x5})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/panic.go:1067 +0x34 fp=0x34d4e1c sp=0x34d4e08 pc=0x9292c
headscale | runtime.sigpanic()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/signal_unix.go:897 +0x104 fp=0x34d4e4c sp=0x34d4e1c pc=0x94cd0
headscale | modernc.org/libc.Xmemset(0x35ca360, 0x769a8088, 0x0, 0x7f78)
headscale | /home/runner/go/pkg/mod/modernc.org/libc@v1.60.1/ccgo_linux_arm.go:146120 +0x24 fp=0x34d4e50 sp=0x34d4e50 pc=0x979e58
headscale | modernc.org/sqlite/lib._walIndexAppend(0x35ca360, 0x76600718, 0x1, 0x1)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:43350 +0xe4 fp=0x34d4e88 sp=0x34d4e50 pc=0x9caf58
headscale | modernc.org/sqlite/lib._walFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:46136 +0xa48 fp=0x34d4f0c sp=0x34d4e88 pc=0x9d1404
headscale | modernc.org/sqlite/lib._sqlite3WalFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:46177 +0x4c fp=0x34d4f30 sp=0x34d4f0c pc=0x9d1728
headscale | modernc.org/sqlite/lib._pagerWalFrames(0x35ca360, 0x76000418, 0x75b07038, 0x1, 0x1)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:37777 +0x114 fp=0x34d4f70 sp=0x34d4f30 pc=0x9c1d88
headscale | modernc.org/sqlite/lib._sqlite3PagerCommitPhaseOne(0x35ca360, 0x76000418, 0x0, 0x0)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:40996 +0x1b0 fp=0x34d4fa4 sp=0x34d4f70 pc=0x9c8510
headscale | modernc.org/sqlite/lib._sqlite3BtreeCommitPhaseOne(0x35ca360, 0x75e00058, 0x0)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:51705 +0xbc fp=0x34d4fc4 sp=0x34d4fa4 pc=0x9db3a0
headscale | modernc.org/sqlite/lib._vdbeCommit(0x35ca360, 0x76000018, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:64640 +0xdcc fp=0x34d508c sp=0x34d4fc4 pc=0x9fbc6c
headscale | modernc.org/sqlite/lib._sqlite3VdbeHalt(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:65053 +0x348 fp=0x34d50b4 sp=0x34d508c pc=0x9fc42c
headscale | modernc.org/sqlite/lib._sqlite3VdbeExec(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:71196 +0x4920 fp=0x34d5594 sp=0x34d50b4 pc=0xa0c450
headscale | modernc.org/sqlite/lib._sqlite3Step(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:67892 +0x8c fp=0x34d55b4 sp=0x34d5594 pc=0xa02ad0
headscale | modernc.org/sqlite/lib.Xsqlite3_step(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:67959 +0xdc fp=0x34d55e4 sp=0x34d55b4 pc=0xa02f2c
headscale | github.com/glebarez/go-sqlite.(*conn).step(0x348b7e0, 0x75a08808)
headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:1002 +0x28 fp=0x34d55fc sp=0x34d55e4 pc=0xb68c90
headscale | github.com/glebarez/go-sqlite.(*stmt).exec.func1(0x34fa6c8, 0x34d566c, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:536 +0x128 fp=0x34d5648 sp=0x34d55fc pc=0xb66aac
headscale | github.com/glebarez/go-sqlite.(*stmt).exec(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:549 +0x1a0 fp=0x34d5680 sp=0x34d5648 pc=0xb66820
headscale | github.com/glebarez/go-sqlite.(*stmt).ExecContext(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite_go18.go:43 +0x44 fp=0x34d56ac sp=0x34d5680 pc=0xb6ba98
headscale | database/sql.ctxDriverStmtExec({0x1a6deb4, 0x27acfa8}, {0x1a6e584, 0x34fa6c8}, {0x27acfa8, 0x0, 0x0})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/ctxutil.go:65 +0xb0 fp=0x34d56fc sp=0x34d56ac pc=0x730da0
headscale | database/sql.resultFromStatement({0x1a6deb4, 0x27acfa8}, {0x1a6d0a0, 0x348b7e0}, 0x36e4800, {0x363a580, 0x0, 0x8})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2672 +0xfc fp=0x34d5750 sp=0x34d56fc pc=0x73bce8
headscale | database/sql.(*Stmt).ExecContext.func1(0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2646 +0xb8 fp=0x34d57a8 sp=0x34d5750 pc=0x73ba88
headscale | database/sql.(*DB).retry(0x3508dc8, 0x34d57f0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1568 +0x78 fp=0x34d57d0 sp=0x34d57a8 pc=0x736d5c
headscale | database/sql.(*Stmt).ExecContext(0x35d82a0, {0x1a6deb4, 0x27acfa8}, {0x363a580, 0x0, 0x8})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2640 +0xe0 fp=0x34d5820 sp=0x34d57d0 pc=0x73b978
headscale | gorm.io/gorm.(*PreparedStmtDB).ExecContext(0x36e4760, {0x1a6deb4, 0x27acfa8}, {0x3506780, 0x7e}, {0x363a580, 0x0, 0x8})
headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/prepare_stmt.go:151 +0xfc fp=0x34d58a0 sp=0x34d5820 pc=0x7c234c
headscale | gorm.io/gorm/callbacks.RawExec(0x36e47c0)
headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/callbacks/raw.go:9 +0x90 fp=0x34d58d4 sp=0x34d58a0 pc=0x967b2c
headscale | gorm.io/gorm.(*processor).Execute(0x3718330, 0x36e47c0)
headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/callbacks.go:130 +0x3fc fp=0x34d5954 sp=0x34d58d4 pc=0x7b3194
headscale | gorm.io/gorm.(*DB).Exec(0x348b680, {0x17dd172, 0x7e}, {0x0, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/finisher_api.go:769 +0x188 fp=0x34d5994 sp=0x34d5954 pc=0x7be7ec
headscale | github.com/juanfont/headscale/hscontrol/db.openDB({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...})
headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:462 +0xc3c fp=0x34d5a84 sp=0x34d5994 pc=0xd182ec
headscale | github.com/juanfont/headscale/hscontrol/db.NewHeadscaleDatabase({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...}, ...)
headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:45 +0x38 fp=0x34d5b40 sp=0x34d5a84 pc=0xd14454
headscale | github.com/juanfont/headscale/hscontrol.NewHeadscale(0x37386c8)
headscale | /home/runner/work/headscale/headscale/hscontrol/app.go:139 +0x188 fp=0x34d5cc8 sp=0x34d5b40 pc=0x13eb474
headscale | github.com/juanfont/headscale/cmd/headscale/cli.newHeadscaleServerWithConfig()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:35 +0x8c fp=0x34d5cf0 sp=0x34d5cc8 pc=0x141e14c
headscale | github.com/juanfont/headscale/cmd/headscale/cli.init.func28(0x2795238, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:22 +0x14 fp=0x34d5d14 sp=0x34d5cf0 pc=0x14171bc
headscale | github.com/spf13/cobra.(*Command).execute(0x2795238, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0x9b0 fp=0x34d5dd4 sp=0x34d5d14 pc=0x867fb8
headscale | github.com/spf13/cobra.(*Command).ExecuteC(0x2794968)
headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x44c fp=0x34d5e4c sp=0x34d5dd4 pc=0x868890
headscale | github.com/spf13/cobra.(*Command).Execute(...)
headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
headscale | github.com/juanfont/headscale/cmd/headscale/cli.Execute()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/root.go:97 +0x20 fp=0x34d5e78 sp=0x34d5e4c pc=0x141d438
headscale | main.main()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/headscale.go:42 +0x1a4 fp=0x34d5fa8 sp=0x34d5e78 pc=0x141f93c
headscale | runtime.main()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:272 +0x2ec fp=0x34d5fec sp=0x34d5fa8 pc=0x54fe4
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34d5fec sp=0x34d5fec pc=0x9a508
headscale |
headscale | goroutine 2 gp=0x3402488 m=nil [force gc (idle)]:
headscale | runtime.gopark(0x183e348, 0x279d4e0, 0x11, 0xa, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345cfd4 sp=0x345cfc0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.forcegchelper()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:337 +0xe4 fp=0x345cfec sp=0x345cfd4 pc=0x55450
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345cfec sp=0x345cfec pc=0x9a508
headscale | created by runtime.init.6 in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:325 +0x1c
headscale |
headscale | goroutine 3 gp=0x34027e8 m=nil [GC sweep wait]:
headscale | runtime.gopark(0x183e348, 0x279e360, 0xc, 0x9, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345d7c4 sp=0x345d7b0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.bgsweep(0x347a000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:317 +0x11c fp=0x345d7e4 sp=0x345d7c4 pc=0x3af10
headscale | runtime.gcenable.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x28 fp=0x345d7ec sp=0x345d7e4 pc=0x2ad90
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345d7ec sp=0x345d7ec pc=0x9a508
headscale | created by runtime.gcenable in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x74
headscale |
headscale | goroutine 4 gp=0x3402908 m=nil [GC scavenge wait]:
headscale | runtime.gopark(0x183e348, 0x27a03e8, 0xd, 0xa, 0x2)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345dfb4 sp=0x345dfa0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.(*scavengerState).park(0x27a03e8)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:425 +0x68 fp=0x345dfc8 sp=0x345dfb4 pc=0x3827c
headscale | runtime.bgscavenge(0x347a000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:658 +0x60 fp=0x345dfe4 sp=0x345dfc8 pc=0x389a8
headscale | runtime.gcenable.gowrap2()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0x28 fp=0x345dfec sp=0x345dfe4 pc=0x2ad3c
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345dfec sp=0x345dfec pc=0x9a508
headscale | created by runtime.gcenable in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0xbc
headscale |
headscale | goroutine 17 gp=0x34bc008 m=nil [finalizer wait]:
headscale | runtime.gopark(0x183e1f0, 0x27ad038, 0x10, 0xa, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345c78c sp=0x345c778 pc=0x92a6c
headscale | runtime.runfinq()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:193 +0x110 fp=0x345c7ec sp=0x345c78c pc=0x29bbc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345c7ec sp=0x345c7ec pc=0x9a508
headscale | created by runtime.createfing in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:163 +0x5c
headscale |
headscale | goroutine 18 gp=0x34bca28 m=nil [chan receive]:
headscale | runtime.gopark(0x183e1d0, 0x3489534, 0xe, 0x7, 0x2)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345878c sp=0x3458778 pc=0x92a6c
headscale | runtime.chanrecv(0x3489500, 0x0, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x4e0 fp=0x34587c8 sp=0x345878c pc=0x18648
headscale | runtime.chanrecv1(0x3489500, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:489 +0x20 fp=0x34587dc sp=0x34587c8 pc=0x18138
headscale | runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1732
headscale | runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1735 +0x40 fp=0x34587ec sp=0x34587dc pc=0x2edb4
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34587ec sp=0x34587ec pc=0x9a508
headscale | created by unique.runtime_registerUniqueMapCleanup in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1730 +0xa4
headscale |
headscale | goroutine 51 gp=0x35977a8 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x34596f4 sp=0x34596e0 pc=0x92a6c
headscale | runtime.selectgo(0x34597d0, 0x34597c4, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x34597a0 sp=0x34596f4 pc=0x6a574
headscale | github.com/patrickmn/go-cache.(*janitor).Run(0x38f5950, 0x37181b0)
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x98 fp=0x34597e0 sp=0x34597a0 pc=0xcf9ee4
headscale | github.com/patrickmn/go-cache.runJanitor.gowrap1()
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0x30 fp=0x34597ec sp=0x34597e0 pc=0xcfa0b8
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34597ec sp=0x34597ec pc=0x9a508
headscale | created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xfc
headscale |
headscale | goroutine 52 gp=0x373afc8 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x3459f00 sp=0x3459eec pc=0x92a6c
headscale | runtime.selectgo(0x3459fdc, 0x3459fd0, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x3459fac sp=0x3459f00 pc=0x6a574
headscale | github.com/juanfont/headscale/hscontrol/notifier.(*batcher).doWork(...)
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:419
headscale | github.com/juanfont/headscale/hscontrol/notifier.NewNotifier.gowrap1()
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x8c fp=0x3459fec sp=0x3459fac pc=0xe51360
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x3459fec sp=0x3459fec pc=0x9a508
headscale | created by github.com/juanfont/headscale/hscontrol/notifier.NewNotifier in goroutine 1
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x1b0
headscale |
headscale | goroutine 15 gp=0x373b0e8 m=nil [GC worker (idle)]:
headscale | runtime.gopark(0x183e200, 0x3442960, 0x1a, 0xa, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345ef88 sp=0x345ef74 pc=0x92a6c
headscale | runtime.gcBgMarkWorker(0x37e4e40)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345efe4 sp=0x345ef88 pc=0x2dbec
headscale | runtime.gcBgMarkStartWorkers.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345efec sp=0x345efe4 pc=0x2dacc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345efec sp=0x345efec pc=0x9a508
headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c
headscale |
headscale | goroutine 49 gp=0x373b208 m=nil [IO wait]:
headscale | runtime.gopark(0x183e33c, 0x76c02e88, 0x2, 0x2, 0x5)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x36e3ae0 sp=0x36e3acc pc=0x92a6c
headscale | runtime.netpollblock(0x76c02e78, 0x72, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:575 +0x100 fp=0x36e3af8 sp=0x36e3ae0 pc=0x4d420
headscale | internal/poll.runtime_pollWait(0x76c02e78, 0x72)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:351 +0x54 fp=0x36e3b0c sp=0x36e3af8 pc=0x91c04
headscale | internal/poll.(*pollDesc).wait(0x36d01f8, 0x72, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:84 +0x30 fp=0x36e3b20 sp=0x36e3b0c pc=0xe3c20
headscale | internal/poll.(*pollDesc).waitRead(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:89
headscale | internal/poll.(*FD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_unix.go:165 +0x22c fp=0x36e3b68 sp=0x36e3b20 pc=0xe4d9c
headscale | net.(*netFD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/fd_posix.go:55 +0x38 fp=0x36e3b94 sp=0x36e3b68 pc=0x360bec
headscale | net.(*conn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/net.go:189 +0x48 fp=0x36e3bc0 sp=0x36e3b94 pc=0x373be8
headscale | net.(*TCPConn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000})
headscale | :1 +0x44 fp=0x36e3be0 sp=0x36e3bc0 pc=0x388f94
headscale | crypto/tls.(*atLeastReader).Read(0x38f5140, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:809 +0x78 fp=0x36e3c0c sp=0x36e3be0 pc=0x3d6208
headscale | bytes.(*Buffer).ReadFrom(0x37395d4, {0x1a62418, 0x38f5140})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bytes/buffer.go:211 +0xa4 fp=0x36e3c48 sp=0x36e3c0c pc=0x135a08
headscale | crypto/tls.(*Conn).readFromUntil(0x3739448, {0x1a61db8, 0x3798860}, 0x5)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:831 +0xd4 fp=0x36e3c70 sp=0x36e3c48 pc=0x3d6464
headscale | crypto/tls.(*Conn).readRecordOrCCS(0x3739448, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:629 +0x134 fp=0x36e3dd0 sp=0x36e3c70 pc=0x3d3b68
headscale | crypto/tls.(*Conn).readRecord(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:591
headscale | crypto/tls.(*Conn).Read(0x3739448, {0x35a9000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:1385 +0x148 fp=0x36e3e00 sp=0x36e3dd0 pc=0x3d9940
headscale | bufio.(*Reader).Read(0x37fd440, {0x34da024, 0x9, 0x9})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bufio/bufio.go:241 +0x214 fp=0x36e3e24 sp=0x36e3e00 pc=0x201338
headscale | io.ReadAtLeast({0x1a61d78, 0x37fd440}, {0x34da024, 0x9, 0x9}, 0x9)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:335 +0x90 fp=0x36e3e50 sp=0x36e3e24 pc=0xd8560
headscale | io.ReadFull(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:354
headscale | net/http.http2readFrameHeader({0x34da024, 0x9, 0x9}, {0x1a61d78, 0x37fd440})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1642 +0x54 fp=0x36e3e78 sp=0x36e3e50 pc=0x455bb4
headscale | net/http.(*http2Framer).ReadFrame(0x34da000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1909 +0x88 fp=0x36e3ef4 sp=0x36e3e78 pc=0x45634c
headscale | net/http.(*http2clientConnReadLoop).run(0x36e3fdc)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9496 +0x10c fp=0x36e3fa4 sp=0x36e3ef4 pc=0x47ae4c
headscale | net/http.(*http2ClientConn).readLoop(0x3638008)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9392 +0x80 fp=0x36e3fe4 sp=0x36e3fa4 pc=0x47a3dc
headscale | net/http.(*http2Transport).newClientConn.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0x28 fp=0x36e3fec sp=0x36e3fe4 pc=0x47334c
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x36e3fec sp=0x36e3fec pc=0x9a508
headscale | created by net/http.(*http2Transport).newClientConn in goroutine 16
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0xd3c
headscale |
headscale | goroutine 35 gp=0x373b568 m=nil [GC worker (idle)]:
headscale | runtime.gopark(0x183e200, 0x3442948, 0x1a, 0xa, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345e788 sp=0x345e774 pc=0x92a6c
headscale | runtime.gcBgMarkWorker(0x37e4e40)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345e7e4 sp=0x345e788 pc=0x2dbec
headscale | runtime.gcBgMarkStartWorkers.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345e7ec sp=0x345e7e4 pc=0x2dacc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345e7ec sp=0x345e7ec pc=0x9a508
headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c
headscale |
headscale | goroutine 53 gp=0x373bb08 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345a6f4 sp=0x345a6e0 pc=0x92a6c
headscale | runtime.selectgo(0x345a7cc, 0x345a7c4, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x345a7a0 sp=0x345a6f4 pc=0x6a574
headscale | database/sql.(*DB).connectionOpener(0x3508dc8, {0x1a6df98, 0x3718360})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1253 +0x9c fp=0x345a7dc sp=0x345a7a0 pc=0x735300
headscale | database/sql.OpenDB.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x38 fp=0x345a7ec sp=0x345a7dc pc=0x7335ac
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345a7ec sp=0x345a7ec pc=0x9a508
headscale | created by database/sql.OpenDB in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x13c
headscale exited with code 0

Expected Behavior

Opening the SQLite db does not crash the startup process of headscale on armv7.

Steps To Reproduce

  1. Run headscale in docker container using the headscale/headscale image for linux/arm/v7 on an armv7 board
  2. Start the container
  3. headscale starts, tries to open the SQLite db and fails
  4. See error

Environment

- OS: Alpine Linux 3.20
- Headscale version: 0.23
- Tailscale version: xx

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

I built the docker image using debian:slim as a base; it fails with the exact same error.

Originally created by @m-kloeckner on GitHub (Oct 11, 2024). ### Is this a support request? - [X] This is not a support request ### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior I am running headscale on a armv7 SOC (Cubieboard2) in a docker container since quite a while now. After switching to version 0.23 I'm running in an error. When trying to start headscale using the official headscale/headscale docker image for armv7 the process fails with the following error after trying to open the SQLite db. > headscale | 2024-10-11T08:13:30Z WRN headscale | WARN: The "dns.use_username_in_magic_dns" configuration key is deprecated and has been removed. Please see the changelog for more details. headscale | headscale | 2024-10-11T08:13:30Z INF No private key file at path, creating... path=/var/lib/headscale/noise_private.key headscale | 2024-10-11T08:13:30Z INF Opening database database=sqlite3 path=/etc/headscale/db.sqlite headscale | unexpected fault address 0x769affff headscale | fatal error: fault headscale | [signal SIGBUS: bus error code=0x2 addr=0x769affff pc=0x979e58] headscale | headscale | goroutine 1 gp=0x3402128 m=4 mp=0x3461408 [running]: headscale | runtime.throw({0x175ca21, 0x5}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/panic.go:1067 +0x34 fp=0x34d4e1c sp=0x34d4e08 pc=0x9292c headscale | runtime.sigpanic() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/signal_unix.go:897 +0x104 fp=0x34d4e4c sp=0x34d4e1c pc=0x94cd0 headscale | modernc.org/libc.Xmemset(0x35ca360, 0x769a8088, 0x0, 0x7f78) headscale | /home/runner/go/pkg/mod/modernc.org/libc@v1.60.1/ccgo_linux_arm.go:146120 +0x24 fp=0x34d4e50 sp=0x34d4e50 pc=0x979e58 headscale | modernc.org/sqlite/lib._walIndexAppend(0x35ca360, 0x76600718, 0x1, 0x1) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:43350 +0xe4 fp=0x34d4e88 sp=0x34d4e50 pc=0x9caf58 headscale | modernc.org/sqlite/lib._walFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:46136 +0xa48 fp=0x34d4f0c sp=0x34d4e88 pc=0x9d1404 headscale | modernc.org/sqlite/lib._sqlite3WalFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:46177 +0x4c fp=0x34d4f30 sp=0x34d4f0c pc=0x9d1728 headscale | modernc.org/sqlite/lib._pagerWalFrames(0x35ca360, 0x76000418, 0x75b07038, 0x1, 0x1) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:37777 +0x114 fp=0x34d4f70 sp=0x34d4f30 pc=0x9c1d88 headscale | modernc.org/sqlite/lib._sqlite3PagerCommitPhaseOne(0x35ca360, 0x76000418, 0x0, 0x0) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:40996 +0x1b0 fp=0x34d4fa4 sp=0x34d4f70 pc=0x9c8510 headscale | modernc.org/sqlite/lib._sqlite3BtreeCommitPhaseOne(0x35ca360, 0x75e00058, 0x0) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:51705 +0xbc fp=0x34d4fc4 sp=0x34d4fa4 pc=0x9db3a0 headscale | modernc.org/sqlite/lib._vdbeCommit(0x35ca360, 0x76000018, 0x75a08808) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:64640 +0xdcc fp=0x34d508c sp=0x34d4fc4 pc=0x9fbc6c headscale | modernc.org/sqlite/lib._sqlite3VdbeHalt(0x35ca360, 0x75a08808) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:65053 +0x348 fp=0x34d50b4 sp=0x34d508c pc=0x9fc42c headscale | modernc.org/sqlite/lib._sqlite3VdbeExec(0x35ca360, 0x75a08808) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:71196 +0x4920 fp=0x34d5594 sp=0x34d50b4 pc=0xa0c450 headscale | modernc.org/sqlite/lib._sqlite3Step(0x35ca360, 0x75a08808) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:67892 +0x8c fp=0x34d55b4 sp=0x34d5594 pc=0xa02ad0 headscale | modernc.org/sqlite/lib.Xsqlite3_step(0x35ca360, 0x75a08808) headscale | /home/runner/go/pkg/mod/modernc.org/sqlite@v1.32.0/lib/sqlite_linux_arm.go:67959 +0xdc fp=0x34d55e4 sp=0x34d55b4 pc=0xa02f2c headscale | github.com/glebarez/go-sqlite.(*conn).step(0x348b7e0, 0x75a08808) headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:1002 +0x28 fp=0x34d55fc sp=0x34d55e4 pc=0xb68c90 headscale | github.com/glebarez/go-sqlite.(*stmt).exec.func1(0x34fa6c8, 0x34d566c, {0x27acfa8, 0x0, 0x0}) headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:536 +0x128 fp=0x34d5648 sp=0x34d55fc pc=0xb66aac headscale | github.com/glebarez/go-sqlite.(*stmt).exec(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0}) headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite.go:549 +0x1a0 fp=0x34d5680 sp=0x34d5648 pc=0xb66820 headscale | github.com/glebarez/go-sqlite.(*stmt).ExecContext(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0}) headscale | /home/runner/go/pkg/mod/github.com/glebarez/go-sqlite@v1.22.0/sqlite_go18.go:43 +0x44 fp=0x34d56ac sp=0x34d5680 pc=0xb6ba98 headscale | database/sql.ctxDriverStmtExec({0x1a6deb4, 0x27acfa8}, {0x1a6e584, 0x34fa6c8}, {0x27acfa8, 0x0, 0x0}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/ctxutil.go:65 +0xb0 fp=0x34d56fc sp=0x34d56ac pc=0x730da0 headscale | database/sql.resultFromStatement({0x1a6deb4, 0x27acfa8}, {0x1a6d0a0, 0x348b7e0}, 0x36e4800, {0x363a580, 0x0, 0x8}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2672 +0xfc fp=0x34d5750 sp=0x34d56fc pc=0x73bce8 headscale | database/sql.(*Stmt).ExecContext.func1(0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2646 +0xb8 fp=0x34d57a8 sp=0x34d5750 pc=0x73ba88 headscale | database/sql.(*DB).retry(0x3508dc8, 0x34d57f0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1568 +0x78 fp=0x34d57d0 sp=0x34d57a8 pc=0x736d5c headscale | database/sql.(*Stmt).ExecContext(0x35d82a0, {0x1a6deb4, 0x27acfa8}, {0x363a580, 0x0, 0x8}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2640 +0xe0 fp=0x34d5820 sp=0x34d57d0 pc=0x73b978 headscale | gorm.io/gorm.(*PreparedStmtDB).ExecContext(0x36e4760, {0x1a6deb4, 0x27acfa8}, {0x3506780, 0x7e}, {0x363a580, 0x0, 0x8}) headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/prepare_stmt.go:151 +0xfc fp=0x34d58a0 sp=0x34d5820 pc=0x7c234c headscale | gorm.io/gorm/callbacks.RawExec(0x36e47c0) headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/callbacks/raw.go:9 +0x90 fp=0x34d58d4 sp=0x34d58a0 pc=0x967b2c headscale | gorm.io/gorm.(*processor).Execute(0x3718330, 0x36e47c0) headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/callbacks.go:130 +0x3fc fp=0x34d5954 sp=0x34d58d4 pc=0x7b3194 headscale | gorm.io/gorm.(*DB).Exec(0x348b680, {0x17dd172, 0x7e}, {0x0, 0x0, 0x0}) headscale | /home/runner/go/pkg/mod/gorm.io/gorm@v1.25.11/finisher_api.go:769 +0x188 fp=0x34d5994 sp=0x34d5954 pc=0x7be7ec headscale | github.com/juanfont/headscale/hscontrol/db.openDB({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...}) headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:462 +0xc3c fp=0x34d5a84 sp=0x34d5994 pc=0xd182ec headscale | github.com/juanfont/headscale/hscontrol/db.NewHeadscaleDatabase({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...}, ...) headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:45 +0x38 fp=0x34d5b40 sp=0x34d5a84 pc=0xd14454 headscale | github.com/juanfont/headscale/hscontrol.NewHeadscale(0x37386c8) headscale | /home/runner/work/headscale/headscale/hscontrol/app.go:139 +0x188 fp=0x34d5cc8 sp=0x34d5b40 pc=0x13eb474 headscale | github.com/juanfont/headscale/cmd/headscale/cli.newHeadscaleServerWithConfig() headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:35 +0x8c fp=0x34d5cf0 sp=0x34d5cc8 pc=0x141e14c headscale | github.com/juanfont/headscale/cmd/headscale/cli.init.func28(0x2795238, {0x27acfa8, 0x0, 0x0}) headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:22 +0x14 fp=0x34d5d14 sp=0x34d5cf0 pc=0x14171bc headscale | github.com/spf13/cobra.(*Command).execute(0x2795238, {0x27acfa8, 0x0, 0x0}) headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0x9b0 fp=0x34d5dd4 sp=0x34d5d14 pc=0x867fb8 headscale | github.com/spf13/cobra.(*Command).ExecuteC(0x2794968) headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x44c fp=0x34d5e4c sp=0x34d5dd4 pc=0x868890 headscale | github.com/spf13/cobra.(*Command).Execute(...) headscale | /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 headscale | github.com/juanfont/headscale/cmd/headscale/cli.Execute() headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/root.go:97 +0x20 fp=0x34d5e78 sp=0x34d5e4c pc=0x141d438 headscale | main.main() headscale | /home/runner/work/headscale/headscale/cmd/headscale/headscale.go:42 +0x1a4 fp=0x34d5fa8 sp=0x34d5e78 pc=0x141f93c headscale | runtime.main() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:272 +0x2ec fp=0x34d5fec sp=0x34d5fa8 pc=0x54fe4 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34d5fec sp=0x34d5fec pc=0x9a508 headscale | headscale | goroutine 2 gp=0x3402488 m=nil [force gc (idle)]: headscale | runtime.gopark(0x183e348, 0x279d4e0, 0x11, 0xa, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345cfd4 sp=0x345cfc0 pc=0x92a6c headscale | runtime.goparkunlock(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430 headscale | runtime.forcegchelper() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:337 +0xe4 fp=0x345cfec sp=0x345cfd4 pc=0x55450 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345cfec sp=0x345cfec pc=0x9a508 headscale | created by runtime.init.6 in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:325 +0x1c headscale | headscale | goroutine 3 gp=0x34027e8 m=nil [GC sweep wait]: headscale | runtime.gopark(0x183e348, 0x279e360, 0xc, 0x9, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345d7c4 sp=0x345d7b0 pc=0x92a6c headscale | runtime.goparkunlock(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430 headscale | runtime.bgsweep(0x347a000) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:317 +0x11c fp=0x345d7e4 sp=0x345d7c4 pc=0x3af10 headscale | runtime.gcenable.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x28 fp=0x345d7ec sp=0x345d7e4 pc=0x2ad90 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345d7ec sp=0x345d7ec pc=0x9a508 headscale | created by runtime.gcenable in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x74 headscale | headscale | goroutine 4 gp=0x3402908 m=nil [GC scavenge wait]: headscale | runtime.gopark(0x183e348, 0x27a03e8, 0xd, 0xa, 0x2) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345dfb4 sp=0x345dfa0 pc=0x92a6c headscale | runtime.goparkunlock(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430 headscale | runtime.(*scavengerState).park(0x27a03e8) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:425 +0x68 fp=0x345dfc8 sp=0x345dfb4 pc=0x3827c headscale | runtime.bgscavenge(0x347a000) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:658 +0x60 fp=0x345dfe4 sp=0x345dfc8 pc=0x389a8 headscale | runtime.gcenable.gowrap2() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0x28 fp=0x345dfec sp=0x345dfe4 pc=0x2ad3c headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345dfec sp=0x345dfec pc=0x9a508 headscale | created by runtime.gcenable in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0xbc headscale | headscale | goroutine 17 gp=0x34bc008 m=nil [finalizer wait]: headscale | runtime.gopark(0x183e1f0, 0x27ad038, 0x10, 0xa, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345c78c sp=0x345c778 pc=0x92a6c headscale | runtime.runfinq() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:193 +0x110 fp=0x345c7ec sp=0x345c78c pc=0x29bbc headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345c7ec sp=0x345c7ec pc=0x9a508 headscale | created by runtime.createfing in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:163 +0x5c headscale | headscale | goroutine 18 gp=0x34bca28 m=nil [chan receive]: headscale | runtime.gopark(0x183e1d0, 0x3489534, 0xe, 0x7, 0x2) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345878c sp=0x3458778 pc=0x92a6c headscale | runtime.chanrecv(0x3489500, 0x0, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x4e0 fp=0x34587c8 sp=0x345878c pc=0x18648 headscale | runtime.chanrecv1(0x3489500, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:489 +0x20 fp=0x34587dc sp=0x34587c8 pc=0x18138 headscale | runtime.unique_runtime_registerUniqueMapCleanup.func1(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1732 headscale | runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1735 +0x40 fp=0x34587ec sp=0x34587dc pc=0x2edb4 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34587ec sp=0x34587ec pc=0x9a508 headscale | created by unique.runtime_registerUniqueMapCleanup in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1730 +0xa4 headscale | headscale | goroutine 51 gp=0x35977a8 m=nil [select]: headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x34596f4 sp=0x34596e0 pc=0x92a6c headscale | runtime.selectgo(0x34597d0, 0x34597c4, 0x0, 0x0, 0x2, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x34597a0 sp=0x34596f4 pc=0x6a574 headscale | github.com/patrickmn/go-cache.(*janitor).Run(0x38f5950, 0x37181b0) headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x98 fp=0x34597e0 sp=0x34597a0 pc=0xcf9ee4 headscale | github.com/patrickmn/go-cache.runJanitor.gowrap1() headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0x30 fp=0x34597ec sp=0x34597e0 pc=0xcfa0b8 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34597ec sp=0x34597ec pc=0x9a508 headscale | created by github.com/patrickmn/go-cache.runJanitor in goroutine 1 headscale | /home/runner/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xfc headscale | headscale | goroutine 52 gp=0x373afc8 m=nil [select]: headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x3459f00 sp=0x3459eec pc=0x92a6c headscale | runtime.selectgo(0x3459fdc, 0x3459fd0, 0x0, 0x0, 0x2, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x3459fac sp=0x3459f00 pc=0x6a574 headscale | github.com/juanfont/headscale/hscontrol/notifier.(*batcher).doWork(...) headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:419 headscale | github.com/juanfont/headscale/hscontrol/notifier.NewNotifier.gowrap1() headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x8c fp=0x3459fec sp=0x3459fac pc=0xe51360 headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x3459fec sp=0x3459fec pc=0x9a508 headscale | created by github.com/juanfont/headscale/hscontrol/notifier.NewNotifier in goroutine 1 headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x1b0 headscale | headscale | goroutine 15 gp=0x373b0e8 m=nil [GC worker (idle)]: headscale | runtime.gopark(0x183e200, 0x3442960, 0x1a, 0xa, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345ef88 sp=0x345ef74 pc=0x92a6c headscale | runtime.gcBgMarkWorker(0x37e4e40) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345efe4 sp=0x345ef88 pc=0x2dbec headscale | runtime.gcBgMarkStartWorkers.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345efec sp=0x345efe4 pc=0x2dacc headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345efec sp=0x345efec pc=0x9a508 headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c headscale | headscale | goroutine 49 gp=0x373b208 m=nil [IO wait]: headscale | runtime.gopark(0x183e33c, 0x76c02e88, 0x2, 0x2, 0x5) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x36e3ae0 sp=0x36e3acc pc=0x92a6c headscale | runtime.netpollblock(0x76c02e78, 0x72, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:575 +0x100 fp=0x36e3af8 sp=0x36e3ae0 pc=0x4d420 headscale | internal/poll.runtime_pollWait(0x76c02e78, 0x72) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:351 +0x54 fp=0x36e3b0c sp=0x36e3af8 pc=0x91c04 headscale | internal/poll.(*pollDesc).wait(0x36d01f8, 0x72, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:84 +0x30 fp=0x36e3b20 sp=0x36e3b0c pc=0xe3c20 headscale | internal/poll.(*pollDesc).waitRead(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:89 headscale | internal/poll.(*FD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_unix.go:165 +0x22c fp=0x36e3b68 sp=0x36e3b20 pc=0xe4d9c headscale | net.(*netFD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/fd_posix.go:55 +0x38 fp=0x36e3b94 sp=0x36e3b68 pc=0x360bec headscale | net.(*conn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/net.go:189 +0x48 fp=0x36e3bc0 sp=0x36e3b94 pc=0x373be8 headscale | net.(*TCPConn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000}) headscale | <autogenerated>:1 +0x44 fp=0x36e3be0 sp=0x36e3bc0 pc=0x388f94 headscale | crypto/tls.(*atLeastReader).Read(0x38f5140, {0x37e6000, 0x1000, 0x1000}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:809 +0x78 fp=0x36e3c0c sp=0x36e3be0 pc=0x3d6208 headscale | bytes.(*Buffer).ReadFrom(0x37395d4, {0x1a62418, 0x38f5140}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bytes/buffer.go:211 +0xa4 fp=0x36e3c48 sp=0x36e3c0c pc=0x135a08 headscale | crypto/tls.(*Conn).readFromUntil(0x3739448, {0x1a61db8, 0x3798860}, 0x5) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:831 +0xd4 fp=0x36e3c70 sp=0x36e3c48 pc=0x3d6464 headscale | crypto/tls.(*Conn).readRecordOrCCS(0x3739448, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:629 +0x134 fp=0x36e3dd0 sp=0x36e3c70 pc=0x3d3b68 headscale | crypto/tls.(*Conn).readRecord(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:591 headscale | crypto/tls.(*Conn).Read(0x3739448, {0x35a9000, 0x1000, 0x1000}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:1385 +0x148 fp=0x36e3e00 sp=0x36e3dd0 pc=0x3d9940 headscale | bufio.(*Reader).Read(0x37fd440, {0x34da024, 0x9, 0x9}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bufio/bufio.go:241 +0x214 fp=0x36e3e24 sp=0x36e3e00 pc=0x201338 headscale | io.ReadAtLeast({0x1a61d78, 0x37fd440}, {0x34da024, 0x9, 0x9}, 0x9) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:335 +0x90 fp=0x36e3e50 sp=0x36e3e24 pc=0xd8560 headscale | io.ReadFull(...) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:354 headscale | net/http.http2readFrameHeader({0x34da024, 0x9, 0x9}, {0x1a61d78, 0x37fd440}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1642 +0x54 fp=0x36e3e78 sp=0x36e3e50 pc=0x455bb4 headscale | net/http.(*http2Framer).ReadFrame(0x34da000) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1909 +0x88 fp=0x36e3ef4 sp=0x36e3e78 pc=0x45634c headscale | net/http.(*http2clientConnReadLoop).run(0x36e3fdc) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9496 +0x10c fp=0x36e3fa4 sp=0x36e3ef4 pc=0x47ae4c headscale | net/http.(*http2ClientConn).readLoop(0x3638008) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9392 +0x80 fp=0x36e3fe4 sp=0x36e3fa4 pc=0x47a3dc headscale | net/http.(*http2Transport).newClientConn.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0x28 fp=0x36e3fec sp=0x36e3fe4 pc=0x47334c headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x36e3fec sp=0x36e3fec pc=0x9a508 headscale | created by net/http.(*http2Transport).newClientConn in goroutine 16 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0xd3c headscale | headscale | goroutine 35 gp=0x373b568 m=nil [GC worker (idle)]: headscale | runtime.gopark(0x183e200, 0x3442948, 0x1a, 0xa, 0x0) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345e788 sp=0x345e774 pc=0x92a6c headscale | runtime.gcBgMarkWorker(0x37e4e40) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345e7e4 sp=0x345e788 pc=0x2dbec headscale | runtime.gcBgMarkStartWorkers.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345e7ec sp=0x345e7e4 pc=0x2dacc headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345e7ec sp=0x345e7ec pc=0x9a508 headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c headscale | headscale | goroutine 53 gp=0x373bb08 m=nil [select]: headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345a6f4 sp=0x345a6e0 pc=0x92a6c headscale | runtime.selectgo(0x345a7cc, 0x345a7c4, 0x0, 0x0, 0x2, 0x1) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x345a7a0 sp=0x345a6f4 pc=0x6a574 headscale | database/sql.(*DB).connectionOpener(0x3508dc8, {0x1a6df98, 0x3718360}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1253 +0x9c fp=0x345a7dc sp=0x345a7a0 pc=0x735300 headscale | database/sql.OpenDB.gowrap1() headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x38 fp=0x345a7ec sp=0x345a7dc pc=0x7335ac headscale | runtime.goexit({}) headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345a7ec sp=0x345a7ec pc=0x9a508 headscale | created by database/sql.OpenDB in goroutine 1 headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x13c headscale exited with code 0 ### Expected Behavior Opening the SQLite db does not crash the startup process of headscale on armv7. ### Steps To Reproduce 1. Run headscale in docker container using the headscale/headscale image for linux/arm/v7 on an armv7 board 2. Start the container 3. headscale starts, tries to open the SQLite db and fails 4. See error ### Environment ```markdown - OS: Alpine Linux 3.20 - Headscale version: 0.23 - Tailscale version: xx ``` ### Runtime environment - [X] Headscale is behind a (reverse) proxy - [X] Headscale runs in a container ### Anything else? I built the docker image using debian:slim as a base; it fails with the exact same error.
adam added the bughelp wantedgood first issue labels 2025-12-29 02:24:32 +01:00
adam closed this issue 2025-12-29 02:24:32 +01:00
adam commented 2025-12-29 02:24:32 +01:00
Author
Owner
Copy Link

@maxlysenko commented on GitHub (Nov 28, 2024):

Hello,
Ran into the same issue on a ARMv7 board (OrangePi).
Found a statement about fragile modernc.org/libc dependency in the documentation of modernc.org/sqlite.
Tried changing the versions in the go.mod file and managed to build working executable.
The exact steps were:

  1. git clone https://github.com/juanfont/headscale.git
  2. cd headscale
  3. git checkout v0.23.0
  4. Modified go.mod file
--- go.mod.old
+++ go.mod.new
@@ -203,8 +203,8 @@
        google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
        gopkg.in/yaml.v2 v2.4.0 // indirect
        gvisor.dev/gvisor v0.0.0-20240722211153-64c016c92987 // indirect
-       modernc.org/libc v1.60.1 // indirect
+       modernc.org/libc v1.55.3 // indirect
        modernc.org/mathutil v1.6.0 // indirect
        modernc.org/memory v1.8.0 // indirect
-       modernc.org/sqlite v1.32.0 // indirect
+       modernc.org/sqlite v1.34.1 // indirect
 )
  1. go mod tidy
  2. GOOS="linux" GOARCH="arm" GOARM="7" go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale
@maxlysenko commented on GitHub (Nov 28, 2024): Hello, Ran into the same issue on a ARMv7 board (OrangePi). Found a statement about [fragile modernc.org/libc dependency](https://pkg.go.dev/modernc.org/sqlite#hdr-Fragile_modernc_org_libc_dependency) in the documentation of `modernc.org/sqlite`. Tried changing the versions in the go.mod file and managed to build working executable. The exact steps were: 1. `git clone https://github.com/juanfont/headscale.git` 2. `cd headscale` 3. `git checkout v0.23.0` 4. Modified go.mod file ```diff --- go.mod.old +++ go.mod.new @@ -203,8 +203,8 @@ google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gvisor.dev/gvisor v0.0.0-20240722211153-64c016c92987 // indirect - modernc.org/libc v1.60.1 // indirect + modernc.org/libc v1.55.3 // indirect modernc.org/mathutil v1.6.0 // indirect modernc.org/memory v1.8.0 // indirect - modernc.org/sqlite v1.32.0 // indirect + modernc.org/sqlite v1.34.1 // indirect ) ``` 5. `go mod tidy ` 6. `GOOS="linux" GOARCH="arm" GOARM="7" go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale`
adam commented 2025-12-29 02:24:33 +01:00
Author
Owner
Copy Link

@jumper047 commented on GitHub (Nov 30, 2024):

@maxlysenko thank you for your fix, has same issue on my rpi 1b (armv6). Your patch helps me too.

@jumper047 commented on GitHub (Nov 30, 2024): @maxlysenko thank you for your fix, has same issue on my rpi 1b (armv6). Your patch helps me too.
adam commented 2025-12-29 02:24:33 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 2, 2024):

I wonder if this would trigger when doing armv6 builds on other machinery, if that is the case, the github action for just running the build could just compile for all ARCHes so we get feedback when something like that breaks.

If someone is up for it, I'll accept a PR doing:

  • Keep the latest sqlite package, but revert libc to one that works for all ARCHes
  • changes the CI job to build all CPU arches so we detect this in the future
@kradalby commented on GitHub (Dec 2, 2024): I wonder if this would trigger when doing armv6 builds on other machinery, if that is the case, the github action for just running the build could just compile for all ARCHes so we get feedback when something like that breaks. If someone is up for it, I'll accept a PR doing: - Keep the latest sqlite package, but revert libc to one that works for all ARCHes - changes the CI job to build all CPU arches so we detect this in the future
adam commented 2025-12-29 02:24:33 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 5, 2024):

I've set up the build job in #2270, they all currently build, which means that the dependency might be in sync on master?

I dont have a Arm6 board at hand, but I've added so all the binaries are added to the build job, can someone have a look at trying the relevant binary from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270 to see if they start on the relevant board?

Please dont do in production, running the CLI or something should be fine.

@kradalby commented on GitHub (Dec 5, 2024): I've set up the build job in #2270, they all currently build, which means that the dependency might be in sync on master? I dont have a Arm6 board at hand, but I've added so all the binaries are added to the build job, can someone have a look at trying the relevant binary from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270 to see if they start on the relevant board? Please dont do in production, running the CLI or something should be fine.
adam commented 2025-12-29 02:24:33 +01:00
Author
Owner
Copy Link

@maxlysenko commented on GitHub (Dec 5, 2024):

Hello,
Tested arm7 build from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270, nothing changed, same crash when opening database.
The problem is probably not in the build process or cross-compilation, but in the versions of dependencies in the go.mod file.
Most likely nothing will change until you change version of modernc.org/libc to corresponding with modernc.org/sqlite.
According to description in modernc.org/sqlite docs you shoud use in your go.mod the exact same version of modernc.org/libc as in the go.mod file of the modernc.org/sqlite itself.
So if in your go.mod file you have modernc.org/sqlite v1.32.0 then you should use modernc.org/libc v1.55.3, because go.mod file for v1.32.0 of modernc.org/sqlite contains this version.
Also it seems from commit cc087474 that from v1.33.0 modernc.org/sqlite uses internal/libc, so bumping version of modernc.org/sqlite to v1.33.0 or higher also may help.
Upgrading only modernc.org/sqlite version did't work for me. Tried versions from v1.33.0 to v1.34.1.
Out of interest tried

git clone https://github.com/juanfont/headscale
cd headscale
git checkout v0.23.0
mv go.mod go.mod.old
go mod init github.com/juanfont/headscale
go mod tidy

to see which versions of modules go mod will pull automatically. The versions are

modernc.org/libc v1.55.3 // indirect
modernc.org/sqlite v1.33.1 // indirect

So for now my workaround is:

git clone https://github.com/juanfont/headscale
cd headscale
git checkout v0.23.0
go get modernc.org/libc@v1.55.3 modernc.org/sqlite@v1.33.1
go mod tidy
go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale
@maxlysenko commented on GitHub (Dec 5, 2024): Hello, Tested arm7 build from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270, nothing changed, same crash when opening database. The problem is probably not in the build process or cross-compilation, but in the versions of dependencies in the go.mod file. Most likely nothing will change until you change version of modernc.org/libc to corresponding with modernc.org/sqlite. According to [description](https://pkg.go.dev/modernc.org/sqlite#hdr-Fragile_modernc_org_libc_dependency) in modernc.org/sqlite docs you shoud use in your go.mod the exact same version of modernc.org/libc as in the go.mod file of the modernc.org/sqlite itself. So if in your go.mod file you have `modernc.org/sqlite v1.32.0` then you should use `modernc.org/libc v1.55.3`, because go.mod file for v1.32.0 of modernc.org/sqlite contains this version. ~~Also it seems from commit [cc087474](https://gitlab.com/cznic/sqlite/-/commit/cc087474129cdd331e2b018ff441af66c8be93aa) that from v1.33.0 modernc.org/sqlite uses internal/libc, so bumping version of modernc.org/sqlite to v1.33.0 or higher also may help.~~ Upgrading only modernc.org/sqlite version did't work for me. Tried versions from v1.33.0 to v1.34.1. Out of interest tried ```shell git clone https://github.com/juanfont/headscale cd headscale git checkout v0.23.0 mv go.mod go.mod.old go mod init github.com/juanfont/headscale go mod tidy ``` to see which versions of modules go mod will pull automatically. The versions are ``` modernc.org/libc v1.55.3 // indirect modernc.org/sqlite v1.33.1 // indirect ``` So for now my workaround is: ```shell git clone https://github.com/juanfont/headscale cd headscale git checkout v0.23.0 go get modernc.org/libc@v1.55.3 modernc.org/sqlite@v1.33.1 go mod tidy go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale ```
adam commented 2025-12-29 02:24:34 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 6, 2024):

hmm, I need a way to detect this at build time, we dont have any machinery available to ensure we dont have this regression at runtime. Open for proposals.

@kradalby commented on GitHub (Dec 6, 2024): hmm, I need a way to detect this at build time, we dont have any machinery available to ensure we dont have this regression at runtime. Open for proposals.
adam commented 2025-12-29 02:24:34 +01:00
Author
Owner
Copy Link

@maxlysenko commented on GitHub (Dec 6, 2024):

modernc.org/libc was upgraded to v1.60.1 in go.mod in #2098.
Why it was done, why this version was chosen?
As far as I understand (looking at the go mod graph and go mod why output) none of the imported packages/modules requires this version.
So just downgrading modernc.org/libc to v1.55.3 in go.mod should fix the problem.
Unfortunatly I do not know how to detect runtime problem at build time.

@maxlysenko commented on GitHub (Dec 6, 2024): modernc.org/libc was upgraded to v1.60.1 in go.mod in #2098. Why it was done, why this version was chosen? As far as I understand (looking at the `go mod graph` and `go mod why` output) none of the imported packages/modules requires this version. So just downgrading modernc.org/libc to v1.55.3 in go.mod should fix the problem. Unfortunatly I do not know how to detect runtime problem at build time.
adam commented 2025-12-29 02:24:34 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 6, 2024):

In general they are just updated to latest, there is no other rational, so I am happy to have them adjusted so it works again, what I am concerned about is me/someone else upgrading it without ensuring the lockstep.

Typically I just run a go get -u to upgrade all. If we dont have a test/ci failing, it will be hard to detect/prevent this.

@kradalby commented on GitHub (Dec 6, 2024): In general they are just updated to latest, there is no other rational, so I am happy to have them adjusted so it works again, what I am concerned about is me/someone else upgrading it without ensuring the lockstep. Typically I just run a `go get -u` to upgrade all. If we dont have a test/ci failing, it will be hard to detect/prevent this.
adam commented 2025-12-29 02:24:34 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 6, 2024):

I've updated my PR with the dependency version you mention, but I am hesitant calling it a "fix" as it doesnt protect for the future.

@kradalby commented on GitHub (Dec 6, 2024): I've updated my PR with the dependency version you mention, but I am hesitant calling it a "fix" as it doesnt protect for the future.
adam commented 2025-12-29 02:24:34 +01:00
Author
Owner
Copy Link

@maxlysenko commented on GitHub (Dec 6, 2024):

Tested arm builds from https://github.com/juanfont/headscale/actions/runs/12199297972, they don't crash on database access, they start and work.

@maxlysenko commented on GitHub (Dec 6, 2024): Tested arm builds from https://github.com/juanfont/headscale/actions/runs/12199297972, they don't crash on database access, they start and work.
adam commented 2025-12-29 02:24:35 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Dec 9, 2024):

Great, the best measure I think I can do for now to ensure we keep this in lock is documentation, I have added a section to our go.mod file in f1b97dd (#2270)

@kradalby commented on GitHub (Dec 9, 2024): Great, the best measure I think I can do for now to ensure we keep this in lock is documentation, I have added a section to our go.mod file in [`f1b97dd` (#2270)](https://github.com/juanfont/headscale/pull/2270/commits/f1b97dd260c0472856abb6fcb20f2b951867ffe1)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug good first issue help wanted
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#828
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?