starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

Node with empty disco_key in DB prevents to start headscale #661

New Issue
Closed
opened 2025-12-29 02:21:45 +01:00 by adam · 2 comments
adam commented 2025-12-29 02:21:45 +01:00
Owner
Copy Link

Originally created by @winterheart on GitHub (Mar 6, 2024).

Bug description

On updating headscale from 0.22.3 to 0.23.0-alpha5 headscale fails to starts due incorrect entry in database. One of nodes had empty disco_key ("disco_key:"). This entry is causing to headscale fail to start:

headscale Cannot get nodes: failed to unmarshal disco key from db: key hex has the wrong size, got 0 want 64

After manually deleting that node in database headscale starts normally. I have no idea how I got that node in database, but in 0.22.3 this error is not critical in order to start headscale.

Environment

  • OS: Linux (Docker container)
  • Headscale version: 0.23.0-alpha5
  • Tailscale version: 1.58
  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

To Reproduce

Have node with empty disco_key in database, restart Headscale.

Logs and attachments

Originally created by @winterheart on GitHub (Mar 6, 2024). <!-- Before posting a bug report, discuss the behaviour you are expecting with the Discord community to make sure that it is truly a bug. The issue tracker is not the place to ask for support or how to set up Headscale. Bug reports without the sufficient information will be closed. Headscale is a multinational community across the globe. Our language is English. All bug reports needs to be in English. --> ## Bug description <!-- A clear and concise description of what the bug is. Describe the expected bahavior and how it is currently different. If you are unsure if it is a bug, consider discussing it on our Discord server first. --> On updating headscale from 0.22.3 to 0.23.0-alpha5 headscale fails to starts due incorrect entry in database. One of nodes had empty disco_key ("disco_key:"). This entry is causing to headscale fail to start: ``` headscale Cannot get nodes: failed to unmarshal disco key from db: key hex has the wrong size, got 0 want 64 ``` After manually deleting that node in database headscale starts normally. I have no idea how I got that node in database, but in 0.22.3 this error is not critical in order to start headscale. ## Environment <!-- Please add relevant information about your system. For example: - Version of headscale used - Version of tailscale client - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version - Kernel version - The relevant config parameters you used - Log output --> - OS: Linux (Docker container) - Headscale version: 0.23.0-alpha5 - Tailscale version: 1.58 <!-- We do not support running Headscale in a container nor behind a (reverse) proxy. If either of these are true for your environment, ask the community in Discord instead of filing a bug report. --> - [ ] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ## To Reproduce <!-- Steps to reproduce the behavior. --> Have node with empty disco_key in database, restart Headscale. ## Logs and attachments <!-- Please attach files with: - Client netmap dump (see below) - ACL configuration - Headscale configuration Dump the netmap of tailscale clients: `tailscale debug netmap > DESCRIPTIVE_NAME.json` Please provide information describing the netmap, which client, which headscale version etc. -->
adam added the bug label 2025-12-29 02:21:45 +01:00
adam closed this issue 2025-12-29 02:21:45 +01:00
adam commented 2025-12-29 02:21:45 +01:00
Author
Owner
Copy Link

@gabe565 commented on GitHub (Sep 18, 2024):

@kradalby I know this issue has been resolved for a while, but I have two separate Headscale instances that I upgraded to v0.23.0 today. One upgrade went smoothly, but the other logged the error repeatedly for all clients:

2024-09-18T17:54:17Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:705 > node has connected, mapSession: 0xc00015e600, chan: 0xc000444380 node=... node.id=26 omitPeers=false readOnly=false stream=true
2024-09-18T17:54:17Z ERR home/runner/work/headscale/headscale/hscontrol/poll.go:725 > Could not get the create map update error="unmarshalling disco key from db: key hex has the wrong size, got 0 want 64" node=... node.id=26 omitPeers=false readOnly=false stream=true
2024-09-18T17:54:17Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:705 > node has disconnected, mapSession: 0xc00015e600, chan: 0xc000444380 node=... node.id=26 omitPeers=false readOnly=false stream=true

I really appreciate the work you do! I'm really excited for v0.23.0.

Edit: Interestingly, the same error was logged during startup.

2024-09-18T17:54:13Z INF Opening database database=postgres path="host=postgresql-rw dbname=headscale user=headscale"
2024-09-18T17:54:13Z ERR Error accessing db error="unmarshalling disco key from db: key hex has the wrong size, got 0 want 64"
2024-09-18T17:54:14Z INF Setting up a DERPMap update worker frequency=86400000
2024-09-18T17:54:14Z WRN Listening without TLS but ServerURL does not start with http://
2024-09-18T17:54:14Z INF listening and serving HTTP on: 0.0.0.0:8080
2024-09-18T17:54:14Z INF listening and serving debug and metrics on: 0.0.0.0:9090

I wonder if the migration from machines to nodes failed because of the error?

Another Edit: Yep, I think the migration may have failed. I noticed there was a device that hadn't connected in ~5 months, and it had an empty disco_key. I removed that device before the upgrade, and everything is working this time around!

@gabe565 commented on GitHub (Sep 18, 2024): @kradalby I know this issue has been resolved for a while, but I have two separate Headscale instances that I upgraded to v0.23.0 today. One upgrade went smoothly, but the other logged the error repeatedly for all clients: ``` 2024-09-18T17:54:17Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:705 > node has connected, mapSession: 0xc00015e600, chan: 0xc000444380 node=... node.id=26 omitPeers=false readOnly=false stream=true 2024-09-18T17:54:17Z ERR home/runner/work/headscale/headscale/hscontrol/poll.go:725 > Could not get the create map update error="unmarshalling disco key from db: key hex has the wrong size, got 0 want 64" node=... node.id=26 omitPeers=false readOnly=false stream=true 2024-09-18T17:54:17Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:705 > node has disconnected, mapSession: 0xc00015e600, chan: 0xc000444380 node=... node.id=26 omitPeers=false readOnly=false stream=true ``` I really appreciate the work you do! I'm really excited for v0.23.0. Edit: Interestingly, the same error was logged during startup. ``` 2024-09-18T17:54:13Z INF Opening database database=postgres path="host=postgresql-rw dbname=headscale user=headscale" 2024-09-18T17:54:13Z ERR Error accessing db error="unmarshalling disco key from db: key hex has the wrong size, got 0 want 64" 2024-09-18T17:54:14Z INF Setting up a DERPMap update worker frequency=86400000 2024-09-18T17:54:14Z WRN Listening without TLS but ServerURL does not start with http:// 2024-09-18T17:54:14Z INF listening and serving HTTP on: 0.0.0.0:8080 2024-09-18T17:54:14Z INF listening and serving debug and metrics on: 0.0.0.0:9090 ``` I wonder if the migration from machines to nodes failed because of the error? Another Edit: Yep, I think the migration may have failed. I noticed there was a device that hadn't connected in ~5 months, and it had an empty `disco_key`. I removed that device before the upgrade, and everything is working this time around!
adam commented 2025-12-29 02:21:45 +01:00
Author
Owner
Copy Link

@yuphing-ong commented on GitHub (Mar 6, 2025):

I bumped into this as well. To resolve, install sqlite/sqlite3, run sqlite and delete the offending row:

sqlite /var/lib/headscale/db.sqlite
select * from nodes where disco_key='discokey:;

verify this node can be deleted. In my case, this was a node that was setup, but never connected, so it was fine for me to delete it, with:

delete from nodes where disco_key='discokey:;
.quit

@yuphing-ong commented on GitHub (Mar 6, 2025): I bumped into this as well. To resolve, install sqlite/sqlite3, run sqlite and delete the offending row: `sqlite /var/lib/headscale/db.sqlite` `select * from nodes where disco_key='discokey:`; verify this node can be deleted. In my case, this was a node that was setup, but never connected, so it was fine for me to delete it, with: `delete from nodes where disco_key='discokey:`; `.quit`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#661
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?