[Bug] Upgrade from 0.26.1 --> 0.27.0 makes all clients unable to connect to headscale #1141

New Issue

Closed

opened 2025-12-29 02:28:31 +01:00 by adam · 5 comments

adam commented 2025-12-29 02:28:31 +01:00

Owner

Copy Link

Originally created by @phxyz12 on GitHub (Nov 8, 2025).

Is this a support request?

• This is not a support request

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Dear all

I tried to upgrade Tailscale from version 0.26.1 to 0.27.0. I am using Docker containers and Docker Compose. I followed the upgrade steps found here: https://headscale.net/stable/setup/upgrade/. I compared the config files and merged the few changes in v0.27.0 into the existing config file. I took a backup of the 0.26.1 instance.

After starting v0.27.0, all clients were not able anymore to connect to headscale. In the logs, I saw lots of errors like this:

ERR noise upgrade failed error="noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed"

I didn't find a way to solve this other than deleting a client and registering it anew.

Expected Behavior

I would have wished, that clients stay ok, but I don't know if this was intended. Maybe migrating registered clients between 0.26 and 0.27 is not supported?

Steps To Reproduce

1. Install v0.26.1 using a Docker container and docker compose
2. Front Headscale with Traefik reverse proxy
3. Carry out upgrade procedure to 0.27.0 as described here: https://headscale.net/stable/setup/upgrade/
4. Start headscale
5. Try to connect with an existing (= already registered) client

Environment

- OS: Ubuntu 24.04
- Headscale version: 0.26.1 / 0.27.0
- Tailscale version: 1.90.6 and also 1.88.x
- Docker and Docker Compose (current versions for ubuntu)
- Traefik reverse proxy v3.5

Runtime environment

• Headscale is behind a (reverse) proxy
• Headscale runs in a container

Debug information

Headscale logs contain many lines like this:

ERR noise upgrade failed error="noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed"

Traefik labels (in compose file):

labels:
      - traefik.enable=true
      - traefik.http.services.headscale.loadbalancer.server.port=8080
      - traefik.http.services.headscale.loadbalancer.server.scheme=http
      - traefik.http.routers.headscale-https.entrypoints=websecure
      - traefik.http.routers.headscale-https.rule=Host(`headscale.domain.org`)
      - traefik.http.routers.headscale-https.tls=true
      - traefik.http.routers.headscale-https.tls.certresolver=lencrypt
      - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file  # geo block

Note: Headscale v0.26.1 worked without issues using the same Traefik config.

Originally created by @phxyz12 on GitHub (Nov 8, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Dear all I tried to upgrade Tailscale from version 0.26.1 to 0.27.0. I am using Docker containers and Docker Compose. I followed the upgrade steps found here: https://headscale.net/stable/setup/upgrade/. I compared the config files and merged the few changes in v0.27.0 into the existing config file. I took a backup of the 0.26.1 instance. After starting v0.27.0, all clients were not able anymore to connect to headscale. In the logs, I saw lots of errors like this: ```bash ERR noise upgrade failed error="noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed" ``` I didn't find a way to solve this other than deleting a client and registering it anew. ### Expected Behavior I would have wished, that clients stay ok, but I don't know if this was intended. Maybe migrating registered clients between 0.26 and 0.27 is not supported? ### Steps To Reproduce 1. Install v0.26.1 using a Docker container and docker compose 2. Front Headscale with Traefik reverse proxy 3. Carry out upgrade procedure to 0.27.0 as described here: https://headscale.net/stable/setup/upgrade/ 4. Start headscale 5. Try to connect with an existing (= already registered) client ### Environment ```markdown - OS: Ubuntu 24.04 - Headscale version: 0.26.1 / 0.27.0 - Tailscale version: 1.90.6 and also 1.88.x - Docker and Docker Compose (current versions for ubuntu) - Traefik reverse proxy v3.5 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information Headscale logs contain many lines like this: ```bash ERR noise upgrade failed error="noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed" ``` Traefik labels (in compose file): ```yaml labels: - traefik.enable=true - traefik.http.services.headscale.loadbalancer.server.port=8080 - traefik.http.services.headscale.loadbalancer.server.scheme=http - traefik.http.routers.headscale-https.entrypoints=websecure - traefik.http.routers.headscale-https.rule=Host(`headscale.domain.org`) - traefik.http.routers.headscale-https.tls=true - traefik.http.routers.headscale-https.tls.certresolver=lencrypt - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file # geo block ``` _Note: Headscale v0.26.1 worked without issues using the same Traefik config._

adam added the bug label 2025-12-29 02:28:31 +01:00

adam closed this issue 2025-12-29 02:28:31 +01:00

adam commented 2025-12-29 02:28:31 +01:00

Author

Owner

Copy Link

@nblock commented on GitHub (Nov 8, 2025):

I would have wished, that clients stay ok, but I don't know if this was intended. Maybe migrating registered clients between 0.26 and 0.27 is not supported?

That's sad to hear, upgrading from 0.26 to 0.27 should keep all clients connected unless they are too old. But that's likely not the problem here.

Could you please share your docker compose file?

@nblock commented on GitHub (Nov 8, 2025): > I would have wished, that clients stay ok, but I don't know if this was intended. Maybe migrating registered clients between 0.26 and 0.27 is not supported? That's sad to hear, upgrading from 0.26 to 0.27 should keep all clients connected unless they are too old. But that's likely not the problem here. Could you please share your docker compose file?

adam commented 2025-12-29 02:28:32 +01:00

Author

Owner

Copy Link

@phxyz12 commented on GitHub (Nov 8, 2025):

Hi,

here we go:

Docker compose file for v 0.26.1:

services:
  headscale:
    container_name: headscale
    image: headscale/headscale:0.26.1
    volumes:
      - ./config:/etc/headscale/  # Configuration files
      - ./data:/var/lib/headscale # Data persistence
    # ports:
    #  - "8282:8080" 
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
    command: serve
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.services.headscale.loadbalancer.server.port=8080
      - traefik.http.services.headscale.loadbalancer.server.scheme=http
      - traefik.http.routers.headscale-https.entrypoints=websecure
      - traefik.http.routers.headscale-https.rule=Host(`headscale.somedomain.org`)
      - traefik.http.routers.headscale-https.tls=true
      - traefik.http.routers.headscale-https.tls.certresolver=ionos
      - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file  # geo block
    networks:
      - frontend

networks:
  frontend:
    external: true

Docker compose file for v0.27.0:

services:
  headscale:
    container_name: headscale
    image: headscale/headscale:0.27.0
    volumes:
      - ./config:/etc/headscale/  # Configuration files
      - ./lib:/var/lib/headscale     # Data persistence
      - ./run:/var/run/headscale
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
    command: serve
    healthcheck:
        test: ["CMD", "headscale", "health"]
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.services.headscale.loadbalancer.server.port=8080
      - traefik.http.services.headscale.loadbalancer.server.scheme=http
      - traefik.http.routers.headscale-https.entrypoints=websecure
      - traefik.http.routers.headscale-https.rule=Host(`headscale.somedomain.org`)
      - traefik.http.routers.headscale-https.tls=true
      - traefik.http.routers.headscale-https.tls.certresolver=ionos
      - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file  # geo block
    networks:
      - frontend

networks:
  frontend:
    external: true

As you can see, there is not much difference. I added mounting of /var/run and a healthcheck section since I found these elements in the latest Docker compose example file.

A side note:

I use headscale (in both versions) with SQLite db as backend for the ACL configs. I changed to SQLite since I wanted to use the headplane (https://headplane.net/) frontend which only supports management of ACLs when SQLite is used.

@phxyz12 commented on GitHub (Nov 8, 2025): Hi, here we go: Docker compose file for v 0.26.1: ```yaml services: headscale: container_name: headscale image: headscale/headscale:0.26.1 volumes: - ./config:/etc/headscale/ # Configuration files - ./data:/var/lib/headscale # Data persistence # ports: # - "8282:8080" cap_add: - NET_ADMIN - SYS_MODULE sysctls: - net.ipv4.ip_forward=1 - net.ipv4.conf.all.src_valid_mark=1 command: serve restart: unless-stopped labels: - traefik.enable=true - traefik.http.services.headscale.loadbalancer.server.port=8080 - traefik.http.services.headscale.loadbalancer.server.scheme=http - traefik.http.routers.headscale-https.entrypoints=websecure - traefik.http.routers.headscale-https.rule=Host(`headscale.somedomain.org`) - traefik.http.routers.headscale-https.tls=true - traefik.http.routers.headscale-https.tls.certresolver=ionos - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file # geo block networks: - frontend networks: frontend: external: true ``` Docker compose file for v0.27.0: ```yaml services: headscale: container_name: headscale image: headscale/headscale:0.27.0 volumes: - ./config:/etc/headscale/ # Configuration files - ./lib:/var/lib/headscale # Data persistence - ./run:/var/run/headscale cap_add: - NET_ADMIN - SYS_MODULE sysctls: - net.ipv4.ip_forward=1 - net.ipv4.conf.all.src_valid_mark=1 command: serve healthcheck: test: ["CMD", "headscale", "health"] restart: unless-stopped labels: - traefik.enable=true - traefik.http.services.headscale.loadbalancer.server.port=8080 - traefik.http.services.headscale.loadbalancer.server.scheme=http - traefik.http.routers.headscale-https.entrypoints=websecure - traefik.http.routers.headscale-https.rule=Host(`headscale.somedomain.org`) - traefik.http.routers.headscale-https.tls=true - traefik.http.routers.headscale-https.tls.certresolver=ionos - traefik.http.routers.headscale-https.middlewares=my-traefik-plugin-geoblock@file # geo block networks: - frontend networks: frontend: external: true ``` As you can see, there is not much difference. I added mounting of `/var/run` and a healthcheck section since I found these elements in the latest Docker compose example file. A side note: I use headscale (in both versions) with SQLite db as backend for the ACL configs. I changed to SQLite since I wanted to use the headplane (https://headplane.net/) frontend which only supports management of ACLs when SQLite is used.

adam commented 2025-12-29 02:28:32 +01:00

Author

Owner

Copy Link

@nblock commented on GitHub (Nov 9, 2025):

The error noise upgrade failed: noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed occurs when the noise_private.key is changed.

The compose file for 0.26.1 contains:

    volumes:
      - ./config:/etc/headscale/  # Configuration files
      - ./data:/var/lib/headscale # Data persistence

while the compose file for 0.27.0 contains:

    volumes:
      - ./config:/etc/headscale/  # Configuration files
      - ./lib:/var/lib/headscale     # Data persistence
      - ./run:/var/run/headscale

Is it possible that you forgot to rename the local directory data to lib before starting 0.27? It looks like
Headscale did not find the noise_private.key at some point and created it anew causing the error.

@nblock commented on GitHub (Nov 9, 2025): The error `noise upgrade failed: noise handshake failed: decrypting machine key: chacha20poly1305: message authentication failed` occurs when the `noise_private.key` is changed. The compose file for 0.26.1 contains: ```yaml volumes: - ./config:/etc/headscale/ # Configuration files - ./data:/var/lib/headscale # Data persistence ``` while the compose file for 0.27.0 contains: ```yaml volumes: - ./config:/etc/headscale/ # Configuration files - ./lib:/var/lib/headscale # Data persistence - ./run:/var/run/headscale ``` Is it possible that you forgot to rename the local directory `data` to `lib` before starting 0.27? It looks like Headscale did not find the `noise_private.key` at some point and created it anew causing the error.

adam commented 2025-12-29 02:28:32 +01:00

Author

Owner

Copy Link

@phxyz12 commented on GitHub (Nov 9, 2025):

I checked directories, I did rename datato lib:

drwxrwxr-x 2 user user 4096 Nov  4 15:44 config/
-rw-rw-r-- 1 user user 2407 Nov  4 15:49 docker-compose.yml
drwxrwxr-x 2 user user 4096 Oct  7 13:21 headplane-config/
drwxrwxr-x 2 user user 4096 Sep 12 11:05 headplane-data/
drwxrwxr-x 2 user user 4096 Nov  4 15:50 lib/
drwxrwxr-x 2 user user 4096 Nov  4 15:50 run/

Inside lib, I have:

-rw-r--r-- 1 root   root   290816 Nov  4 15:50 db.sqlite
-rw------- 1 root   root       72 Sep 12 10:46 noise_private.key

I see one difference regarding the file db.sqlite regarding it's ownership:

v0.26.1:   -rw-r--r-- 1 user user  290816 Nov  8 21:45 db.sqlite
v0.27.0:   -rw-r--r-- 1 root   root   290816 Nov  4 15:50 db.sqlite

Ownership/permissions of the file noise_private.key are the same in both installations.

@phxyz12 commented on GitHub (Nov 9, 2025): I checked directories, I did rename `data`to `lib`: ```bash drwxrwxr-x 2 user user 4096 Nov 4 15:44 config/ -rw-rw-r-- 1 user user 2407 Nov 4 15:49 docker-compose.yml drwxrwxr-x 2 user user 4096 Oct 7 13:21 headplane-config/ drwxrwxr-x 2 user user 4096 Sep 12 11:05 headplane-data/ drwxrwxr-x 2 user user 4096 Nov 4 15:50 lib/ drwxrwxr-x 2 user user 4096 Nov 4 15:50 run/ ``` Inside `lib`, I have: ```bash -rw-r--r-- 1 root root 290816 Nov 4 15:50 db.sqlite -rw------- 1 root root 72 Sep 12 10:46 noise_private.key ``` I see one difference regarding the file `db.sqlite` regarding it's ownership: ```bash v0.26.1: -rw-r--r-- 1 user user 290816 Nov 8 21:45 db.sqlite v0.27.0: -rw-r--r-- 1 root root 290816 Nov 4 15:50 db.sqlite ``` Ownership/permissions of the file `noise_private.key` are the same in both installations.

adam commented 2025-12-29 02:28:32 +01:00

Author

Owner

Copy Link

@nblock commented on GitHub (Nov 12, 2025):

0.27.1 is now available and the update should preserve node connectivity, please give it a try.

@nblock commented on GitHub (Nov 12, 2025): 0.27.1 is now available and the update should preserve node connectivity, please give it a try.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

update_flake_lock_action

gh-pages

kradalby/release-v0.27.2

dependabot/go_modules/golang.org/x/crypto-0.45.0

dependabot/go_modules/github.com/opencontainers/runc-1.3.3

copilot/investigate-headscale-issue-2788

copilot/investigate-visibility-issue-2788

copilot/investigate-issue-2833

copilot/debug-issue-2846

copilot/fix-issue-2847

dependabot/go_modules/github.com/go-viper/mapstructure/v2-2.4.0

dependabot/go_modules/github.com/docker/docker-28.3.3incompatible

kradalby/cli-experiement3

doc/0.26.1

doc/0.25.1

doc/0.25.0

doc/0.24.3

doc/0.24.2

doc/0.24.1

doc/0.24.0

kradalby/build-docker-on-pr

topic/docu-versioning

topic/docker-kos

juanfont/fix-crash-node-id

juanfont/better-disclaimer

update-contributors

topic/prettier

revert-1893-add-test-stage-to-docs

add-test-stage-to-docs

remove-node-check-interval

fix-empty-prefix

fix-ephemeral-reusable

bug_report-debuginfo

autogroups

logs-to-stderr

revert-1414-topic/fix_unix_socket

rename-machine-node

port-embedded-derp-tests-v2

port-derp-tests

duplicate-word-linter

update-tailscale-1.36

warn-against-apache

ko-fi-link

more-acl-tests

fix-typo-standalone

parallel-nolint

tparallel-fix

rerouting

ssh-changelog-docs

oidc-cleanup

web-auth-flow-tests

kradalby-gh-runner

fix-proto-lint

remove-funding-links

go-1.19

enable-1.30-in-tests

0.16.x

cosmetic-changes-integration

tmp-fix-integration-docker

fix-integration-docker

configurable-update-interval

show-nodes-online

hs2021

acl-syntax-fixes

ts2021-implementation

fix-spurious-updates

unstable-integration-tests

mandatory-stun

embedded-derp

prtemplate-fix

v0.28.0-beta.1

v0.27.2-rc.1

v0.27.1

v0.27.0

v0.27.0-beta.2

v0.27.0-beta.1

v0.26.1

v0.26.0

v0.26.0-beta.2

v0.26.0-beta.1

v0.25.1

v0.25.0

v0.25.0-beta.2

v0.24.3

v0.25.0-beta.1

v0.24.2

v0.24.1

v0.24.0

v0.24.0-beta.2

v0.24.0-beta.1

v0.23.0

v0.23.0-rc.1

v0.23.0-beta.5

v0.23.0-beta.4

v0.23.0-beta3

v0.23.0-beta2

v0.23.0-beta1

v0.23.0-alpha12

v0.23.0-alpha11

v0.23.0-alpha10

v0.23.0-alpha9

v0.23.0-alpha8

v0.23.0-alpha7

v0.23.0-alpha6

v0.23.0-alpha5

v0.23.0-alpha4

v0.23.0-alpha4-docker-ko-test9

v0.23.0-alpha4-docker-ko-test8

v0.23.0-alpha4-docker-ko-test7

v0.23.0-alpha4-docker-ko-test6

v0.23.0-alpha4-docker-ko-test5

v0.23.0-alpha-docker-release-test-debug2

v0.23.0-alpha-docker-release-test-debug

v0.23.0-alpha4-docker-ko-test4

v0.23.0-alpha4-docker-ko-test3

v0.23.0-alpha4-docker-ko-test2

v0.23.0-alpha4-docker-ko-test

v0.23.0-alpha3

v0.23.0-alpha2

v0.23.0-alpha1

v0.22.3

v0.22.2

v0.23.0-alpha-docker-release-test

v0.22.1

v0.22.0

v0.22.0-alpha3

v0.22.0-alpha2

v0.22.0-alpha1

v0.22.0-nfpmtest

v0.21.0

v0.20.0

v0.19.0

v0.19.0-beta2

v0.19.0-beta1

v0.18.0

v0.18.0-beta4

v0.18.0-beta3

v0.18.0-beta2

v0.18.0-beta1

v0.17.1

v0.17.0

v0.17.0-beta5

v0.17.0-beta4

v0.17.0-beta3

v0.17.0-beta2

v0.17.0-beta1

v0.17.0-alpha4

v0.17.0-alpha3

v0.17.0-alpha2

v0.17.0-alpha1

v0.16.4

v0.16.3

v0.16.2

v0.16.1

v0.16.0

v0.16.0-beta7

v0.16.0-beta6

v0.16.0-beta5

v0.16.0-beta4

v0.16.0-beta3

v0.16.0-beta2

v0.16.0-beta1

v0.15.0

v0.15.0-beta6

v0.15.0-beta5

v0.15.0-beta4

v0.15.0-beta3

v0.15.0-beta2

v0.15.0-beta1

v0.14.0

v0.14.0-beta2

v0.14.0-beta1

v0.13.0

v0.13.0-beta3

v0.13.0-beta2

v0.13.0-beta1

upstream/v0.12.4

v0.12.4

v0.12.3

v0.12.2

v0.12.2-beta1

v0.12.1

v0.12.0-beta2

v0.12.0-beta1

v0.11.0

v0.10.8

v0.10.7

v0.10.6

v0.10.5

v0.10.4

v0.10.3

v0.10.2

v0.10.1

v0.10.0

v0.9.3

v0.9.2

v0.9.1

v0.9.0

v0.8.1

v0.8.0

v0.7.1

v0.7.0

v0.6.1

v0.6.0

v0.5.2

v0.5.1

v0.5.0

v0.4.0

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.2

v0.2.1

v0.2.0

v0.1.1

v0.1.0

Labels

Clear labels

CLI

DERP

DNS

Nix

OIDC

SSH

bug

database

documentation

duplicate

enhancement

faq

good first issue

grants

help wanted

might-come

needs design doc

needs investigation

no-stale-bot

out of scope

performance

policy 📝

pull-request

Mirrored from GitHub Pull Request

question

regression

routes

stale

tags

tailscale-feature-gap

well described ❤️

wontfix

No Label bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

adam

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/headscale#1141