starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] Version 0.25.0 breaks exit node routes #947

New Issue
Closed
opened 2025-12-29 02:26:33 +01:00 by adam · 12 comments
adam commented 2025-12-29 02:26:33 +01:00
Owner
Copy Link

Originally created by @jdewinne on GitHub (Feb 17, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

In Headscale v0.24.3 when adding an exit node, two routes are added that can be enabled

tailscale up --timeout 60s --login-server ****--authkey**** --advertise-exit-node;

> headscale routes list
ID | Node     | Prefix    | Advertised | Enabled | Primary
1  | 17ac3ab5 | 0.0.0.0/0 | true       | true    | -
2  | 17ac3ab5 | ::/0      | true       | true    | -

Starting headscale v0.25.0, the output from headscale routes list is empty.

Expected Behavior

I would expect v0.25.0 also to contain routes, when an exit node joins the tailnet.

Steps To Reproduce

  1. Use Headscale 0.25.0
  2. Run tailscale up --timeout 60s --login-server ****--authkey**** --advertise-exit-node
  3. Run headscale routes list and no routes will show.

Environment

- OS: Ubuntu 20.04
- Headscale version: 0.25.0
- Tailscale version: 1.80.2

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

No response

Originally created by @jdewinne on GitHub (Feb 17, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior In Headscale v0.24.3 when adding an exit node, two routes are added that can be enabled ``` tailscale up --timeout 60s --login-server ****--authkey**** --advertise-exit-node; ``` ``` > headscale routes list ID | Node | Prefix | Advertised | Enabled | Primary 1 | 17ac3ab5 | 0.0.0.0/0 | true | true | - 2 | 17ac3ab5 | ::/0 | true | true | - ``` Starting headscale v0.25.0, the output from `headscale routes list` is empty. ### Expected Behavior I would expect v0.25.0 also to contain routes, when an exit node joins the tailnet. ### Steps To Reproduce 1. Use Headscale 0.25.0 2. Run `tailscale up --timeout 60s --login-server ****--authkey**** --advertise-exit-node` 3. Run ` headscale routes list` and no routes will show. ### Environment ```markdown - OS: Ubuntu 20.04 - Headscale version: 0.25.0 - Tailscale version: 1.80.2 ``` ### Runtime environment - [ ] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Anything else? _No response_
adam added the bug label 2025-12-29 02:26:33 +01:00
adam closed this issue 2025-12-29 02:26:33 +01:00
adam commented 2025-12-29 02:26:34 +01:00
Author
Owner
Copy Link

@rfrank commented on GitHub (Feb 18, 2025):

This is happening to me as well. Running host install behind a reverse proxy.

Edit: confirmed that downgrading to v0.24.3 fixes the issue.

@rfrank commented on GitHub (Feb 18, 2025): This is happening to me as well. Running host install behind a reverse proxy. Edit: confirmed that downgrading to v0.24.3 fixes the issue.
adam commented 2025-12-29 02:26:34 +01:00
Author
Owner
Copy Link

@anna-oake commented on GitHub (Feb 18, 2025):

Same issue, and it applies to any advertised routes, not just exit node related.
I confirm this is fixed by downgrading to v0.24.3, deleting the node and enroling it again.

@anna-oake commented on GitHub (Feb 18, 2025): Same issue, and it applies to any advertised routes, not just exit node related. I confirm this is fixed by downgrading to v0.24.3, deleting the node and enroling it again.
adam commented 2025-12-29 02:26:35 +01:00
Author
Owner
Copy Link

@alfs commented on GitHub (Feb 20, 2025):

Same issue, same solution as rfrank and anna-oake.

@alfs commented on GitHub (Feb 20, 2025): Same issue, same solution as rfrank and anna-oake.
adam commented 2025-12-29 02:26:35 +01:00
Author
Owner
Copy Link

@Solanum95 commented on GitHub (Feb 21, 2025):

Same issue,
First time I've used headscale and could´t get exit-node to work, luckily i found this issue and installed v0.24.3 instead and now it works.

@Solanum95 commented on GitHub (Feb 21, 2025): Same issue, First time I've used headscale and could´t get exit-node to work, luckily i found this issue and installed v0.24.3 instead and now it works.
adam commented 2025-12-29 02:26:35 +01:00
Author
Owner
Copy Link

@Berjou commented on GitHub (Feb 22, 2025):

Just to add some info, I had this issue too on a fresh install of headscale v0.25.0 with docker and caddy. So the problem was not caused by an upgrade from a previous version. Like the others have already commented, a downgrade to v0.24.3 also solved it for me.

@Berjou commented on GitHub (Feb 22, 2025): Just to add some info, I had this issue too on a **fresh** install of headscale v0.25.0 with docker and caddy. So the problem was not caused by an upgrade from a previous version. Like the others have already commented, a downgrade to v0.24.3 also solved it for me.
adam commented 2025-12-29 02:26:35 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Feb 23, 2025):

Hmm, this is a bit of a puzzle, we have tests covering advertising nodes and they are all passing, I've added a specific one for exit nodes and that is too passing https://github.com/juanfont/headscale/pull/2444.

I am unable to replicate this, anything you can see in that test case that is different from your setup?

@kradalby commented on GitHub (Feb 23, 2025): Hmm, this is a bit of a puzzle, we have tests covering advertising nodes and they are all passing, I've added a specific one for exit nodes and that is too passing https://github.com/juanfont/headscale/pull/2444. I am unable to replicate this, anything you can see in that test case that is different from your setup?
adam commented 2025-12-29 02:26:35 +01:00
Author
Owner
Copy Link

@anna-oake commented on GitHub (Feb 23, 2025):

@kradalby In my case I use the Linux tailscale client and I tried --advertise-exit-node both in set and up.
I tried doing down and then up, tried completely deleting the node from headscale and adding it again after resetting the client.

In the end I never see any routes in headscale routes, but if I run tailscale debug prefs on the client, I can see them in AdvertiseRoutes.

I use OIDC and always enrol the nodes via browser (haven't tested with pre-auth keys).

Only downgrading to v0.24.3, completely removing the node and adding it again made the routes show up in headscale routes.

  • I don't know if using pre-auth keys would yield a different result
  • I don't know if enroling the node on v0.24.3, then upgrading to v0.25.0, and then enabling --advertise-exit-node would yield a different result. Not sure if the tests you showed enrol clients from scratch or if it's a preconfigured environment
@anna-oake commented on GitHub (Feb 23, 2025): @kradalby In my case I use the Linux tailscale client and I tried `--advertise-exit-node` both in `set` and `up`. I tried doing `down` and then `up`, tried completely deleting the node from headscale and adding it again after resetting the client. In the end I never see any routes in `headscale routes`, but if I run `tailscale debug prefs` on the client, I can see them in `AdvertiseRoutes`. I use OIDC and always enrol the nodes via browser (haven't tested with pre-auth keys). Only downgrading to `v0.24.3`, completely removing the node and adding it again made the routes show up in `headscale routes`. - I don't know if using pre-auth keys would yield a different result - I don't know if enroling the node on `v0.24.3`, then upgrading to `v0.25.0`, and then enabling `--advertise-exit-node` would yield a different result. Not sure if the tests you showed enrol clients from scratch or if it's a preconfigured environment
adam commented 2025-12-29 02:26:36 +01:00
Author
Owner
Copy Link

@s1341 commented on GitHub (Feb 23, 2025):

I’ve tested with preauth keys and have the same behavior.

@s1341 commented on GitHub (Feb 23, 2025): I’ve tested with preauth keys and have the same behavior.
adam commented 2025-12-29 02:26:36 +01:00
Author
Owner
Copy Link

@anna-oake commented on GitHub (Feb 23, 2025):

Skimmed through the integration tests code - it seems that new tailscale clients/nodes/users are set up for the test, so that should be okay.

The only difference I see at first glance is that my user is an OIDC user.

@anna-oake commented on GitHub (Feb 23, 2025): Skimmed through the integration tests code - it seems that new tailscale clients/nodes/users are set up for the test, so that should be okay. The only difference I see at first glance is that my user is an OIDC user.
adam commented 2025-12-29 02:26:36 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Feb 23, 2025):

The only difference I see at first glance is that my user is an OIDC user.

This also happens on non-OIDC setups with preauthkeys

@nblock commented on GitHub (Feb 23, 2025): > The only difference I see at first glance is that my user is an OIDC user. This also happens on non-OIDC setups with preauthkeys
adam commented 2025-12-29 02:26:36 +01:00
Author
Owner
Copy Link

@Berjou commented on GitHub (Feb 23, 2025):

If it helps you to replicate the issue, this is the docker compose setup I'm using to register my exit node (working on v0.24.3).

compose.yaml:

name: exit-node

networks:
  exit-node:
    name: exit-node
    enable_ipv6: true

services:
  exit-node:
    image: tailscale/tailscale:v1.80.0
    container_name: exit-node
    hostname: ${SERVICE}
    environment:
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_EXTRA_ARGS=${TS_EXTRA_ARGS}
      - TS_TAILSCALED_EXTRA_ARGS=--no-logs-no-support
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=false
      - TS_ENABLE_HEALTH_CHECK=true
      - TS_ACCEPT_DNS=true
      - TS_DEBUG_FIREWALL_MODE=auto
    volumes:
      - /var/data/exit-node/ts/state:/var/lib/tailscale
    devices:
      - /dev/net/tun:/dev/net/tun
    sysctls:
      net.ipv4.ip_forward: 1
      net.ipv6.conf.all.forwarding: 1
    cap_add:
      - net_admin
    networks:
      - exit-node
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:9002/healthz"]
      interval: 1m
      timeout: 10s
      retries: 3
      start_period: 10s
    logging:
      driver: syslog
      options:
        tag: docker-exit-node
    restart: always

.env:

SERVICE=exit-node-1

TS_AUTHKEY=<my_preauthkey>

TS_EXTRA_ARGS="--login-server=https://<my_headscale_domain> --advertise-exit-node"

This compose is deployed on a standard Debian 12 VPS.

The preauthkey was generated like that on my headscale server:

docker exec -it headscale headscale preauthkeys create --user <admin_user> --tags tag:exit-node

I do not know if this is relevant, but I also have basic ACLs allowing my nodes to communicate with a pihole node configured as DNS nameservers in headscale and my users to access autogroup:internet:* for the exit node to work.

Also, my users were manually created, no OIDC setup.

@Berjou commented on GitHub (Feb 23, 2025): If it helps you to replicate the issue, this is the docker compose setup I'm using to register my exit node (working on `v0.24.3`). compose.yaml: ``` name: exit-node networks: exit-node: name: exit-node enable_ipv6: true services: exit-node: image: tailscale/tailscale:v1.80.0 container_name: exit-node hostname: ${SERVICE} environment: - TS_AUTHKEY=${TS_AUTHKEY} - TS_EXTRA_ARGS=${TS_EXTRA_ARGS} - TS_TAILSCALED_EXTRA_ARGS=--no-logs-no-support - TS_STATE_DIR=/var/lib/tailscale - TS_USERSPACE=false - TS_ENABLE_HEALTH_CHECK=true - TS_ACCEPT_DNS=true - TS_DEBUG_FIREWALL_MODE=auto volumes: - /var/data/exit-node/ts/state:/var/lib/tailscale devices: - /dev/net/tun:/dev/net/tun sysctls: net.ipv4.ip_forward: 1 net.ipv6.conf.all.forwarding: 1 cap_add: - net_admin networks: - exit-node healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:9002/healthz"] interval: 1m timeout: 10s retries: 3 start_period: 10s logging: driver: syslog options: tag: docker-exit-node restart: always ``` .env: ``` SERVICE=exit-node-1 TS_AUTHKEY=<my_preauthkey> TS_EXTRA_ARGS="--login-server=https://<my_headscale_domain> --advertise-exit-node" ``` This compose is deployed on a standard `Debian 12` VPS. The preauthkey was generated like that on my `headscale` server: `docker exec -it headscale headscale preauthkeys create --user <admin_user> --tags tag:exit-node` I do not know if this is relevant, but I also have basic ACLs allowing my nodes to communicate with a `pihole` node configured as DNS nameservers in `headscale` and my users to access `autogroup:internet:*` for the exit node to work. Also, my users were manually created, no OIDC setup.
adam commented 2025-12-29 02:26:36 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Feb 23, 2025):

In my case I use the Linux tailscale client and I tried --advertise-exit-node both in set and up.

I think I found it, so the tests did an up with no parameters, then used set and it turned out that there was a missing save in the registration path.

This means that setting routes with up was broken, but set should be fine, but probably only if there is no routes.

Can you try to add a node without routes and set them?

I am updating the PR with the fix and a test covering both up and set.

@kradalby commented on GitHub (Feb 23, 2025): > In my case I use the Linux tailscale client and I tried --advertise-exit-node both in set and up. I think I found it, so the tests did an `up` with no parameters, then used `set` and it turned out that there was a missing save in the registration path. This means that setting routes with `up` was broken, but `set` should be fine, but probably only if there is no routes. Can you try to add a node without routes and set them? I am updating the PR with the fix and a test covering both `up` and `set`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#947
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?