starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] subnet router failover not working properly when the subnet router node is not there at client authentication time ? #847

New Issue
Closed
opened 2025-12-29 02:24:48 +01:00 by adam · 2 comments
adam commented 2025-12-29 02:24:48 +01:00
Owner
Copy Link

Originally created by @codingtony-candid on GitHub (Nov 6, 2024).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

If a router node is not online when the tailscale client logs in, the client does not route the traffic to that router node in case of failover. In that situation, when the node that is advertising routes goes offline, its advertised routes are still flagged as primary.

Expected Behavior

If the primary subnet router node disappear, the failover should kick-in immediately, even if the failover node came online after the client authenticated with headscale.

Steps To Reproduce

Topology

We have one headscale server and 2 router nodes both advertising the same routes.
Our router nodes are subnets routers, we do not route internet traffic through them.
We have a tailscale client running on MacOS that need to access an IP routed by the route nodes.

Case 1 : router node comes online after tailscale client

From a tailscale client (macOS, 1.76.1 standalone variant) there is a ping running to a routed IP : 192.168.168.1

Nodes:

39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 15:29:04 | 0001-01-01 00:00:00 | online    | no

Routes:

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | true

We start a 2nd router node (router-node-i-065a1cf0840b26f0d)

Nodes

39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 15:29:04 | 0001-01-01 00:00:00 | online    | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online    | no

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | false
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | true

From the macos tailscale client we see the 2 router-nodes
image

We stop router-node-i-0380d27217348885b (which has the primary route)
On the MacOs tailscale client the ping stop working

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:24:49 | 0001-01-01 00:00:00 | offline   | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online    | no

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | false
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | true

router-node-i-01cd8b42c5599852d is still advertising as primary

The macos client still show 2 node
image

We disconnect and reconnect the macos client and the ping comes back

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:24:49 | 0001-01-01 00:00:00 | offline   | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online    | no

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | false
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | true

What if the two nodes are up when the MacOS tailscale clients logs in

Ping is continuously running on the MacOS tailscale client

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:27:43 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:36:32 | 0001-01-01 00:00:00 | online    | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:28:54 | 0001-01-01 00:00:00 | online    | no

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | true
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | false

We stop the primary : router-node-i-01cd8b42c5599852d

The ping only misses one beat :

64 bytes from 192.168.168.1: icmp_seq=1708 ttl=64 time=26.200 ms
64 bytes from 192.168.168.1: icmp_seq=1709 ttl=64 time=26.138 ms
64 bytes from 192.168.168.1: icmp_seq=1710 ttl=64 time=29.193 ms
Request timeout for icmp_seq 1711
64 bytes from 192.168.168.1: icmp_seq=1712 ttl=64 time=94.161 ms

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:50:47 | 0001-01-01 00:00:00 | online    | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:52:18 | 0001-01-01 00:00:00 | offline   | no

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | false
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | true

The route fails over correctly (primary is now router-node-i-0380d27217348885b ) !

re-starting router-node-i-01cd8b42c5599852d

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:50:47 | 0001-01-01 00:00:00 | online    | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:55:01 | 0001-01-01 00:00:00 | online    | no

(no effect on the ping)

Routes

ID  | Node                            | Prefix           | Advertised | Enabled | Primary
447 | router-node-i-0380d27217348885b | 0.0.0.0/0        | true       | false   | -
448 | router-node-i-0380d27217348885b | ::/0             | true       | false   | -
446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true       | true    | true
452 | router-node-i-01cd8b42c5599852d | ::/0             | true       | false   | -
454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0        | true       | false   | -
453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true       | true    | false

stop both router-nodes

Nodes

ID | Hostname                        | Name                            | MachineKey | NodeKey | User           | IP addresses                  | Ephemeral | Last seen           | Expiration          | Connected | Expired
39 | userM1Pro                 | userm1pro                 | [IytP7]    | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false     | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online    | no
63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak]    | [T6Q47] | router-node    | 100.64.0.4, fd7a:115c:a1e0::4 | false     | 2024-11-06 16:58:26 | 0001-01-01 00:00:00 | offline   | no
65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN]    | [Nu1nL] | router-node    | 100.64.0.8, fd7a:115c:a1e0::8 | false     | 2024-11-06 16:58:36 | 0001-01-01 00:00:00 | offline   | no

As soon as we start router-node-i-01cd8b42c5599852d
ping recover as expected.

Environment

- OS: AmazonLinux2023 for Headscale and tailscale router nodes, MacOS Sonoma 14.6.1 for the tailscale client (standalone variant) 
- Headscale version: 0.23
- Tailscale version: 1.76.1

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

Other details for the setup

ACLs used :

{
  "acls": [
    {
      "action": "accept",
      "src": ["*"],
      "dst": [
                "192.168.168.1/32:*"
      ]
    }
  ],
  "tagOwners": {
    "tag:application-router-node": [
      "router-node"
    ]
  },
  "autoApprovers": {
    "routes": {
            "192.168.168.1/32": ["tag:application-router-node"]
    },
    "exitNode": []
  }
}

router-node tailscale parameters :

--login-server=https://********** --advertise-exit-node --advertise-routes=192.168.168.1/32 --accept-dns=true --advertise-tags=tag:application-router-node
Originally created by @codingtony-candid on GitHub (Nov 6, 2024). ### Is this a support request? - [X] This is not a support request ### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior If a router node is not online when the tailscale client logs in, the client does not route the traffic to that router node in case of failover. In that situation, when the node that is advertising routes goes offline, its advertised routes are still flagged as primary. ### Expected Behavior If the primary subnet router node disappear, the failover should kick-in immediately, even if the failover node came online after the client authenticated with headscale. ### Steps To Reproduce # Topology We have one headscale server and 2 router nodes both advertising the same routes. Our router nodes are subnets routers, we do not route internet traffic through them. We have a tailscale client running on MacOS that need to access an IP routed by the route nodes. # Case 1 : router node comes online after tailscale client From a tailscale client (macOS, 1.76.1 standalone variant) there is a ping running to a routed IP : 192.168.168.1 ## Nodes: ``` 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 15:29:04 | 0001-01-01 00:00:00 | online | no ``` ## Routes: ``` ID | Node | Prefix | Advertised | Enabled | Primary 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | true ``` We start a 2nd router node (router-node-i-065a1cf0840b26f0d) ## Nodes ``` 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 15:29:04 | 0001-01-01 00:00:00 | online | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online | no ``` ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | false 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | true ``` From the macos tailscale client we see the 2 router-nodes ![image](https://github.com/user-attachments/assets/3d1b1023-26f0-4d31-bb3f-0aa02e068d7b) We stop `router-node-i-0380d27217348885b` (which has the primary route) On the MacOs tailscale client the ping stop working ## Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:24:49 | 0001-01-01 00:00:00 | offline | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online | no ``` ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | false 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | true ``` **router-node-i-01cd8b42c5599852d is still advertising as primary** The macos client still show 2 node ![image](https://github.com/user-attachments/assets/4080ab74-c590-4798-9090-19d2db89b11b) We disconnect and reconnect the macos client and the ping comes back ## Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:20:19 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:24:49 | 0001-01-01 00:00:00 | offline | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:22:34 | 0001-01-01 00:00:00 | online | no ``` ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | false 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | true ``` # What if the two nodes are up when the MacOS tailscale clients logs in Ping is continuously running on the MacOS tailscale client ## Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:27:43 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:36:32 | 0001-01-01 00:00:00 | online | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:28:54 | 0001-01-01 00:00:00 | online | no ``` ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | true 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | false ``` We stop the primary : `router-node-i-01cd8b42c5599852d` The ping only misses one beat : ``` 64 bytes from 192.168.168.1: icmp_seq=1708 ttl=64 time=26.200 ms 64 bytes from 192.168.168.1: icmp_seq=1709 ttl=64 time=26.138 ms 64 bytes from 192.168.168.1: icmp_seq=1710 ttl=64 time=29.193 ms Request timeout for icmp_seq 1711 64 bytes from 192.168.168.1: icmp_seq=1712 ttl=64 time=94.161 ms ``` ## Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:50:47 | 0001-01-01 00:00:00 | online | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:52:18 | 0001-01-01 00:00:00 | offline | no ``` ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | false 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | true ``` The route fails over correctly (primary is now `router-node-i-0380d27217348885b` ) ! re-starting `router-node-i-01cd8b42c5599852d ` ## Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:50:47 | 0001-01-01 00:00:00 | online | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:55:01 | 0001-01-01 00:00:00 | online | no ``` (no effect on the ping) ## Routes ``` ID | Node | Prefix | Advertised | Enabled | Primary 447 | router-node-i-0380d27217348885b | 0.0.0.0/0 | true | false | - 448 | router-node-i-0380d27217348885b | ::/0 | true | false | - 446 | router-node-i-0380d27217348885b | 192.168.168.1/32 | true | true | true 452 | router-node-i-01cd8b42c5599852d | ::/0 | true | false | - 454 | router-node-i-01cd8b42c5599852d | 0.0.0.0/0 | true | false | - 453 | router-node-i-01cd8b42c5599852d | 192.168.168.1/32 | true | true | false ``` stop both router-nodes # Nodes ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 39 | userM1Pro | userm1pro | [IytP7] | [uiEpB] | user | 100.64.0.7, fd7a:115c:a1e0::7 | false | 2024-11-06 16:42:35 | 2024-11-12 18:56:13 | online | no 63 | router-node-i-0380d27217348885b | router-node-i-0380d27217348885b | [CnYak] | [T6Q47] | router-node | 100.64.0.4, fd7a:115c:a1e0::4 | false | 2024-11-06 16:58:26 | 0001-01-01 00:00:00 | offline | no 65 | router-node-i-01cd8b42c5599852d | router-node-i-01cd8b42c5599852d | [F3wqN] | [Nu1nL] | router-node | 100.64.0.8, fd7a:115c:a1e0::8 | false | 2024-11-06 16:58:36 | 0001-01-01 00:00:00 | offline | no ``` As soon as we start `router-node-i-01cd8b42c5599852d ` ping recover as expected. ### Environment ```markdown - OS: AmazonLinux2023 for Headscale and tailscale router nodes, MacOS Sonoma 14.6.1 for the tailscale client (standalone variant) - Headscale version: 0.23 - Tailscale version: 1.76.1 ``` ### Runtime environment - [ ] Headscale is behind a (reverse) proxy - [X] Headscale runs in a container ### Anything else? ### Other details for the setup ACLs used : ``` { "acls": [ { "action": "accept", "src": ["*"], "dst": [ "192.168.168.1/32:*" ] } ], "tagOwners": { "tag:application-router-node": [ "router-node" ] }, "autoApprovers": { "routes": { "192.168.168.1/32": ["tag:application-router-node"] }, "exitNode": [] } } ``` router-node tailscale parameters : ``` --login-server=https://********** --advertise-exit-node --advertise-routes=192.168.168.1/32 --accept-dns=true --advertise-tags=tag:application-router-node ```
adam added the bugno-stale-bot labels 2025-12-29 02:24:48 +01:00
adam closed this issue 2025-12-29 02:24:48 +01:00
adam commented 2025-12-29 02:24:48 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Aug 3, 2025):

The routes code changed significantly with Headscale 0.26 and I can no longer reproduce your issue with a slightly modified setup:

  • Headscale 0.26.1
  • 2 Subnetrouters (Tailscale 1.86.2, Debian 12)
  • 1 Client (Tailscale 1.86.2, Debian 12, no MacOS available for testing)

Continuous pings and a http check every second work without any interruption, error or restarts of the client node. The startup order of the three Tailscale clients involved does not matter.

Is this still an issue with Headscale 0.26 in your environment?

@nblock commented on GitHub (Aug 3, 2025): The routes code changed significantly with Headscale 0.26 and I can no longer reproduce your issue with a slightly modified setup: * Headscale 0.26.1 * 2 Subnetrouters (Tailscale 1.86.2, Debian 12) * 1 Client (Tailscale 1.86.2, Debian 12, no MacOS available for testing) Continuous pings and a http check every second work without any interruption, error or restarts of the client node. The startup order of the three Tailscale clients involved does not matter. Is this still an issue with Headscale 0.26 in your environment?
adam commented 2025-12-29 02:24:48 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Aug 19, 2025):

Closing as this should be fixed with Headscale 0.26.0 and missing feedback.

@nblock commented on GitHub (Aug 19, 2025): Closing as this should be fixed with Headscale 0.26.0 and missing feedback.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug no-stale-bot
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#847
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?