starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-13 12:50:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] key is delivered to tailscale node for a long time (6-7 minutes) after re-authentication, traffic doesn't pass in this time #1057

New Issue
Closed
opened 2025-12-29 02:28:00 +01:00 by adam · 6 comments
adam commented 2025-12-29 02:28:00 +01:00
Owner
Copy Link

Originally created by @spymobilfon on GitHub (Jul 8, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

  1. User re-authenticated in Tailscale (in the client side)
  2. I detected the registration in the Headscale log
2025-07-08T07:46:52.934674+00:00 headscale-xxx headscale[648615]: 2025-07-08T07:46:52Z INF github.com/juanfont/headscale/hscontrol/db/node.go:454 > Registering node machine_key=[bWpJu] node=xxxxx-50HP node_key=[Unuw8] vpn=true user=xxxxx@example.org
  1. Traffic didn't pass via Tailscale node which responsible for routing traffic
  2. I checked the Tailscale (router) log
Jul 08 07:46:55 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:46:56 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:02 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:08 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:14 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:19 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:24 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:29 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:34 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:39 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: wgengine: idle peer [Unuw8] now active, reconfiguring WireGuard
Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: magicsock: disco: node [Unuw8] d:7ea3b2e69a300b60 now using 1.2.3.4:38913 mtu=1360 tx=d8eb29444f6e

6-7 minutes to accept new key and pass traffic are excessively long.

Expected Behavior

Maximum 1 minute to accept new key and pass traffic.

Steps To Reproduce

  1. Re-authenticate in Tailscale (in the client side)
  2. Open resource which route via the Tailscale router node
  3. Check the Headscale log
  4. Check the Tailscale log

Environment

Headscale:
- OS: Ubuntu 24.04
- Headscale version: 0.26.1
Tailscale router:
- OS: Ubuntu 22.04
- Tailscale version: 1.82.5
Tailscale client:
- OS: macOS Sequoia 15.5
- Tailscale version: 1.84.1

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

Headscale log

2025-07-08T07:46:52.934674+00:00 headscale-xxx headscale[648615]: 2025-07-08T07:46:52Z INF github.com/juanfont/headscale/hscontrol/db/node.go:454 > Registering node machine_key=[bWpJu] node=xxxxx-50HP node_key=[Unuw8] vpn=true user=xxxxx@example.org

Tailscale log

Jul 08 07:46:55 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:46:56 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:02 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:08 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:14 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:19 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:24 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:29 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:34 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:47:39 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown
Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: wgengine: idle peer [Unuw8] now active, reconfiguring WireGuard
Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: magicsock: disco: node [Unuw8] d:7ea3b2e69a300b60 now using 1.2.3.4:38913 mtu=1360 tx=d8eb29444f6e
Originally created by @spymobilfon on GitHub (Jul 8, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior 1. User re-authenticated in Tailscale (in the client side) 2. I detected the registration in the Headscale log ``` 2025-07-08T07:46:52.934674+00:00 headscale-xxx headscale[648615]: 2025-07-08T07:46:52Z INF github.com/juanfont/headscale/hscontrol/db/node.go:454 > Registering node machine_key=[bWpJu] node=xxxxx-50HP node_key=[Unuw8] vpn=true user=xxxxx@example.org ``` 3. Traffic didn't pass via Tailscale node which responsible for routing traffic 4. I checked the Tailscale (router) log ``` Jul 08 07:46:55 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:46:56 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:02 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:08 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:14 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:19 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:24 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:29 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:34 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:39 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: wgengine: idle peer [Unuw8] now active, reconfiguring WireGuard Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: magicsock: disco: node [Unuw8] d:7ea3b2e69a300b60 now using 1.2.3.4:38913 mtu=1360 tx=d8eb29444f6e ``` 6-7 minutes to accept new key and pass traffic are excessively long. ### Expected Behavior Maximum 1 minute to accept new key and pass traffic. ### Steps To Reproduce 1. Re-authenticate in Tailscale (in the client side) 2. Open resource which route via the Tailscale router node 3. Check the Headscale log 4. Check the Tailscale log ### Environment ```markdown Headscale: - OS: Ubuntu 24.04 - Headscale version: 0.26.1 Tailscale router: - OS: Ubuntu 22.04 - Tailscale version: 1.82.5 Tailscale client: - OS: macOS Sequoia 15.5 - Tailscale version: 1.84.1 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Debug information Headscale log ``` 2025-07-08T07:46:52.934674+00:00 headscale-xxx headscale[648615]: 2025-07-08T07:46:52Z INF github.com/juanfont/headscale/hscontrol/db/node.go:454 > Registering node machine_key=[bWpJu] node=xxxxx-50HP node_key=[Unuw8] vpn=true user=xxxxx@example.org ``` Tailscale log ``` Jul 08 07:46:55 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:46:56 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:02 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:08 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:14 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:19 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:24 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:29 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:34 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:47:39 tailscale-xxx tailscaled[445054]: magicsock: disco: ignoring CallMeMaybe from d:77e58033721ad55b; [Unuw8] is unknown Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: wgengine: idle peer [Unuw8] now active, reconfiguring WireGuard Jul 08 07:53:05 tailscale-xxx tailscaled[445054]: magicsock: disco: node [Unuw8] d:7ea3b2e69a300b60 now using 1.2.3.4:38913 mtu=1360 tx=d8eb29444f6e ```
adam added the bug label 2025-12-29 02:28:00 +01:00
adam closed this issue 2025-12-29 02:28:00 +01:00
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@spymobilfon commented on GitHub (Jul 8, 2025):

I detected one more case. Some our clients passed re-authentication process but their keys didn't delivery to the Tailscale nodes which responsible for routing traffic. Helps just reboot of the Tailscale node((

@spymobilfon commented on GitHub (Jul 8, 2025): I detected one more case. Some our clients passed re-authentication process but their keys didn't delivery to the Tailscale nodes which responsible for routing traffic. Helps just reboot of the Tailscale node((
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@qwerty1q2w commented on GitHub (Jul 9, 2025):

the same case

@qwerty1q2w commented on GitHub (Jul 9, 2025): the same case
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 9, 2025):

How many nodes do you have? I suspect there are changes in main that would improve this.

@kradalby commented on GitHub (Sep 9, 2025): How many nodes do you have? I suspect there are changes in `main` that would improve this.
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@spymobilfon commented on GitHub (Sep 9, 2025):

I have 13 nodes which route traffic and ~1000 clients (users nodes)

@spymobilfon commented on GitHub (Sep 9, 2025): I have 13 nodes which route traffic and ~1000 clients (users nodes)
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 9, 2025):

That is way more than we expect Headscale to really handle: https://headscale.net/stable/about/faq/#scaling-how-many-clients-does-headscale-support

The new main release might be a lot faster, but it might also be a lot slower, we do not really have any large deployments to test with.

@kradalby commented on GitHub (Sep 9, 2025): That is way more than we expect Headscale to really handle: https://headscale.net/stable/about/faq/#scaling-how-many-clients-does-headscale-support The new `main` release might be a lot faster, but it might also be a lot slower, we do not really have any large deployments to test with.
adam commented 2025-12-29 02:28:00 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 10, 2025):

This is not really a bug, Headscale performs the way it does, we improve it over time.

@kradalby commented on GitHub (Sep 10, 2025): This is not really a bug, Headscale performs the way it does, we improve it over time.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1057
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?