starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-12 04:10:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] tailscale switch hangs when using MagicDNS on FreeBSD and breakds whole system DNS resolution #1098

New Issue
Open
opened 2025-12-29 02:28:15 +01:00 by adam · 1 comment
adam commented 2025-12-29 02:28:15 +01:00
Owner
Copy Link

Originally created by @2Belette on GitHub (Sep 3, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I have two headscale instances.

When I am using tailscale --accept-dns=true and I do tailscale switch xxx it hangs forever...

Some investigations show that my system is not able to do any DNS resolution, which is the reason why tailscale switch is not successful.

My /etc/resolv.conf has nameserver 100.100.100.100 which is correct but no more DNS resolution on my system is done until I restart tailscaled.

I also confirmed that this is only when using MagicDNS, if do tailscale set --accept-dns=false, the switch is working perfectly as my /etc/resolv.conf is untouched.

It seems that switching is touching the internal resolver too quickly even before doing the resolution to the new headscale instance to know where to connect to.

Expected Behavior

tailscale switch to work like when no MagicDNS is used

Steps To Reproduce

tailscale --accept-dns=true
service tailscaled start
check your /etc/resolv.conf -> nameserver 100.100.100.100
check some random DNS resolution and connect to some nodes in the first tailnet
tailscale switch xxx
check again to do any DNS resolution -> fail
service tailscaled restart -> DNS resolution is working again and tailscale has switched correctly to the new tailnet

Environment

- OS:FreeBSD 14.3
- Headscale version: dev (0.26.1)
- Tailscale version:1.86.4

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

Please let me know if anything is needed as this this easy to reproduce on any fresh install

Originally created by @2Belette on GitHub (Sep 3, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior I have two headscale instances. When I am using tailscale --accept-dns=true and I do tailscale switch xxx it hangs forever... Some investigations show that my system is not able to do any DNS resolution, which is the reason why tailscale switch is not successful. My /etc/resolv.conf has nameserver 100.100.100.100 which is correct but no more DNS resolution on my system is done until I restart tailscaled. I also confirmed that this is only when using MagicDNS, if do tailscale set --accept-dns=false, the switch is working perfectly as my /etc/resolv.conf is untouched. It seems that switching is touching the internal resolver too quickly even before doing the resolution to the new headscale instance to know where to connect to. ### Expected Behavior tailscale switch to work like when no MagicDNS is used ### Steps To Reproduce tailscale --accept-dns=true service tailscaled start check your /etc/resolv.conf -> nameserver 100.100.100.100 check some random DNS resolution and connect to some nodes in the first tailnet tailscale switch xxx check again to do any DNS resolution -> fail service tailscaled restart -> DNS resolution is working again and tailscale has switched correctly to the new tailnet ### Environment ```markdown - OS:FreeBSD 14.3 - Headscale version: dev (0.26.1) - Tailscale version:1.86.4 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information Please let me know if anything is needed as this this easy to reproduce on any fresh install
adam added the bugno-stale-botwell described ❤️DNS labels 2025-12-29 02:28:15 +01:00
adam commented 2025-12-29 02:28:15 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Oct 12, 2025):

Please let me know if anything is needed as this this easy to reproduce on any fresh install

I could reproduce this on a fresh FreeBSD 14.3 installation and two (internal) Headscale servers. The tailscaled debug daemon-logs logs this after a switch:

Persist{o=, n=[1PTWC] u="alice"}}
resetDialPlan: did reset
active login:
Switching ipn state Running -> NoState (WantRunning=true, nm=false)
magicsock: SetPrivateKey called (zeroed)
magicsock: closing connection to derp-26 (zero-private-key), age 1m55s
magicsock: closing connection to derp-4 (zero-private-key), age 44s
magicsock: 0 active derp conns
wgengine: Reconfig: configuring userspace WireGuard config (with 0/0 peers)
wgengine: Reconfig: configuring router
router: addr del failed: [ifconfig tailscale0 inet 100.64.0.4/32 -alias] => exit status 1
ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
router: addr del failed: [ifconfig tailscale0 inet6 fd7a:115c:a1e0::4/128 -alias] => exit status 1
ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
Reconfig(down): exit status 1
Start
Backend: logs: be:93629e19baafc5c76467c56dc7d56af814fa341ad3791c7fc251e2fd0dc3ca08 fe:
control: client.Login(0)
control: client.Shutdown ...
control: doLogin(regen=false, hasUrl=false)
control: authRoutine: exiting
control: updateRoutine: exiting
control: mapRoutine: exiting
control: Client.Shutdown done.
[RATELIMIT] format("health(warnable=%s): ok") (2 dropped)
health(warnable=no-derp-connection): ok
[RATELIMIT] format("health(warnable=%s): error: %s") (8 dropped)
health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
control: trying bootstrapDNS("derp17d.tailscale.com", "208.111.40.216") for "headscale1.internal" ...
control: trying bootstrapDNS("derp9c.tailscale.com", "2001:19f0:6401:fe7:5400:3ff:fe8d:6d9c") for "headscale1.internal" ...
control: trying bootstrapDNS("derp3f.tailscale.com", "172.237.72.8") for "headscale1.internal" ...
control: trying bootstrapDNS("derp27e.tailscale.com", "2a01:4ff:f0:28d4::1") for "headscale1.internal" ...
control: trying bootstrapDNS("derp15d.tailscale.com", "102.67.165.36") for "headscale1.internal" ...
[RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...")
health(warnable=no-derp-connection): ok
health(warnable=login-state): error: You are logged out. The last login error was: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal"
Received error: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal"
control: doLogin(regen=false, hasUrl=false)
health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
[RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") (7 dropped)
control: trying bootstrapDNS("derp4g.tailscale.com", "185.40.234.113") for "headscale1.internal" ...
control: trying bootstrapDNS("derp12d.tailscale.com", "2607:f740:e::811") for "headscale1.internal" ...
control: trying bootstrapDNS("derp8g.tailscale.com", "176.58.92.254") for "headscale1.internal" ...
[RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...")
Received error: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal"
health(warnable=no-derp-connection): ok
control: doLogin(regen=false, hasUrl=false)
health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
[RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") (9 dropped)
control: trying bootstrapDNS("derp12b.tailscale.com", "45.63.71.144") for "headscale1.internal" ...
control: trying bootstrapDNS("derp24b.tailscale.com", "2001:19f0:c000:c586:5400:4ff:fe26:2ba6") for "headscale1.internal" ...
[RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...")

The connection also hangs for some time when using a Tailscale SaaS account but it is able to recover using the BootstrapDNS mechanism. Logs via tailscaled debug daemon-logs:

health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
control: trying bootstrapDNS("derp16c.tailscale.com", "192.73.243.229") for "controlplane.tailscale.com" ...
health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable.
control: bootstrapDNS("derp16c.tailscale.com", "192.73.243.229") for "controlplane.tailscale.com" = [2606:b740:49::110 2606:b740:49::111 192.200.0.116 192.200.0.101 192.200.0.114 2606:b740:49::112 2606:b740:49::107 192.200.0.115 2606:b740:49::116 2606:b740:49::105 192.200.0.113 2606:b740:49::109 2606:b740:49::106 192.200.0.105 2606:b740:49::115 192.200.0.107 2606:b740:49::103 192.200.0.110 192.200.0.104 192.200.0.106 2606:b740:49::102 192.200.0.112 192.200.0.103 2606:b740:49::108 192.200.0.108 192.200.0.111 2606:b740:49::114 2606:b740:49::113 2606:b740:49::101 192.200.0.102 2606:b740:49::104 192.200.0.109]
control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
control: RegisterReq: onode= node=[ItrV3] fup=false nks=false
control: controlhttp: forcing port 443 dial due to recent noise dial
control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false
health(warnable=no-derp-connection): ok
health(warnable=no-derp-connection): ok
control: netmap: got new dial plan from control
health(warnable=not-in-map-poll): ok

The client probably uses its cached DERP map (or the static fallback) to connect to some DERP server and get IPs for some well-known Tailscale URLs, example: https://derp10.tailscale.com/bootstrap-dns.

@nblock commented on GitHub (Oct 12, 2025): > Please let me know if anything is needed as this this easy to reproduce on any fresh install I could reproduce this on a fresh FreeBSD 14.3 installation and two (internal) Headscale servers. The `tailscaled debug daemon-logs` logs this after a switch: ``` Persist{o=, n=[1PTWC] u="alice"}} resetDialPlan: did reset active login: Switching ipn state Running -> NoState (WantRunning=true, nm=false) magicsock: SetPrivateKey called (zeroed) magicsock: closing connection to derp-26 (zero-private-key), age 1m55s magicsock: closing connection to derp-4 (zero-private-key), age 44s magicsock: 0 active derp conns wgengine: Reconfig: configuring userspace WireGuard config (with 0/0 peers) wgengine: Reconfig: configuring router router: addr del failed: [ifconfig tailscale0 inet 100.64.0.4/32 -alias] => exit status 1 ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address router: addr del failed: [ifconfig tailscale0 inet6 fd7a:115c:a1e0::4/128 -alias] => exit status 1 ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address Reconfig(down): exit status 1 Start Backend: logs: be:93629e19baafc5c76467c56dc7d56af814fa341ad3791c7fc251e2fd0dc3ca08 fe: control: client.Login(0) control: client.Shutdown ... control: doLogin(regen=false, hasUrl=false) control: authRoutine: exiting control: updateRoutine: exiting control: mapRoutine: exiting control: Client.Shutdown done. [RATELIMIT] format("health(warnable=%s): ok") (2 dropped) health(warnable=no-derp-connection): ok [RATELIMIT] format("health(warnable=%s): error: %s") (8 dropped) health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. control: trying bootstrapDNS("derp17d.tailscale.com", "208.111.40.216") for "headscale1.internal" ... control: trying bootstrapDNS("derp9c.tailscale.com", "2001:19f0:6401:fe7:5400:3ff:fe8d:6d9c") for "headscale1.internal" ... control: trying bootstrapDNS("derp3f.tailscale.com", "172.237.72.8") for "headscale1.internal" ... control: trying bootstrapDNS("derp27e.tailscale.com", "2a01:4ff:f0:28d4::1") for "headscale1.internal" ... control: trying bootstrapDNS("derp15d.tailscale.com", "102.67.165.36") for "headscale1.internal" ... [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") health(warnable=no-derp-connection): ok health(warnable=login-state): error: You are logged out. The last login error was: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal" Received error: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal" control: doLogin(regen=false, hasUrl=false) health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") (7 dropped) control: trying bootstrapDNS("derp4g.tailscale.com", "185.40.234.113") for "headscale1.internal" ... control: trying bootstrapDNS("derp12d.tailscale.com", "2607:f740:e::811") for "headscale1.internal" ... control: trying bootstrapDNS("derp8g.tailscale.com", "176.58.92.254") for "headscale1.internal" ... [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") Received error: fetch control key: Get "https://headscale1.internal:8443/key?v=125": failed to resolve "headscale1.internal": no DNS fallback candidates remain for "headscale1.internal" health(warnable=no-derp-connection): ok control: doLogin(regen=false, hasUrl=false) health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") (9 dropped) control: trying bootstrapDNS("derp12b.tailscale.com", "45.63.71.144") for "headscale1.internal" ... control: trying bootstrapDNS("derp24b.tailscale.com", "2001:19f0:c000:c586:5400:4ff:fe26:2ba6") for "headscale1.internal" ... [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...") ``` The connection also hangs for some time when using a Tailscale SaaS account but it is able to recover using the BootstrapDNS mechanism. Logs via `tailscaled debug daemon-logs`: ``` health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. control: trying bootstrapDNS("derp16c.tailscale.com", "192.73.243.229") for "controlplane.tailscale.com" ... health(warnable=no-derp-connection): error: Tailscale could not connect to the 'Nuremberg' relay server. Your Internet connection might be down, or the server might be temporarily unavailable. control: bootstrapDNS("derp16c.tailscale.com", "192.73.243.229") for "controlplane.tailscale.com" = [2606:b740:49::110 2606:b740:49::111 192.200.0.116 192.200.0.101 192.200.0.114 2606:b740:49::112 2606:b740:49::107 192.200.0.115 2606:b740:49::116 2606:b740:49::105 192.200.0.113 2606:b740:49::109 2606:b740:49::106 192.200.0.105 2606:b740:49::115 192.200.0.107 2606:b740:49::103 192.200.0.110 192.200.0.104 192.200.0.106 2606:b740:49::102 192.200.0.112 192.200.0.103 2606:b740:49::108 192.200.0.108 192.200.0.111 2606:b740:49::114 2606:b740:49::113 2606:b740:49::101 192.200.0.102 2606:b740:49::104 192.200.0.109] control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp] control: RegisterReq: onode= node=[ItrV3] fup=false nks=false control: controlhttp: forcing port 443 dial due to recent noise dial control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false health(warnable=no-derp-connection): ok health(warnable=no-derp-connection): ok control: netmap: got new dial plan from control health(warnable=not-in-map-poll): ok ``` The client probably uses its cached DERP map (or the static fallback) to connect to some DERP server and get IPs for some well-known Tailscale URLs, example: `https://derp10.tailscale.com/bootstrap-dns`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label DNS bug no-stale-bot well described ❤️
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1098
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?