[Bug] Headscale v25.1 or Tailscale 1.70+ causes NCSI (Network Status) to appear offline #968

Originally created by @PowershellScripter on GitHub (Mar 6, 2025).

Is this a support request?

This is not a support request

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

When upgrading to headscale 25.1 and the tailscale clients to 1.70+ (Ive tried 1.70 all the way through 1.80.2), After some period of time, the network connection will show me as offline even though I can reach the internal network as well as internet.
Some clients however, have issues also accessing local resources.
Split DNS and Split tunneling are both turned on. Not using full tunnels for any clients.

Headscale v22.3 + tailscale clients on v1.70 didnt have these issues

Expected Behavior

When upgrading tailscale client, client machine should be able to access local resources and printers on the same network and the network status should show online and connected.

Steps To Reproduce

We have tested multiple clients in multiple networks on multiple systems and all are affected when using latest headscale and tailscale

Environment

- OS: EndeavourOS 
- Headscale version: v25.1
- Tailscale version: 1.70+

Runtime environment

Headscale is behind a (reverse) proxy
Headscale runs in a container

Anything else?

headscale_config.txt

Originally created by @PowershellScripter on GitHub (Mar 6, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior When upgrading to headscale 25.1 and the tailscale clients to 1.70+ (Ive tried 1.70 all the way through 1.80.2), After some period of time, the network connection will show me as offline even though I can reach the internal network as well as internet. Some clients however, have issues also accessing local resources. Split DNS and Split tunneling are both turned on. Not using full tunnels for any clients. Headscale v22.3 + tailscale clients on v1.70 didnt have these issues ![Image](https://github.com/user-attachments/assets/523e6223-b3f6-4cba-90c1-04d5e79f0d67) ### Expected Behavior When upgrading tailscale client, client machine should be able to access local resources and printers on the same network and the network status should show online and connected. ### Steps To Reproduce We have tested multiple clients in multiple networks on multiple systems and all are affected when using latest headscale and tailscale ### Environment ```markdown - OS: EndeavourOS - Headscale version: v25.1 - Tailscale version: 1.70+ ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Anything else? [headscale_config.txt](https://github.com/user-attachments/files/19101587/headscale_config.txt)

adam added the stale bug labels 2025-12-29 02:26:49 +01:00

adam closed this issue

adam commented

@PowershellScripter commented on GitHub (Mar 8, 2025):

To add to this - Tailscale BSD package for PFSense (0.1.8) which is tailscale v1.76.1 is falsely reporting the last seen in headscale v25.1

but, if I restart the tailscale service on pfsense, it will update its online status, showing that it is in fact online and talking to headscale

@PowershellScripter commented on GitHub (Mar 8, 2025): To add to this - Tailscale BSD package for PFSense (0.1.8) which is tailscale v1.76.1 is falsely reporting the last seen in headscale v25.1 ![Image](https://github.com/user-attachments/assets/7ef1b5fe-7057-4182-a8b7-6f28a89674d3) ![Image](https://github.com/user-attachments/assets/c95459a1-0638-4bc6-ba46-ebcf2eef52a6) ![Image](https://github.com/user-attachments/assets/d93ff87e-6b85-4f34-a06d-bc45bfdb8659) but, if I restart the tailscale service on pfsense, it will update its online status, showing that it is in fact online and talking to headscale

adam commented

@kradalby commented on GitHub (Mar 8, 2025):

Please read the bug template, we need more information as there is nothing for us to reproduce from or figure out what's happening.

Please attach logs for the tailscale nodes that loose connection and the headscale server.

@kradalby commented on GitHub (Mar 8, 2025): Please read the bug template, we need more information as there is nothing for us to reproduce from or figure out what's happening. Please attach logs for the tailscale nodes that loose connection and the headscale server.

adam commented

@PowershellScripter commented on GitHub (Mar 8, 2025):

Please read the bug template, we need more information as there is nothing for us to reproduce from or figure out what's happening.

Please attach logs for the tailscale nodes that loose connection and the headscale server.

The nodes arent actually losing connection, they are just having very weird network glitches. And each node is experiencing similar but also different issues.

I just re-updated from v1.68.1 to 1.80.2 on my windows system, I will send over the logs once the issue reproduces itself as it is at a random time that it happens.

Should I change the headscale log level from info to debug?

Should I enable logtail?

@PowershellScripter commented on GitHub (Mar 8, 2025): > Please read the bug template, we need more information as there is nothing for us to reproduce from or figure out what's happening. > > Please attach logs for the tailscale nodes that loose connection and the headscale server. The nodes arent actually losing connection, they are just having very weird network glitches. And each node is experiencing similar but also different issues. I just re-updated from v1.68.1 to 1.80.2 on my windows system, I will send over the logs once the issue reproduces itself as it is at a random time that it happens. Should I change the headscale log level from info to debug? Should I enable logtail?

adam commented

@kradalby commented on GitHub (Mar 8, 2025):

Trace for headscale, no logtail

@kradalby commented on GitHub (Mar 8, 2025): Trace for headscale, no logtail

adam commented

headscale_tail-5000_trace.log
tailscale-service-20250308T002035-1741418435_during_NCSI_disconnect_glitch.txt
tailscale-service-20250308T122406-1741461846_right_after_restarting_tailscale.txt

@PowershellScripter commented on GitHub (Mar 8, 2025):

My systems name is WS5015

Here are the logs for:

Headscale (tail 5000 entries)
Tailscale during network glitch issue (NCSI presents offline)
Tailscale right after restarting the connection

If you need any additional logs, let me know what you need and how to grab it for you and I will.

@PowershellScripter commented on GitHub (Mar 8, 2025): My systems name is WS5015 Here are the logs for: - Headscale (tail 5000 entries) - Tailscale during network glitch issue (NCSI presents offline) - Tailscale right after restarting the connection [headscale_tail-5000_trace.log](https://github.com/user-attachments/files/19145511/headscale_tail-5000_trace.log) [tailscale-service-20250308T002035-1741418435_during_NCSI_disconnect_glitch.txt](https://github.com/user-attachments/files/19145510/tailscale-service-20250308T002035-1741418435_during_NCSI_disconnect_glitch.txt) [tailscale-service-20250308T122406-1741461846_right_after_restarting_tailscale.txt](https://github.com/user-attachments/files/19145512/tailscale-service-20250308T122406-1741461846_right_after_restarting_tailscale.txt) If you need any additional logs, let me know what you need and how to grab it for you and I will.

adam commented

@PowershellScripter commented on GitHub (Apr 2, 2025):

Just following up on this. Wanted to see if this is an actual bug or if it is basically due to us coming from a way older build and we just need to blast away and start clean in order for this to not be an issue? To confirm, 25.0 and 25.1 server works without issues with clients 1.68.1 and I believe 1.70, but this issues happens with anything past those client versions.

@PowershellScripter commented on GitHub (Apr 2, 2025): Just following up on this. Wanted to see if this is an actual bug or if it is basically due to us coming from a way older build and we just need to blast away and start clean in order for this to not be an issue? To confirm, 25.0 and 25.1 server works without issues with clients 1.68.1 and I believe 1.70, but this issues happens with anything past those client versions.

adam commented

@PowershellScripter commented on GitHub (May 4, 2025):

@kradalby
I tested upgrading the windows Tailscale client to 1.82.5 and all seemed like it was working properly, until I woke up this morning with my network appearing offline again. Upon disconnecting from the VPN, the network connection icon flipped back online. Its been about 5 min since I have done that (as im writing this) and the network icon flipped back to offline.

(This is important imo because as I hit send, this proves im online even though the vpn is making my system appear offline)

@PowershellScripter commented on GitHub (May 4, 2025): @kradalby I tested upgrading the windows Tailscale client to 1.82.5 and all seemed like it was working properly, until I woke up this morning with my network appearing offline again. Upon disconnecting from the VPN, the network connection icon flipped back online. Its been about 5 min since I have done that (as im writing this) and the network icon flipped back to offline. (This is important imo because as I hit send, this proves im online even though the vpn is making my system appear offline)

adam commented

2mo ago:
https://www.reddit.com/r/Tailscale/comments/1izi7tk/windows_ncis_failing_with_tailscale_involved/

@PowershellScripter commented on GitHub (May 4, 2025):

I did some digging and its possible this may be related to the tailscale clients themselves.

9mo ago:
https://www.reddit.com/r/Tailscale/comments/1efn5pn/windows_reporting_no_internet_access_when/

@PowershellScripter commented on GitHub (May 4, 2025): I did some digging and its possible this may be related to the tailscale clients themselves. 2mo ago: https://www.reddit.com/r/Tailscale/comments/1izi7tk/windows_ncis_failing_with_tailscale_involved/ 9mo ago: https://www.reddit.com/r/Tailscale/comments/1efn5pn/windows_reporting_no_internet_access_when/ ![Image](https://github.com/user-attachments/assets/2789337c-7dd7-4af1-8c30-deb2dddcd0bb)

adam commented

@PowershellScripter commented on GitHub (May 22, 2025):

@kradalby do we know if v26.0 fixes this issue with tailscale clients v1.70+ ?

@PowershellScripter commented on GitHub (May 22, 2025): @kradalby do we know if v26.0 fixes this issue with tailscale clients v1.70+ ?

adam commented