starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

alpha9 takes 90 seconds to stop #698

New Issue
Closed
opened 2025-12-29 02:22:18 +01:00 by adam · 7 comments
adam commented 2025-12-29 02:22:18 +01:00
Owner
Copy Link

Originally created by @mooncaptain108 on GitHub (Apr 22, 2024).

Bug description

After upgrading from alpha8 to alpha9 I found that the stop increased from almost instantly to 90 seconds.
systemctl stop headscale
here's the journalctl log

Apr 22 12:05:09 headscale systemd[1]: Stopping headscale coordination server for Tailscale...
Apr 22 12:05:09 headscale headscale[4098]: 2024-04-22T12:05:09-05:00 INF Received signal to stop, shutting down gracefully signal=terminated
Apr 22 12:06:39 headscale systemd[1]: headscale.service: State 'stop-sigterm' timed out. Killing.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4098 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4099 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4100 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4101 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4102 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4103 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4104 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4105 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Main process exited, code=killed, status=9/KILL
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Failed with result 'timeout'.
Apr 22 12:06:39 headscale systemd[1]: Stopped headscale coordination server for Tailscale.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Consumed 33.940s CPU time.

Environment

  • OS: ubuntu 22.04
  • Headscale version: v0.23.0-alpha9
  • Tailscale version: 1.64.0
  • [x ] Headscale is behind a (reverse) proxy
  • Headscale runs in a container

To Reproduce

  1. a running alpha8 release - which I could stop and start quickly
  2. download the alpha9.deb
  3. apt install headscale.deb
  4. start headscale
  5. I was doing some testing from a remote node and noticed that not all nodes were showing up in my tailscale status list.
  6. made some changes to the acl and restarted headscale.
  7. Noticed the delay - it finally came back and I started to dig in.
    My tailscale status problem was resolved after the restart.
  8. I reverted my acl changes and that didn't make difference.
  9. I copied my backup db back in. and that didn't help.
  10. I did just: systemctl stop headscale and found that the stop was where the delay occurred
  11. described the problem on discord.

Logs and attachments

See above for journalctl output

Originally created by @mooncaptain108 on GitHub (Apr 22, 2024). <!-- Before posting a bug report, discuss the behaviour you are expecting with the Discord community to make sure that it is truly a bug. The issue tracker is not the place to ask for support or how to set up Headscale. Bug reports without the sufficient information will be closed. Headscale is a multinational community across the globe. Our language is English. All bug reports need to be in English. --> ## Bug description After upgrading from alpha8 to alpha9 I found that the stop increased from almost instantly to 90 seconds. `systemctl stop headscale` here's the journalctl log > Apr 22 12:05:09 headscale systemd[1]: Stopping headscale coordination server for Tailscale... > Apr 22 12:05:09 headscale headscale[4098]: 2024-04-22T12:05:09-05:00 INF Received signal to stop, shutting down gracefully signal=terminated > Apr 22 12:06:39 headscale systemd[1]: headscale.service: State 'stop-sigterm' timed out. Killing. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4098 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4099 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4100 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4101 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4102 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4103 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4104 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4105 (headscale) with signal SIGKILL. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Main process exited, code=killed, status=9/KILL > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Failed with result 'timeout'. > Apr 22 12:06:39 headscale systemd[1]: Stopped headscale coordination server for Tailscale. > Apr 22 12:06:39 headscale systemd[1]: headscale.service: Consumed 33.940s CPU time. <!-- A clear and concise description of what the bug is. Describe the expected bahavior and how it is currently different. If you are unsure if it is a bug, consider discussing it on our Discord server first. --> ## Environment <!-- Please add relevant information about your system. For example: - Version of headscale used - Version of tailscale client - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version - Kernel version - The relevant config parameters you used - Log output --> - OS: ubuntu 22.04 - Headscale version: v0.23.0-alpha9 - Tailscale version: 1.64.0 <!-- We do not support running Headscale in a container nor behind a (reverse) proxy. If either of these are true for your environment, ask the community in Discord instead of filing a bug report. --> - [x ] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ## To Reproduce <!-- Steps to reproduce the behavior. --> 0. a running alpha8 release - which I could stop and start quickly 1. download the alpha9.deb 2. apt install headscale.deb 3. start headscale 4. I was doing some testing from a remote node and noticed that not all nodes were showing up in my tailscale status list. 5. made some changes to the acl and restarted headscale. 6. Noticed the delay - it finally came back and I started to dig in. My tailscale status problem was resolved after the restart. 7. I reverted my acl changes and that didn't make difference. 8. I copied my backup db back in. and that didn't help. 9. I did just: `systemctl stop headscale` and found that the stop was where the delay occurred 10. described the problem on discord. ## Logs and attachments See above for journalctl output <!-- Please attach files with: - Client netmap dump (see below) - ACL configuration - Headscale configuration Dump the netmap of tailscale clients: `tailscale debug netmap > DESCRIPTIVE_NAME.json` Please provide information describing the netmap, which client, which headscale version etc. -->
adam added the bug label 2025-12-29 02:22:18 +01:00
adam closed this issue 2025-12-29 02:22:18 +01:00
adam commented 2025-12-29 02:22:18 +01:00
Author
Owner
Copy Link

@cy1der commented on GitHub (Apr 23, 2024):

+1 on this, same issue

@cy1der commented on GitHub (Apr 23, 2024): +1 on this, same issue
adam commented 2025-12-29 02:22:18 +01:00
Author
Owner
Copy Link

@remisharrock commented on GitHub (Apr 24, 2024):

same here

@remisharrock commented on GitHub (Apr 24, 2024): same here
adam commented 2025-12-29 02:22:19 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (May 2, 2024):

@mooncaptain108, I'm struggling to replicate this in my own setup, could you try https://github.com/juanfont/headscale/pull/1922 and post the shutdown log?

@kradalby commented on GitHub (May 2, 2024): @mooncaptain108, I'm struggling to replicate this in my own setup, could you try https://github.com/juanfont/headscale/pull/1922 and post the shutdown log?
adam commented 2025-12-29 02:22:19 +01:00
Author
Owner
Copy Link

@mooncaptain108 commented on GitHub (May 2, 2024):

Hi Juan,

Please forgive my ignorance here. I would really like to help. It looks like #1922 is a code change that would require me to build headscale locally. Or is there a build asset (amd64.deb) that I can download and install? I could probably figure out how to do the build. I have dabbled a tiny bit in go but I may not get it right.

Cheers,

paul

From: Kristoffer Dalby @.>
Sent: Thursday, May 2, 2024 4:32 AM
To: juanfont/headscale @.>
Cc: mooncaptain108 @.>; Mention @.>
Subject: Re: [juanfont/headscale] alpha9 takes 90 seconds to stop (Issue #1911)

@mooncaptain108https://github.com/mooncaptain108, I'm struggling to replicate this in my own setup, could you try #1922https://github.com/juanfont/headscale/pull/1922 and post the shutdown log?

—
Reply to this email directly, view it on GitHubhttps://github.com/juanfont/headscale/issues/1911#issuecomment-2090014103, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQZHACBBV5L5LH47LQWT273ZAIBZ7AVCNFSM6AAAAABGTHHGYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGAYTIMJQGM.
You are receiving this because you were mentioned.

@mooncaptain108 commented on GitHub (May 2, 2024): Hi Juan, Please forgive my ignorance here. I would really like to help. It looks like #1922 is a code change that would require me to build headscale locally. Or is there a build asset (amd64.deb) that I can download and install? I could probably figure out how to do the build. I have dabbled a tiny bit in go but I may not get it right. Cheers, paul ________________________________ From: Kristoffer Dalby ***@***.***> Sent: Thursday, May 2, 2024 4:32 AM To: juanfont/headscale ***@***.***> Cc: mooncaptain108 ***@***.***>; Mention ***@***.***> Subject: Re: [juanfont/headscale] alpha9 takes 90 seconds to stop (Issue #1911) @mooncaptain108<https://github.com/mooncaptain108>, I'm struggling to replicate this in my own setup, could you try #1922<https://github.com/juanfont/headscale/pull/1922> and post the shutdown log? — Reply to this email directly, view it on GitHub<https://github.com/juanfont/headscale/issues/1911#issuecomment-2090014103>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQZHACBBV5L5LH47LQWT273ZAIBZ7AVCNFSM6AAAAABGTHHGYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGAYTIMJQGM>. You are receiving this because you were mentioned.
adam commented 2025-12-29 02:22:19 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (May 2, 2024):

I just released https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha11, the change is in there.

@kradalby commented on GitHub (May 2, 2024): I just released https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha11, the change is in there.
adam commented 2025-12-29 02:22:19 +01:00
Author
Owner
Copy Link

@mooncaptain108 commented on GitHub (May 2, 2024):

The problem is not present in alpha11.
I did a systemctl stop headscale
and here's what I got in journalctl.

May 02 09:44:43 headscale systemd[1]: Stopping headscale coordination server for Tailscale...
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Received signal to stop, shutting down gracefully signal=terminated
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Headscale stopped
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 FTL ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="http: Server closed"
May 02 09:44:43 headscale systemd[1]: headscale.service: Main process exited, code=exited, status=1/FAILURE
May 02 09:44:43 headscale systemd[1]: headscale.service: Failed with result 'exit-code'.
May 02 09:44:43 headscale systemd[1]: Stopped headscale coordination server for Tailscale.

headscale stopped immediately and I was able also to do restart and have it immediately startup again.

Just so you know when I installed alpha11 the install script had to wait 90 seconds to shutdown alpha10.

@mooncaptain108 commented on GitHub (May 2, 2024): The problem is not present in alpha11. I did a `systemctl stop headscale` and here's what I got in journalctl. > May 02 09:44:43 headscale systemd[1]: Stopping headscale coordination server for Tailscale... May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Received signal to stop, shutting down gracefully signal=terminated May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Headscale stopped May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 FTL ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="http: Server closed" May 02 09:44:43 headscale systemd[1]: headscale.service: Main process exited, code=exited, status=1/FAILURE May 02 09:44:43 headscale systemd[1]: headscale.service: Failed with result 'exit-code'. May 02 09:44:43 headscale systemd[1]: Stopped headscale coordination server for Tailscale. headscale stopped immediately and I was able also to do restart and have it immediately startup again. Just so you know when I installed alpha11 the install script had to wait 90 seconds to shutdown alpha10.
adam commented 2025-12-29 02:22:20 +01:00
Author
Owner
Copy Link

@mooncaptain108 commented on GitHub (May 2, 2024):

this problem is closed for me.

@mooncaptain108 commented on GitHub (May 2, 2024): this problem is closed for me.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#698
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?