[Bug] LetsEncrypt cache dir lost - unable to bring headscale back up. #994

Closed
opened 2025-12-29 02:27:12 +01:00 by adam · 2 comments
Owner

Originally created by @Bretos on GitHub (Apr 9, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Due to a series of unfortunate events I lost the content of tls_letsencrypt_cache_dir directory. Now headscale falls back to http

  1. restarting headscale:
    1.1 error="configuring TLS settings: open >file< no such file or directory"
    1.2 if I remove TLS paths from config file I get the warning
Listening without TLS but ServerURL does not start with http://

and connecting any clients results in

http: server gave HTTP response to HTTPS client
  1. generating self signed CA (tailscale client throws an error)
  2. launching headscale with https addres and empty cert and key strings, and waiting for cert provisioning - doesn't happen

Expected Behavior

headscale reprovisions LetsEncrypt certificates and runs without any warning or errors, tailscale clients connect without any issue

Steps To Reproduce

  1. set up headscale with LE certificates
  2. remove key and certificate from local filesystem
  3. restart tailscale

Environment

- OS:OPNsense 25.1.4_1-amd64
- Headscale version: v0.25.1
- Tailscale version:

1.82.0
  tailscale commit: 6676b1261e51e0629553ca06b22e6631f8641100
  other commit: 3ec4bfb9c87718a3806a123585c825189cbceda4
  go version: go1.24.1


headscale downloaded as binary from GitHub, running natively without any proxies in between.

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

headscale_config.txt

tailscaled logs

Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: Received error: fetch control key: Get "https://hostname:8880/key?v=115": http: server gave HTTP response to HTTPS client
Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: control: LoginInteractive -> regen=true
Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: [RATELIMIT] format("control: LoginInteractive -> regen=true")
Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: Received error: fetch control key: Get "https://hostname:8880/key?v=115": http: server gave HTTP response to HTTPS client
Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: [RATELIMIT] format("Received error: %v")
Originally created by @Bretos on GitHub (Apr 9, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Due to a series of unfortunate events I lost the content of `tls_letsencrypt_cache_dir` directory. Now `headscale` falls back to http 1. restarting `headscale`: 1.1 `error="configuring TLS settings: open >file< no such file or directory"` 1.2 if I remove TLS paths from config file I get the warning ``` Listening without TLS but ServerURL does not start with http:// ``` and connecting any clients results in ``` http: server gave HTTP response to HTTPS client ``` 2. generating self signed CA (`tailscale` client throws an error) 3. launching `headscale` with https addres and empty cert and key strings, and waiting for cert provisioning - doesn't happen ### Expected Behavior `headscale` reprovisions LetsEncrypt certificates and runs without any warning or errors, tailscale clients connect without any issue ### Steps To Reproduce 1. set up `headscale` with LE certificates 2. remove key and certificate from local filesystem 3. restart tailscale ### Environment ```markdown - OS:OPNsense 25.1.4_1-amd64 - Headscale version: v0.25.1 - Tailscale version: 1.82.0 tailscale commit: 6676b1261e51e0629553ca06b22e6631f8641100 other commit: 3ec4bfb9c87718a3806a123585c825189cbceda4 go version: go1.24.1 headscale downloaded as binary from GitHub, running natively without any proxies in between. ``` ### Runtime environment - [ ] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Debug information [headscale_config.txt](https://github.com/user-attachments/files/19663990/headscale_config.txt) tailscaled logs ``` Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: Received error: fetch control key: Get "https://hostname:8880/key?v=115": http: server gave HTTP response to HTTPS client Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: control: LoginInteractive -> regen=true Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: [RATELIMIT] format("control: LoginInteractive -> regen=true") Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: Received error: fetch control key: Get "https://hostname:8880/key?v=115": http: server gave HTTP response to HTTPS client Apr 09 11:15:16 Modlin-TrueNAS tailscaled[793718]: [RATELIMIT] format("Received error: %v") ```
adam added the stalebug labels 2025-12-29 02:27:12 +01:00
adam closed this issue 2025-12-29 02:27:12 +01:00
Author
Owner

@github-actions[bot] commented on GitHub (Jul 9, 2025):

This issue is stale because it has been open for 90 days with no activity.

@github-actions[bot] commented on GitHub (Jul 9, 2025): This issue is stale because it has been open for 90 days with no activity.
Author
Owner

@github-actions[bot] commented on GitHub (Jul 16, 2025):

This issue was closed because it has been inactive for 14 days since being marked as stale.

@github-actions[bot] commented on GitHub (Jul 16, 2025): This issue was closed because it has been inactive for 14 days since being marked as stale.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#994