starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

Doc request: Expected renewal behavior for autocert #628

New Issue
Closed
opened 2025-12-29 02:21:21 +01:00 by adam · 7 comments
adam commented 2025-12-29 02:21:21 +01:00
Owner
Copy Link

Originally created by @lgrn on GitHub (Feb 8, 2024).

Why

The current documentation regarding built-in TLS certificate auto-renewals isn't completely clear on what to expect, it currently only states:

The certificate will automatically be renewed as needed.

Description

I think it would be helpful to document some further information on what intervals should be expected here, and what errors should be actioned/ignored. For example:

  • What actual interval might successful renewals happen at? One could guess it's 30 days since that seems to be the LetsEncrypt standard, but that's implicit and not stated in the doc.
  • How often are attempts at renewals done? Daily, weekly? What log lines can one look out for to verify that the challenge succeeded or failed and is actually working as expected?

Also, the meaning or interpretation of some common log lines would be helpful, like:

  • acme/autocert: missing server name (?)
  • acme/autocert: host "[redacted-external-ip]" not configured in HostWhitelist (?)

I understand that there are alternate solutions to cert renewals and that this could be set up separately ("Bring your own certificate"), but that's not the scope for this request.

Originally created by @lgrn on GitHub (Feb 8, 2024). ## Why The [current documentation](https://headscale.net/tls/) regarding built-in TLS certificate auto-renewals isn't completely clear on what to expect, it currently only states: > The certificate will automatically be renewed as needed. ## Description I think it would be helpful to document some further information on what intervals should be expected here, and what errors should be actioned/ignored. For example: - What actual interval might successful renewals happen at? One could guess it's [30 days](https://community.letsencrypt.org/t/solved-how-often-to-renew/13678) since that seems to be the LetsEncrypt standard, but that's implicit and not stated in the doc. - How often are attempts at renewals done? Daily, weekly? What log lines can one look out for to verify that the challenge succeeded or failed and is actually working as expected? Also, the meaning or interpretation of some common log lines would be helpful, like: - `acme/autocert: missing server name` (?) - `acme/autocert: host "[redacted-external-ip]" not configured in HostWhitelist` (?) I understand that there are alternate solutions to cert renewals and that this could be set up separately ("Bring your own certificate"), but that's not the scope for this request.
adam added the enhancement label 2025-12-29 02:21:21 +01:00
adam closed this issue 2025-12-29 02:21:21 +01:00
adam commented 2025-12-29 02:21:21 +01:00
Author
Owner
Copy Link

@ohdearaugustin commented on GitHub (Feb 9, 2024):

As usual PRs, which improve the docs are very welcome.

@ohdearaugustin commented on GitHub (Feb 9, 2024): As usual PRs, which improve the docs are very welcome.
adam commented 2025-12-29 02:21:22 +01:00
Author
Owner
Copy Link

@lgrn commented on GitHub (Feb 9, 2024):

Answers to the questions are very welcome too, it makes writing docs a bit easier.

@lgrn commented on GitHub (Feb 9, 2024): Answers to the questions are very welcome too, it makes writing docs a bit easier.
adam commented 2025-12-29 02:21:22 +01:00
Author
Owner
Copy Link

@ohdearaugustin commented on GitHub (Feb 9, 2024):

What actual interval might successful renewals happen at? One could guess it's 30 days since that seems to be the LetsEncrypt standard, but that's implicit and not stated in the doc.

So headscale uses the autocert for the implementation for acme, in this case letsencrypt.

The corresponding code can be found in app.go:
c3257e2146/hscontrol/app.go (L846-L882)
As you can see only basic configuration is passed to the library. This configuration doesn't give an any clue about the renewal time.

I guess as you know letsencrypt certificates are normally valid for 3 month (approx. 90 days). This lead us to take a look at the library itself:
https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=134-139

If zero, they're renewed 30 days before expiration.

As we can see in the code snippet from headscale, we do not explicitly set RenewBefore therefore we will get the default value of 720 * time.Hour

Found here: https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=1034

This should answer the first part.

@ohdearaugustin commented on GitHub (Feb 9, 2024): > What actual interval might successful renewals happen at? One could guess it's [30 days](https://community.letsencrypt.org/t/solved-how-often-to-renew/13678) since that seems to be the LetsEncrypt standard, but that's implicit and not stated in the doc. So headscale uses the [autocert](https://pkg.go.dev/golang.org/x/crypto/acme/autocert) for the implementation for acme, in this case letsencrypt. The corresponding code can be found in app.go: https://github.com/juanfont/headscale/blob/c3257e2146304c52e588c6de2fd28bcc0f13b1ad/hscontrol/app.go#L846-L882 As you can see only basic configuration is passed to the library. This configuration doesn't give an any clue about the renewal time. I guess as you know letsencrypt certificates are normally valid for 3 month (approx. 90 days). This lead us to take a look at the library itself: https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=134-139 >If zero, they're renewed 30 days before expiration. As we can see in the code snippet from headscale, we do not explicitly set `RenewBefore` therefore we will get the default value of `720 * time.Hour` Found here: https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=1034 This should answer the first part.
adam commented 2025-12-29 02:21:23 +01:00
Author
Owner
Copy Link

@ohdearaugustin commented on GitHub (Feb 9, 2024):

acme/autocert: missing server name (?)
This can be found here

	if name == "" {
		return nil, errors.New("acme/autocert: missing server name")
	}

This could mean no hostname for the certificate is set so probably tls_letsencrypt_hostname is not set in your config.

Edit: The library gives also this error message when you directly curl the IP of your headscale server and not the fqdn. As the letsencrypt certificate does not include the ip.

acme/autocert: host "[redacted-external-ip]" not configured in HostWhitelist

This answer can be found here. I just assume now that you got both error messages at the same time, which mean your tls_letsencrypt_hostname is not valid.

Edit: This can also mean that someone tries to resolve a different fqdn to your server.

Saw this errors today also on my server. Pretty useless error messages from the library to be honest.

@ohdearaugustin commented on GitHub (Feb 9, 2024): > `acme/autocert: missing server name (?)` This can be found [here](https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=254) ``` name := hello.ServerName if name == "" { return nil, errors.New("acme/autocert: missing server name") } ``` ~~This could mean no hostname for the certificate is set so probably `tls_letsencrypt_hostname` is not set in your config.~~ Edit: The library gives also this error message when you directly curl the IP of your headscale server and not the fqdn. As the letsencrypt certificate does not include the ip. > `acme/autocert: host "[redacted-external-ip]" not configured in HostWhitelist` This answer can be found [here](https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go;drc=4ba4fb4dd9e7f7ed9053fb45482e9a725c7e3fb4;l=75). ~~I just assume now that you got both error messages at the same time, which mean your tls_letsencrypt_hostname is not valid.~~ Edit: This can also mean that someone tries to resolve a different fqdn to your server. Saw this errors today also on my server. Pretty useless error messages from the library to be honest.
adam commented 2025-12-29 02:21:24 +01:00
Author
Owner
Copy Link

@ohdearaugustin commented on GitHub (Feb 9, 2024):

How often are attempts at renewals done? Daily, weekly?

This question is harder to answer because you have to take a even deeper dive into the code. The corresponding code can be found here.

	if err != nil {
		next = renewJitter / 2
		next += time.Duration(pseudoRand.int63n(int64(next)))
	}

As far as I understand it the renewJitter is set to 1h. On a failed renewal of the certificate this is split to 30mins + a random time duration between 0 and 30mins. So your min retry attempt is 30mins after the first or max 60min after the first. And probably after some time you will run into the rate limits of letsencrypt.

What log lines can one look out for to verify that the challenge succeeded or failed and is actually working as expected?

It seems like that there is no log written for successful renewal of the certificate. Rather the library only gives out errors on specific occasions like: acme/autocert: invalid new order status %q; order URL: %q" or acme/autocert: invalid account key found in cache for more possible error message look at the autocert

So if the challenge worked it should have created the certificate and be serving it. The easiest way to check if the worked is to check the certificate in the browser or with openssl x509 -in <FQDN> -noout -text in the cache folder of headscale.

@ohdearaugustin commented on GitHub (Feb 9, 2024): > How often are attempts at renewals done? Daily, weekly? This question is harder to answer because you have to take a even deeper dive into the code. The corresponding code can be found [here](https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/renewal.go;drc=f4118a5b28e206eb1f30037b11ab68a3cd5e6442;l=81). ``` if err != nil { next = renewJitter / 2 next += time.Duration(pseudoRand.int63n(int64(next))) } ``` As far as I understand it the `renewJitter` is set to 1h. On a failed renewal of the certificate this is split to 30mins + a random time duration between 0 and 30mins. So your min retry attempt is 30mins after the first or max 60min after the first. And probably after some time you will run into the [rate limits](https://letsencrypt.org/docs/rate-limits/) of letsencrypt. > What log lines can one look out for to verify that the challenge succeeded or failed and is actually working as expected? It seems like that there is no log written for successful renewal of the certificate. Rather the library only gives out errors on specific occasions like: `acme/autocert: invalid new order status %q; order URL: %q"` or `acme/autocert: invalid account key found in cache` for more possible error message look at the [autocert](https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.19.0:acme/autocert/autocert.go) So if the challenge worked it should have created the certificate and be serving it. The easiest way to check if the worked is to check the certificate in the browser or with `openssl x509 -in <FQDN> -noout -text` in the cache folder of headscale.
adam commented 2025-12-29 02:21:24 +01:00
Author
Owner
Copy Link

@lgrn commented on GitHub (Feb 10, 2024):

I've created a draft PR here, feel free to have a look when you have the time: https://github.com/juanfont/headscale/pull/1733

Both feedback and direct commits are welcome, I'm under no illusions that it's perfect as-is.

@lgrn commented on GitHub (Feb 10, 2024): I've created a draft PR here, feel free to have a look when you have the time: https://github.com/juanfont/headscale/pull/1733 Both feedback and direct commits are welcome, I'm under no illusions that it's perfect as-is.
adam commented 2025-12-29 02:21:24 +01:00
Author
Owner
Copy Link

@TotoTheDragon commented on GitHub (Feb 20, 2024):

@kradalby this has been merged, issue can be closed

@TotoTheDragon commented on GitHub (Feb 20, 2024): @kradalby this has been merged, issue can be closed
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#628
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?