starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

acme/autocert: missing certificate #318

New Issue
Closed
opened 2025-12-29 01:26:47 +01:00 by adam · 11 comments
adam commented 2025-12-29 01:26:47 +01:00
Owner
Copy Link

Originally created by @db48x on GitHub (Sep 1, 2022).

Bug description

I am attempting to log a computer into a vpn managed by headscale, but it isn’t working. The tailscale daemon doesn’t report any errors, but the headscale server is printing numerous messages like this to the journal:

Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59028: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/
Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59498: acme/autocert: missing certificate
Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59768: acme/autocert: missing certificate
Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59338: acme/autocert: missing certificate
Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:60274: acme/autocert: missing certificate
Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59912: acme/autocert: missing certificate
Sep 01 04:25:29 vogar headscale[2856685]: 2022/09/01 04:25:29 http: TLS handshake error from 64.71.131.242:60572: acme/autocert: missing certificate

64.71.131.242 is the machine that is trying to join the vpn.

To Reproduce

I’m not entirely sure which details are relevant to reproducing this error. This is the first machine that I have tried to connect to this vpn. I am using the tailscale --login-server and --auth-key options.

Context info

Headscale is version v0.16.4.
Tailscale is version 1.28.0 built from the following commits:
tailscale commit: 80313cdee14dee22452ea2b278e7bbcfe4af1a9c
other commit: d26dd4a68f4c18686b003051acd7e44d8788a4b7
go version: go1.18.4-ts149f7d88f1

The server is running Debian 11.4 with kernel version 5.10.0-8-amd64.

Originally created by @db48x on GitHub (Sep 1, 2022). **Bug description** I am attempting to log a computer into a vpn managed by headscale, but it isn’t working. The tailscale daemon doesn’t report any errors, but the headscale server is printing numerous messages like this to the journal: ``` Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59028: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/ Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59498: acme/autocert: missing certificate Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59768: acme/autocert: missing certificate Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59338: acme/autocert: missing certificate Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:60274: acme/autocert: missing certificate Sep 01 04:25:19 vogar headscale[2856685]: 2022/09/01 04:25:19 http: TLS handshake error from 64.71.131.242:59912: acme/autocert: missing certificate Sep 01 04:25:29 vogar headscale[2856685]: 2022/09/01 04:25:29 http: TLS handshake error from 64.71.131.242:60572: acme/autocert: missing certificate ``` 64.71.131.242 is the machine that is trying to join the vpn. **To Reproduce** I’m not entirely sure which details are relevant to reproducing this error. This is the first machine that I have tried to connect to this vpn. I am using the tailscale `--login-server` and `--auth-key` options. **Context info** Headscale is version v0.16.4. Tailscale is version 1.28.0 built from the following commits: tailscale commit: 80313cdee14dee22452ea2b278e7bbcfe4af1a9c other commit: d26dd4a68f4c18686b003051acd7e44d8788a4b7 go version: go1.18.4-ts149f7d88f1 The server is running Debian 11.4 with kernel version 5.10.0-8-amd64.
adam added the bug label 2025-12-29 01:26:47 +01:00
adam closed this issue 2025-12-29 01:26:47 +01:00
adam commented 2025-12-29 01:26:48 +01:00
Author
Owner
Copy Link

@db48x commented on GitHub (Sep 1, 2022):

I have tls_letsencrypt_hostname set to "vpn.core.headline.com", and tls_letsencrypt_listen set to ":81". Nginx is already listening on port 80, so it is configured to proxy any request with Host: vpn.core.headline.com to port 81.

That appears to be working correctly, but it is still failing to pass the challenge.

If I connect to port 81 myself, the server responds with a different error:

$ curl -v "http://vpn.core.headline.com:81/.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30"                                                                                           
*   Trying 2001:470:1:b95::76:81...
* Connected to vpn.core.headline.com (2001:470:1:b95::76) port 81 (#0)
> GET /.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30 HTTP/1.1
> Host: vpn.core.headline.com:81
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 01 Sep 2022 06:24:21 GMT
< Content-Length: 79
< 
acme/autocert: host "vpn.core.headline.com:81" not configured in HostWhitelist

That makes sense, given how it is configured:

		certManager := autocert.Manager{
			Prompt:     autocert.AcceptTOS,
			HostPolicy: autocert.HostWhitelist(h.cfg.TLS.LetsEncrypt.Hostname),
			…
		}

When I test it via the nginx proxy:

$ curl -v "http://vpn.core.headline.com/.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30"  
*   Trying 2001:470:1:b95::76:80...
* Connected to vpn.core.headline.com (2001:470:1:b95::76) port 80 (#0)
> GET /.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30 HTTP/1.1
> Host: vpn.core.headline.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx/1.22.0
< Date: Thu, 01 Sep 2022 06:38:40 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 38
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< 
acme/autocert: certificate cache miss

I instead get a message which I believe indicates that it doesn’t know about this challenge. Fair enough; I copied it out of the logs from a previous run so there’s no reason for it to still be valid.

It doesn’t shed much light on what is going wrong though, and I’m still getting the “acme/autocert: missing certificate” message from headscale.

@db48x commented on GitHub (Sep 1, 2022): I have `tls_letsencrypt_hostname` set to `"vpn.core.headline.com"`, and `tls_letsencrypt_listen` set to `":81"`. Nginx is already listening on port 80, so it is configured to proxy any request with `Host: vpn.core.headline.com` to port 81. That appears to be working correctly, but it is still failing to pass the challenge. If I connect to port 81 myself, the server responds with a different error: ``` $ curl -v "http://vpn.core.headline.com:81/.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30" * Trying 2001:470:1:b95::76:81... * Connected to vpn.core.headline.com (2001:470:1:b95::76) port 81 (#0) > GET /.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30 HTTP/1.1 > Host: vpn.core.headline.com:81 > User-Agent: curl/7.74.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 403 Forbidden < Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff < Date: Thu, 01 Sep 2022 06:24:21 GMT < Content-Length: 79 < acme/autocert: host "vpn.core.headline.com:81" not configured in HostWhitelist ``` That makes sense, given how it is configured: ``` certManager := autocert.Manager{ Prompt: autocert.AcceptTOS, HostPolicy: autocert.HostWhitelist(h.cfg.TLS.LetsEncrypt.Hostname), … } ``` When I test it via the nginx proxy: ``` $ curl -v "http://vpn.core.headline.com/.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30" * Trying 2001:470:1:b95::76:80... * Connected to vpn.core.headline.com (2001:470:1:b95::76) port 80 (#0) > GET /.well-known/acme-challenge/8ROQg9MiuSaPRviX0mueN7870M78GPGOuYGk0qhUI30 HTTP/1.1 > Host: vpn.core.headline.com > User-Agent: curl/7.74.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 404 Not Found < Server: nginx/1.22.0 < Date: Thu, 01 Sep 2022 06:38:40 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 38 < Connection: keep-alive < X-Content-Type-Options: nosniff < acme/autocert: certificate cache miss ``` I instead get a message which I believe indicates that it doesn’t know about this challenge. Fair enough; I copied it out of the logs from a previous run so there’s no reason for it to still be valid. It doesn’t shed much light on what is going wrong though, and I’m still getting the “acme/autocert: missing certificate” message from headscale.
adam commented 2025-12-29 01:26:48 +01:00
Author
Owner
Copy Link

@db48x commented on GitHub (Sep 1, 2022):

Now I am more confused. I left it sitting there for a while and when I returned:

…
Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:39118: acme/autocert: missing certificate                                                           
Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38244: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many failed authorizations recently: see 
https://letsencrypt.org/docs/failed-validation-limit/                                                                                                                                                                                  Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:39462: acme/autocert: missing certificate                                                                                   
Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38718: acme/autocert: missing certificate                                                                                   
Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38924: acme/autocert: missing certificate                                                           
Sep 01 07:17:48 vogar headscale[3350371]: 2022/09/01 07:17:48 http: TLS handshake error from 64.71.131.242:39744: acme/autocert: missing certificate                                                                                   
Sep 01 07:18:36 vogar headscale[3350371]: 2022/09/01 07:18:36 http: TLS handshake error from 64.71.131.242:39846: EOF                                                                                                                  
Sep 01 07:19:05 vogar headscale[3350371]: 2022-09-01T07:19:05Z ERR home/runner/work/headscale/headscale/api.go:759 > Failed authentication via AuthKey error="AuthKey expired" func=handleAuthKey machine=vogar
Sep 01 07:19:05 vogar headscale[3350371]: 2022-09-01T07:19:05Z ERR home/runner/work/headscale/headscale/api.go:791 > Failed authentication via AuthKey func=handleAuthKey machine=vogar                        
Sep 01 07:19:21 vogar headscale[3350371]: 2022-09-01T07:19:21Z ERR home/runner/work/headscale/headscale/api.go:759 > Failed authentication via AuthKey error="AuthKey expired" func=handleAuthKey machine=vogar
Sep 01 07:19:21 vogar headscale[3350371]: 2022-09-01T07:19:21Z ERR home/runner/work/headscale/headscale/api.go:791 > Failed authentication via AuthKey func=handleAuthKey machine=vogar
…

It has started working! Except the auth-key I created yesterday has expired. Creating a new key and then joining the network worked fine:

Sep 01 08:26:15 vogar headscale[3350371]: 2022-09-01T08:26:15Z INF Successfully authenticated via AuthKey func=handleAuthKey ips="fd7a:115c:a1e0::1, 100.64.0.1" machine=vogar

Thankfully everything appears to work fine now, so I don’t need any support at the moment. However, I leave this bug report open in the hope that someone can either fix or document the “acme/autocert: missing certificate” problem.

@db48x commented on GitHub (Sep 1, 2022): Now I am more confused. I left it sitting there for a while and when I returned: ``` … Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:39118: acme/autocert: missing certificate Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38244: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/ Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:39462: acme/autocert: missing certificate Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38718: acme/autocert: missing certificate Sep 01 07:17:22 vogar headscale[3350371]: 2022/09/01 07:17:22 http: TLS handshake error from 64.71.131.242:38924: acme/autocert: missing certificate Sep 01 07:17:48 vogar headscale[3350371]: 2022/09/01 07:17:48 http: TLS handshake error from 64.71.131.242:39744: acme/autocert: missing certificate Sep 01 07:18:36 vogar headscale[3350371]: 2022/09/01 07:18:36 http: TLS handshake error from 64.71.131.242:39846: EOF Sep 01 07:19:05 vogar headscale[3350371]: 2022-09-01T07:19:05Z ERR home/runner/work/headscale/headscale/api.go:759 > Failed authentication via AuthKey error="AuthKey expired" func=handleAuthKey machine=vogar Sep 01 07:19:05 vogar headscale[3350371]: 2022-09-01T07:19:05Z ERR home/runner/work/headscale/headscale/api.go:791 > Failed authentication via AuthKey func=handleAuthKey machine=vogar Sep 01 07:19:21 vogar headscale[3350371]: 2022-09-01T07:19:21Z ERR home/runner/work/headscale/headscale/api.go:759 > Failed authentication via AuthKey error="AuthKey expired" func=handleAuthKey machine=vogar Sep 01 07:19:21 vogar headscale[3350371]: 2022-09-01T07:19:21Z ERR home/runner/work/headscale/headscale/api.go:791 > Failed authentication via AuthKey func=handleAuthKey machine=vogar … ``` It has started working! Except the auth-key I created yesterday has expired. Creating a new key and then joining the network worked fine: ``` Sep 01 08:26:15 vogar headscale[3350371]: 2022-09-01T08:26:15Z INF Successfully authenticated via AuthKey func=handleAuthKey ips="fd7a:115c:a1e0::1, 100.64.0.1" machine=vogar ``` Thankfully everything appears to work fine now, so I don’t need any support at the moment. However, I leave this bug report open in the hope that someone can either fix or document the “acme/autocert: missing certificate” problem.
adam commented 2025-12-29 01:26:48 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Sep 4, 2022):

Hi @db48x, can you try with 0.17.0-alpha2 ?

@juanfont commented on GitHub (Sep 4, 2022): Hi @db48x, can you try with 0.17.0-alpha2 ?
adam commented 2025-12-29 01:26:49 +01:00
Author
Owner
Copy Link

@db48x commented on GitHub (Sep 4, 2022):

I’ll give it a go on Monday. Which of the most recent changes are most relevant? The websocket warnings? The timeout?

@db48x commented on GitHub (Sep 4, 2022): I’ll give it a go on Monday. Which of the most recent changes are most relevant? The websocket warnings? The timeout?
adam commented 2025-12-29 01:26:50 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Sep 4, 2022):

I think the issue is that now modern Tailscale clients try to initiate the Tailscale v2 connection over tcp/80 and do an upgrade over websockets.

Let's try if updating your server helps...

@juanfont commented on GitHub (Sep 4, 2022): I think the issue is that now modern Tailscale clients try to initiate the Tailscale v2 connection over tcp/80 and do an upgrade over websockets. Let's try if updating your server helps...
adam commented 2025-12-29 01:26:50 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Sep 7, 2022):

@db48x have you been able to test again?

@juanfont commented on GitHub (Sep 7, 2022): @db48x have you been able to test again?
adam commented 2025-12-29 01:26:50 +01:00
Author
Owner
Copy Link

@db48x commented on GitHub (Sep 8, 2022):

Ok, I’ve got alpha2 installed, but it isn’t working at all. It’s supposed to listen on :8080 (listen_addr: 0.0.0.0:8080), but it isn’t. Is that #799?

@db48x commented on GitHub (Sep 8, 2022): Ok, I’ve got alpha2 installed, but it isn’t working at all. It’s supposed to listen on :8080 (`listen_addr: 0.0.0.0:8080`), but it isn’t. Is that #799?
adam commented 2025-12-29 01:26:51 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Sep 20, 2022):

@db48x what do you get? Are you exposing 443?

@juanfont commented on GitHub (Sep 20, 2022): @db48x what do you get? Are you exposing 443?
adam commented 2025-12-29 01:26:51 +01:00
Author
Owner
Copy Link

@db48x commented on GitHub (Sep 21, 2022):

When I run it, it does not open a listening socket on the listen_addr, so nobody can join. I believe 799 already covers it, so I reverted to the previous release. I can’t use port 443 as it is already in use by another service.

@db48x commented on GitHub (Sep 21, 2022): When I run it, it does not open a listening socket on the `listen_addr`, so nobody can join. I believe 799 already covers it, so I reverted to the previous release. I can’t use port 443 as it is already in use by another service.
adam commented 2025-12-29 01:26:51 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 27, 2022):

Can you try alpha 4?

@kradalby commented on GitHub (Sep 27, 2022): Can you try alpha 4?
adam commented 2025-12-29 01:26:52 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Oct 28, 2022):

Closing this, reopen if it still is relevant

@kradalby commented on GitHub (Oct 28, 2022): Closing this, reopen if it still is relevant
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#318
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?