tailscale cert + serve tracking #1000

and the documentation about DNS challenge

@Pamplemousse commented on GitHub (Jul 7, 2025):

From the original post

tailscale cert requires headscale to implement /machine/set-dns, which in terms requires headscale to automatically create TXT ACME records "somewhere" for the given domain.

headscale needs to be able to set DNS for the base_domain zone on behalf of clients. This can likely be done with libdns

This challenge asks you to prove that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. [...] After Let’s Encrypt gives your ACME client a token, your client will create a TXT record derived from that token and your account key, and put that record at _acme-challenge.<YOUR_DOMAIN>. Then Let’s Encrypt will query the DNS system for that record. If it finds a match, you can proceed to issue a certificate!

, the proposed solution would only allow to create certificates for "valid" FQDN (my.vpn.example.com), but not for "made-up" domains (my.vpn). Because the domain needs to be reachable from the public DNS system, and not rely on MagicDNS resolution.

Not that it's a hard requirement on my end, but I found it convenient to be able to refer to my machines and services through a explicitly custom domain (I'm gonna miss accessing grafana.potato)...

@Pamplemousse commented on GitHub (Jul 7, 2025): From the original post > tailscale cert requires headscale to implement [/machine/set-dns](https://github.com/tailscale/tailscale/blob/1ed958fe231c12890b77025c6b2aa2be0698c7ec/control/controlclient/direct.go#L1488), which in terms requires headscale to automatically create [TXT ACME records "somewhere"](https://github.com/tailscale/tailscale/blob/f28c8d0ec0b4dbdccd87ee43aa13ce13485dc2b1/ipn/ipnlocal/cert.go#L480) for the given domain. > > headscale needs to be able to set DNS for the base_domain zone on behalf of clients. This can likely be done with [libdns](https://github.com/libdns/libdns) and [the documentation about DNS challenge](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) > This challenge asks you to prove that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. [...] After Let’s Encrypt gives your ACME client a token, your client will create a TXT record derived from that token and your account key, and put that record at _acme-challenge.<YOUR_DOMAIN>. Then Let’s Encrypt will query the DNS system for that record. If it finds a match, you can proceed to issue a certificate! , the proposed solution would only allow to create certificates for "valid" FQDN (`my.vpn.example.com`), but not for "made-up" domains (`my.vpn`). Because the domain needs to be reachable from the public DNS system, and not rely on `MagicDNS` resolution. Not that it's a hard requirement on my end, but I found it convenient to be able to refer to my machines and services through a explicitly custom domain (I'm gonna miss accessing `grafana.potato`)...

adam commented

@Pamplemousse commented on GitHub (Jul 7, 2025):

Also,

Headscale can set base_domain TXT records

seems non trivial: as far as I can tell, it implies that headscale users either have a local DNS server, or use a DNS provider that permit to set DNS records via API.
In either cases, I wonder how headscale would be able to implement such a feature without making too many assumptions about the underlying system 🤔

@Pamplemousse commented on GitHub (Jul 7, 2025): Also, > Headscale can set base_domain TXT records seems non trivial: as far as I can tell, it implies that `headscale` users either have a local DNS server, or use a DNS provider that permit to set DNS records via API. In either cases, I wonder how `headscale` would be able to implement such a feature without making too many assumptions about the underlying system 🤔

adam commented

@1fexd commented on GitHub (Jul 18, 2025):

In either cases, I wonder how headscale would be able to implement such a feature without making too many assumptions about the underlying system 🤔

You'd probably have to implement some sort of adapter that delegates setting the TXT record to an external program (some providers have CLIs) or by sending a (generic) API call to the user's provider

@1fexd commented on GitHub (Jul 18, 2025): > In either cases, I wonder how `headscale` would be able to implement such a feature without making too many assumptions about the underlying system 🤔 You'd probably have to implement some sort of adapter that delegates setting the TXT record to an external program (some providers have CLIs) or by sending a (generic) API call to the user's provider

adam commented

@kotx commented on GitHub (Jul 18, 2025):

You'd probably have to implement some sort of adapter that delegates setting the TXT record to an external program (some providers have CLIs) or by sending a (generic) API call to the user's provider

Caddy's certmagic uses https://github.com/libdns/libdns for DNS-01 challenges. Perhaps this could be used?

@kotx commented on GitHub (Jul 18, 2025): > You'd probably have to implement some sort of adapter that delegates setting the TXT record to an external program (some providers have CLIs) or by sending a (generic) API call to the user's provider Caddy's certmagic uses https://github.com/libdns/libdns for DNS-01 challenges. Perhaps this could be used?

adam commented

@1fexd commented on GitHub (Jul 19, 2025):

I've implemented a PoC using libdns and their Hetzner package in #2696

@1fexd commented on GitHub (Jul 19, 2025): I've implemented a PoC using libdns and their Hetzner package in #2696

adam commented

@rbollampally commented on GitHub (Jul 21, 2025):

Implementing setting of DNS records in hundreds, if not thousands, of DNS providers is not ideal.
I think the best approach for headscale would be to implement a local DNS server which can respond to base_domain and controlplane domain DNS queries.

This https://github.com/siilike/certbot-dns-standalone should solve majority of the above requirements.

The rest of #2137 could also be solved by this PR: https://github.com/juanfont/headscale/pull/2312

From the admin perspective, one only has to now point the NS records for base_domain to be same as controlplane.

@rbollampally commented on GitHub (Jul 21, 2025): Implementing setting of DNS records in hundreds, if not thousands, of DNS providers is not ideal. I think the best approach for headscale would be to implement a local DNS server which can respond to base_domain and controlplane domain DNS queries. This https://github.com/siilike/certbot-dns-standalone should solve majority of the above requirements. The rest of [#2137](https://github.com/juanfont/headscale/issues/2137) could also be solved by this PR: https://github.com/juanfont/headscale/pull/2312 From the admin perspective, one only has to now point the NS records for base_domain to be same as controlplane.

adam commented

@stinovlas commented on GitHub (Jul 21, 2025):

Hi! I'm new to headscale, but I'd like to point out that implementing yet another DNS server is not ideal, especially for people that already host their own DNS server (which is fairly easy with software like knot).

Did you consider implementing RFC 2136? This is an open standard supported by multiple DNS server implementations.

@stinovlas commented on GitHub (Jul 21, 2025): Hi! I'm new to headscale, but I'd like to point out that implementing yet another DNS server is not ideal, especially for people that already host their own DNS server (which is fairly easy with software like `knot`). Did you consider implementing [RFC 2136](https://www.rfc-editor.org/rfc/rfc2136)? This is an open standard supported by multiple DNS server implementations.

adam commented

@rbollampally commented on GitHub (Jul 22, 2025):

Hi! I'm new to headscale, but I'd like to point out that implementing yet another DNS server is not ideal, especially for people that already host their own DNS server (which is fairly easy with software like knot).

Did you consider implementing RFC 2136? This is an open standard supported by multiple DNS server implementations.

You make a good point I completely forgot the selfhost DNS crowd. I have no simple solution. Not every DNS host is RFC 2136 complaint.

@rbollampally commented on GitHub (Jul 22, 2025): > Hi! I'm new to headscale, but I'd like to point out that implementing yet another DNS server is not ideal, especially for people that already host their own DNS server (which is fairly easy with software like `knot`). > > Did you consider implementing [RFC 2136](https://www.rfc-editor.org/rfc/rfc2136)? This is an open standard supported by multiple DNS server implementations. You make a good point I completely forgot the selfhost DNS crowd. I have no simple solution. Not every DNS host is RFC 2136 complaint.

adam commented

@kradalby commented on GitHub (Jul 23, 2025):

Note that I've added a milestone to this, this is to try to provide some sort of idea for people when we think we will work on it, it might change, but at least its the idea.

It would also be meaningful to rework the old (not been touch since initiation) TLS config we have built in at the same time.

For now it is great to get input, which should make the implementation more informed.

Implementing setting of DNS records in hundreds, if not thousands, of DNS providers is not ideal.

This is true, some sort of library like the one mentioned from Caddy (I think libdns) and there are a couple of other candidates.

Did you consider implementing RFC 2136?

I feel that make sense to include, even if we need to sidestep the library for that one.

I've implemented a PoC using libdns and their Hetzner package in https://github.com/juanfont/headscale/pull/2696

Only having had a quick look, I think there are parts here that are useful, but there is way to much that is too Hertzner specific for us to be able to maintain, so hopefully we can find some library that can handle the configuration of the individual provider in a more generic way.

@kradalby commented on GitHub (Jul 23, 2025): Note that I've added a milestone to this, this is to try to provide some sort of idea for people when we think we will work on it, it might change, but at least its the idea. It would also be meaningful to rework the old (not been touch since initiation) TLS config we have built in at the same time. For now it is great to get input, which should make the implementation more informed. > Implementing setting of DNS records in hundreds, if not thousands, of DNS providers is not ideal. This is true, some sort of library like the one mentioned from Caddy (I think libdns) and there are a couple of other candidates. > Did you consider implementing [RFC 2136](https://www.rfc-editor.org/rfc/rfc2136)? I feel that make sense to include, even if we need to sidestep the library for that one. > I've implemented a PoC using libdns and their Hetzner package in https://github.com/juanfont/headscale/pull/2696 Only having had a quick look, I think there are parts here that are useful, but there is way to much that is too Hertzner specific for us to be able to maintain, so hopefully we can find some library that can handle the configuration of the individual provider in a more generic way.

adam commented

@tuarrep commented on GitHub (Aug 5, 2025):

I stumbled upon this quite popular lib which, IMO, fits the needs of this issue
https://github.com/go-acme/lego

@tuarrep commented on GitHub (Aug 5, 2025): I stumbled upon this quite popular lib which, IMO, fits the needs of this issue https://github.com/go-acme/lego

adam commented

@kotx commented on GitHub (Aug 6, 2025):

The client uses the /machine/set-dns endpoint, so an ACME library would likely not work, we need a generic DNS library.

@kotx commented on GitHub (Aug 6, 2025): The client uses the /machine/set-dns endpoint, so an ACME library would likely not work, we need a generic DNS library.

adam commented

@tuarrep commented on GitHub (Aug 7, 2025):

Sorry if don’t get it, but why can’t we use ACME DNS challenge? Is that not what’s required to obtain a certificate?

This is what I do (and how I find this library) to workaround this lack of feature in Headscale.
I have a Nginx proxy in my telnet, which obtains cert with this lib (through Nginx-ui package) using ACME DNS challenge. Even though the domain name resolve a private IP address.

BTW, if the need is only putting a TXT record in a DNS zone, lego supports this for a wide variety of DNS providers.

@tuarrep commented on GitHub (Aug 7, 2025): Sorry if don’t get it, but why can’t we use ACME DNS challenge? Is that not what’s required to obtain a certificate? This is what I do (and how I find this library) to workaround this lack of feature in Headscale. I have a Nginx proxy in my telnet, which obtains cert with this lib (through Nginx-ui package) using ACME DNS challenge. Even though the domain name resolve a private IP address. BTW, if the need is only putting a TXT record in a DNS zone, lego supports this for a wide variety of DNS providers.

adam commented

@mys721tx commented on GitHub (Aug 7, 2025):

So far my set up is to have certbot using one of the DNS plugin to use DNS challenge. As long as Letsencrypt can access the DNS record, there is no need for the tailnet member to be publicly accessible.

If the number of DNS providers is a problem, we should have a standard API and let plugin to handle the DNS update.

@mys721tx commented on GitHub (Aug 7, 2025): So far my set up is to have certbot using one of the DNS plugin to use DNS challenge. As long as Letsencrypt can access the DNS record, there is no need for the tailnet member to be publicly accessible. If the number of DNS providers is a problem, we should have a standard API and let plugin to handle the DNS update.

adam commented

@kotx commented on GitHub (Aug 7, 2025):

Sorry if don’t get it, but why can’t we use ACME DNS challenge? Is that not what’s required to obtain a certificate?

I believe the Tailscale client performs ACME, not the coordination server- the client uses the coordination server to set the DNS record for ACME challenges, so it's a lower-level API. I've only glanced at the client code, so let me know if I'm wrong!

@kotx commented on GitHub (Aug 7, 2025): > Sorry if don’t get it, but why can’t we use ACME DNS challenge? Is that not what’s required to obtain a certificate? I believe the Tailscale client performs ACME, not the coordination server- the client uses the coordination server to set the DNS record for ACME challenges, so it's a lower-level API. I've only glanced at the client code, so let me know if I'm wrong!

adam commented

0012e20e52/challenge/dns01/dns_challenge.go (L74-L76)

@tuarrep commented on GitHub (Aug 13, 2025):

You’re right.
The challenge is generated by client.

But I’m confident we can use Lego as DNS library as there is a whole lot of providers and dns methods for ACME DNS challenge are built in (setting value, waiting for propagation, clean up) and exposed as an abstraction.

0012e20e52/challenge/dns01/dns_challenge.go (L161)

@tuarrep commented on GitHub (Aug 13, 2025): You’re right. The challenge is generated by client. But I’m confident we can use Lego as DNS library as there is a whole lot of providers and dns methods for ACME DNS challenge are built in (setting value, waiting for propagation, clean up) and exposed as an abstraction. https://github.com/go-acme/lego/blob/0012e20e52ada01f1790990d0e3a8a63e4c19aef/challenge/dns01/dns_challenge.go#L74-L76 https://github.com/go-acme/lego/blob/0012e20e52ada01f1790990d0e3a8a63e4c19aef/challenge/dns01/dns_challenge.go#L161

adam commented

@rbollampally commented on GitHub (Aug 13, 2025):

Actually, after getting #2312 to work, I realised that cert-bot is not needed. All I had to do was create a shell script interacting with AWS CLI/Route53 to update my DNS records. So, if Lego solves DNS management, along with #2312 should solve #2137

A few gotchas while getting this to work if anyone wants to try this:

At somepoint, we implemented #2503 which I had to revert
After updating the DNS records, I had to wait few seconds before in shell script. Otherwise, letsencrypt was getting old record; this is not handled in the PR

@rbollampally commented on GitHub (Aug 13, 2025): Actually, after getting #2312 to work, I realised that cert-bot is not needed. All I had to do was create a shell script interacting with AWS CLI/Route53 to update my DNS records. So, if Lego solves DNS management, along with #2312 should solve #2137 A few gotchas while getting this to work if anyone wants to try this: 1. At somepoint, we implemented #2503 which I had to [revert](https://github.com/juanfont/headscale/pull/2503#issuecomment-2915738577) 2. After updating the DNS records, I had to wait few seconds before in shell script. Otherwise, letsencrypt was getting old record; this is not handled in the PR

adam commented

@nom3ad commented on GitHub (Aug 15, 2025):

@rbollampally
Author of #2312 here.

I occasionally rebase that work on top of the latest release, as I am using my private build of headscale to get the serve feature on my tailnets. Since that PR was a POC draft, I never cared to push my updates once it was closed.

I didn't revert #2503, but I think I got the root cause resolved in my last update. I just rebased it on top of main branch force pushed to my PR branch

BTW, In my Cloudflare setup, I never had to put sleep commands to get the DNS record correct. Not sure why you faced that issue. Previously, a route53 setup I tested also worked fine.
Not sure what caused the old record to serve from your Rotue53.

AFAIK, Letsencrypt does full recursive resolution by its own to avoid caching. That means it should directly query the authoritative nameservers (Route53). And I believe Tailscale client would only initiate DNS01 challenge verification after set-dns call is completed, which ensure that route53 is already updated by the script.

@nom3ad commented on GitHub (Aug 15, 2025): @rbollampally Author of #2312 here. I occasionally rebase that work on top of the latest release, as I am using my private build of headscale to get the `serve` feature on my tailnets. Since that PR was a POC draft, I never cared to push my updates once it was closed. I didn't revert #2503, but I think I got the root cause resolved in my last update. I just rebased it on top of main branch force pushed to my [PR branch](https://github.com/nom3ad/headscale/tree/feature/tailscale-serve) BTW, In my Cloudflare setup, I never had to put sleep commands to get the DNS record correct. Not sure why you faced that issue. Previously, a route53 setup I tested also worked fine. Not sure what caused the old record to serve from your Rotue53. AFAIK, Letsencrypt does full recursive resolution by its own to avoid caching. That means it should directly query the authoritative nameservers (Route53). And I believe Tailscale client would only initiate DNS01 challenge verification after set-dns call is completed, which ensure that route53 is already updated by the script.

adam commented

@rbollampally commented on GitHub (Aug 15, 2025):

@nom3ad Thanks for the update. I tried debugging the issues caused by #2503 and got a point that I determined the issue was preventing me from reaching the code you wrote. So, I just reverted it to get going. I'm not a GO programmer. So, don't know what would be the fix.

I again had to do a lot of debugging to figure out that the Route53 record setting was causing the issues. I'm running it directly on an EC2 with role attached to it; which could probably be leading to super fast AWS API. The TTL is not even that long. Maybe we should try and poll letsencrypt for second time within headscale, just in case?

@rbollampally commented on GitHub (Aug 15, 2025): @nom3ad Thanks for the update. I tried debugging the issues caused by #2503 and got a point that I determined the issue was preventing me from reaching the code you wrote. So, I just reverted it to get going. I'm not a GO programmer. So, don't know what would be the fix. I again had to do a lot of debugging to figure out that the Route53 record setting was causing the issues. I'm running it directly on an EC2 with role attached to it; which could probably be leading to super fast AWS API. The TTL is not even that long. Maybe we should try and poll letsencrypt for second time within headscale, just in case?

adam commented

@nom3ad commented on GitHub (Aug 15, 2025):

Yeah, I agree that polling would be a good idea just to make sure records are propagated. I've seen ACME clients implementing polling before initiating DNS01 challenge verification.
IMO, it would have been nice if Tailscale was actually doing it. If not, headscale could do the same.

I'd have to look into logs in my setup to see if I am actually having DNS issues during certificate renewals.

BTW,
Last time I checked, I couldn't find a cleanup api call implementation in Tailscale client code. I was looking for something like unset-dns. That would have been taking care of stale records.
I don't have any better ideas on how to properly do the cleanup - other than some hacks like differed cleanup script execution.
In my setup, I just leave them as they any way get updated on renewals.

@nom3ad commented on GitHub (Aug 15, 2025): Yeah, I agree that polling would be a good idea just to make sure records are propagated. I've seen ACME clients implementing polling before initiating DNS01 challenge verification. IMO, it would have been nice if Tailscale was actually doing it. If not, headscale could do the same. I'd have to look into logs in my setup to see if I am actually having DNS issues during certificate renewals. BTW, Last time I checked, I couldn't find a cleanup api call implementation in Tailscale client code. I was looking for something like `unset-dns`. That would have been taking care of stale records. I don't have any better ideas on how to properly do the cleanup - other than some hacks like differed cleanup script execution. In my setup, I just leave them as they any way get updated on renewals.

adam commented

@xkxx commented on GitHub (Aug 26, 2025):

Hey all,

I'm trying to deploy tsidp which requires HTTPS serve. As such, I'm pretty invested in the progress of this feature. Are you taking contributors in pushing this over the finish line by any chance? Thanks!

@xkxx commented on GitHub (Aug 26, 2025): Hey all, I'm trying to deploy tsidp which requires HTTPS serve. As such, I'm pretty invested in the progress of this feature. Are you taking contributors in pushing this over the finish line by any chance? Thanks!

adam commented

@mattdale77 commented on GitHub (Oct 21, 2025):

I'm just registering my interest so I can get https working with TSDProxy

@mattdale77 commented on GitHub (Oct 21, 2025): I'm just registering my interest so I can get https working with TSDProxy

adam commented

@almereyda commented on GitHub (Oct 28, 2025):

Today upstream presented a possible follow up to this story:

Tailscale Services: Define resources on your tailnet, with granular controls

I don't know how much Headscale would be or is affected by this.

@almereyda commented on GitHub (Oct 28, 2025): Today upstream presented a possible follow up to this story: - [Tailscale Services: Define resources on your tailnet, with granular controls](https://tailscale.com/blog/services-beta) I don't know how much Headscale would be or is affected by this.

adam commented

@evilhamsterman commented on GitHub (Nov 11, 2025):

I have a suggestion. Don't deal with the DNS in the Headscale controller at all, it adds a lot of complexity and maintenance.

I propose that you define a webhook. Then the community can create interfaces for whatever DNS provider and API they want with any language they want.

If you want to provide some webhook provders for common DNS like Route53, Azure, a Zone file, etc. they could be maintained separately as well.

@evilhamsterman commented on GitHub (Nov 11, 2025): I have a suggestion. Don't deal with the DNS in the Headscale controller at all, it adds a lot of complexity and maintenance. I propose that you define a webhook. Then the community can create interfaces for whatever DNS provider and API they want with any language they want. If you want to provide some webhook provders for common DNS like Route53, Azure, a Zone file, etc. they could be maintained separately as well.

adam commented

@timka commented on GitHub (Nov 26, 2025):

Let's assume I have a headscale instance running serving hosts in ovl1.example.com. Now only the 100.100.100.100 MagicDNS resolver knows about the existence of my ovl1.example.com subdomain.

Now imagine I create the ovl1.example.com zone (NS record) and configure my DNS server (coredns) to use its ACME plugin (coredns-acme) to work with Let's Encrypt DNS01 challenge for my headscale subdomain.

Will that make the 100.100.100.100 resolver use my authoritative server for names in my headscale subdomain and break MagicDNS?

UPD: fixed example domain

@timka commented on GitHub (Nov 26, 2025): Let's assume I have a headscale instance running serving hosts in `ovl1.example.com`. Now only the 100.100.100.100 MagicDNS resolver knows about the existence of my `ovl1.example.com` subdomain. Now imagine I create the `ovl1.example.com` zone (NS record) and configure my DNS server (coredns) to use its ACME plugin (coredns-acme) to work with Let's Encrypt DNS01 challenge for my headscale subdomain. Will that make the 100.100.100.100 resolver use my authoritative server for names in my headscale subdomain and break MagicDNS? UPD: fixed example domain

adam commented

@Giggitybyte commented on GitHub (Nov 26, 2025):

@timka I use Caddy with a DNS provider plugin for my domain registrar. Caddy is not accessible from the internet, only through headscale/tailscale (no public DNS entries, only MagicDNS A records), and it's able to get Let's Encrypt certs for my private MagicDNS domains using a DNS01 challenge. There's no need to complicate things with your own DNS server in the middle.

However, with the setup you described, you'd need set override_local_dns to true and to add your DNS server to nameservers.global in the config, and your DNS server would need to be accessible over the internet or at the very least hosted on a tailscale node. Though I'm not sure it'll work smoothly, as I've never needed to set things up like that.

I recommend using Caddy with a DNS provider plugin to get certificates over trying to mess with DNS.

@Giggitybyte commented on GitHub (Nov 26, 2025): @timka I use Caddy with a [DNS provider plugin](https://github.com/caddy-dns/porkbun) for my domain registrar. Caddy is not accessible from the internet, only through headscale/tailscale (no public DNS entries, only MagicDNS A records), and it's able to get Let's Encrypt certs for my private MagicDNS domains using a DNS01 challenge. There's no need to complicate things with your own DNS server in the middle. However, with the setup you described, you'd need set `override_local_dns` to `true` and to add your DNS server to `nameservers.global` in the config, and your DNS server would need to be accessible over the internet or at the very least hosted on a tailscale node. Though I'm not sure it'll work smoothly, as I've never needed to set things up like that. I recommend using Caddy with a DNS provider plugin to get certificates over trying to mess with DNS.

adam commented

@timka commented on GitHub (Nov 27, 2025):

@Giggitybyte Running a DNS server has been usual practice for ages. I already do that and don't see it as complication. So that would be just adding another subdomain to my coredns configuration, plus the ACME plugin.

AFAIU, in your setup Caddy plugin talks to your DNS provider's API which creates public TXT records for Let's Encrypt. And from your message I implicitly assume that your DNS provider's server (e.g. Hetzner, Route53, whatever) is authoritative only for your 2nd level domain (example.com in my case). That is there's no NS record for your MagicDNS subdomain. Is that correct?

@timka commented on GitHub (Nov 27, 2025): @Giggitybyte Running a DNS server has been usual practice for ages. I already do that and don't see it as complication. So that would be just adding another subdomain to my coredns configuration, plus the ACME plugin. AFAIU, in your setup Caddy plugin talks to your DNS provider's API which creates public TXT records for Let's Encrypt. And from your message I implicitly assume that your DNS provider's server (e.g. Hetzner, Route53, whatever) is authoritative only for your 2nd level domain (`example.com` in my case). That is there's no NS record for your MagicDNS subdomain. Is that correct?

adam commented