[Feature] Support tailscale serve with HTTPS #702

@Hypnotist1148 commented on GitHub (Jun 14, 2024):

This would also be the first step to have stuff like funnel working!

@Hypnotist1148 commented on GitHub (Jun 14, 2024): This would also be the first step to have stuff like funnel working!

adam commented

@bentemple commented on GitHub (Aug 1, 2024):

I fully support this. Tried to setup a serve for a client yesterday, as I wanted to use a tailscale sidecar to expose a service, but I can't connect to it except through http. Would be so cool if it would work with https. Especially since I'm using a domain name with hsts so I have no choice but to connect via IP to use HTTP (I'd much prefer magicDNS of course)

@bentemple commented on GitHub (Aug 1, 2024): I fully support this. Tried to setup a serve for a client yesterday, as I wanted to use a tailscale sidecar to expose a service, but I can't connect to it except through http. Would be so cool if it would work with https. Especially since I'm using a domain name with hsts so I have no choice but to connect via IP to use HTTP (I'd much prefer magicDNS of course)

adam commented

@ananthb commented on GitHub (Aug 2, 2024):

I can pitch in code for this as well @teleclimber. Would love to see this feature on Headscale.

@ananthb commented on GitHub (Aug 2, 2024): I can pitch in code for this as well @teleclimber. Would love to see this feature on Headscale.

adam commented

@pavanbuzz commented on GitHub (Aug 8, 2024):

I spent some time going over the Tailscale client code to see what needs to happen.

Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain.

It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https

Additional fact: DNS-01 is the only LetsEncrypt challenge that the tailscale client can solve. See this line.

The following options are ruled out unless Tailscale make changes to their clients:

Using a wildcard certificate is not possible. There is currently no way to tell tailscale serve to use that cert, AFAIK.

Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too.

With that out of the way the only path forwards is to have Headscale implement DNS-01. I know of two approaches to this:

Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and plenty of Go Code for this.

Embed something like acme-dns into headscale.

I'd be interested to know maintainer's thoughts on this at this point. Thanks.

@teleclimber Tracing it further, tailscale already creates the acme challenge record (key,value). It then calls the control plane to SetDNS at #L472.

If we trace this call, noiseClient sends a POST request to the control server at api endpoint /machine/set-dns with NodeKey and Body SetDNSRequest.

So if we handle this endpoint in Headscale by adding a new TXT record into the corresponding provider (like cloudflare, digitaloceans, etc). Then tailscale serve would be able to obtain a TLS certificate.

We need a way to let users configure their dns provider. Traefik uses environment variables/secret files to configure. Also they use a library like lego to handle DNS challenge. But in this case, we just need to add a new record to the provider.

@pavanbuzz commented on GitHub (Aug 8, 2024): > I spent some time going over the Tailscale client code to see what needs to happen. > > Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain. > > It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https > > Additional fact: `DNS-01` is the only LetsEncrypt challenge that the tailscale client can solve. See [this line](https://github.com/tailscale/tailscale/blob/f97d0ac99418256ab4971d002ee807cfdf778453/ipn/ipnlocal/cert.go#L454). > > The following options are ruled out unless Tailscale make changes to their clients: > > * Using a wildcard certificate is not possible. There is currently no way to tell `tailscale serve` to use that cert, AFAIK. > * Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too. > > With that out of the way the only path forwards is to have Headscale implement `DNS-01`. I know of two approaches to this: > > * Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and [plenty of Go Code for this](https://github.com/libdns). > * Embed something like [acme-dns](https://github.com/joohoi/acme-dns) into headscale. > > I'd be interested to know maintainer's thoughts on this at this point. Thanks. @teleclimber Tracing it further, tailscale already creates the acme challenge record (key,value). It then calls the control plane to SetDNS at [#L472](https://github.com/tailscale/tailscale/blob/f97d0ac99418256ab4971d002ee807cfdf778453/ipn/ipnlocal/cert.go#L472). If we trace this call, noiseClient sends a `POST` request to the control server at api endpoint [/machine/set-dns](https://github.com/tailscale/tailscale/blob/1ed958fe231c12890b77025c6b2aa2be0698c7ec/control/controlclient/direct.go#L1488) with NodeKey and Body [SetDNSRequest](https://github.com/tailscale/tailscale/blob/main/tailcfg/tailcfg.go#L2347). So if we handle this endpoint in Headscale by adding a new `TXT` record into the corresponding provider (like cloudflare, digitaloceans, etc). Then `tailscale serve` would be able to obtain a TLS certificate. We need a way to let users configure their dns provider. Traefik uses environment variables/secret files to configure. Also they use a library like [lego](https://github.com/go-acme/lego/) to handle DNS challenge. But in this case, we just need to add a new record to the provider.

adam commented

@ananthb commented on GitHub (Aug 8, 2024):

@pavanbuzz since the new beta release changes Node Magic DNS names to <node>.<your-domain> instead of <node>.<user>.<your-domain>, we could also solve HTTP-01 or TLS-ALPN-01 challenges.

Users can point *. to their headscale instance via DNS.

The advantage being that we don't need to support upstream DNS APIs.

@ananthb commented on GitHub (Aug 8, 2024): @pavanbuzz since the new beta release changes Node Magic DNS names to `<node>.<your-domain>` instead of `<node>.<user>.<your-domain>`, we could also solve HTTP-01 or TLS-ALPN-01 challenges. Users can point *.<your-domain> to their headscale instance via DNS. The advantage being that we don't need to support upstream DNS APIs.

adam commented

@pavanbuzz commented on GitHub (Aug 8, 2024):

@pavanbuzz since the new beta release changes Node Magic DNS names to <node>.<your-domain> instead of <node>.<user>.<your-domain>, we could also solve HTTP-01 or TLS-ALPN-01 challenges.

@ananthb I understand. I think the recent beta changes for Magic DNS to <node>.<your-domain> will be similar to the one tailscale offers <node>.<ts-name>.ts.net. Headscale already handles HTTP-01 or TLS-ALPN-01 challenges to get a TLS certificate for headscale instance.

Users can point *. to their headscale instance via DNS.

The advantage being that we don't need to support upstream DNS APIs.

It's definitely less maintenance. Does it mean then headscale stores all the certificates/keys? How will the node that is requesting a TLS be able to get a certificate then? This approach is similar to caddy-tailscale (Tailscale plugin for caddy). Please correct me, if am mistaken.

Tailscale cli/api requests a TLS certificate on the node where serve/funnel is invoked and saves the certificate/key in that node. In order to get tailscale serve work natively with TLS, Headscale should handle /machine/set-dns endpoint and create a corresponding dns entry in the authoritative dns server for that domain. I believe this is how Tailscale is accomplishing this feature.

This functionality can be extended to accomplish Funnel feature, but it requires atleast one node in the tailnet which can receive public traffic and do a tcp routing based on server name indication. This way TLS traffic is not decrypted by that public facing node.

@pavanbuzz commented on GitHub (Aug 8, 2024): > @pavanbuzz since the new beta release changes Node Magic DNS names to `<node>.<your-domain>` instead of `<node>.<user>.<your-domain>`, we could also solve HTTP-01 or TLS-ALPN-01 challenges. @ananthb I understand. I think the recent beta changes for Magic DNS to `<node>.<your-domain>` will be similar to the one tailscale offers `<node>.<ts-name>.ts.net`. Headscale already handles HTTP-01 or TLS-ALPN-01 challenges to get a `TLS` certificate for headscale instance. > Users can point *. to their headscale instance via DNS. > The advantage being that we don't need to support upstream DNS APIs. It's definitely less maintenance. Does it mean then headscale stores all the certificates/keys? How will the node that is requesting a `TLS` be able to get a certificate then? This approach is similar to [caddy-tailscale](https://github.com/tailscale/caddy-tailscale) (Tailscale plugin for caddy). Please correct me, if am mistaken. Tailscale cli/api requests a `TLS` certificate on the node where `serve/funnel` is invoked and saves the certificate/key in that node. In order to get `tailscale serve` work natively with `TLS`, Headscale should handle [/machine/set-dns](https://github.com/tailscale/tailscale/blob/1ed958fe231c12890b77025c6b2aa2be0698c7ec/control/controlclient/direct.go#L1488) endpoint and create a corresponding dns entry in the authoritative dns server for that domain. I believe this is how Tailscale is accomplishing this feature. This functionality can be extended to accomplish Funnel feature, but it requires atleast one node in the tailnet which can receive public traffic and do a tcp routing based on server name indication. This way `TLS` traffic is not decrypted by that public facing node.

adam commented

@ananthb commented on GitHub (Aug 8, 2024):

@pavanbuzz I get it now. With the DNS challenge, the node requesting the cert can fetch it directly from an ACME issuer.

Letting the node handle its own secret material is infinitely better.

@ananthb commented on GitHub (Aug 8, 2024): @pavanbuzz I get it now. With the DNS challenge, the node requesting the cert can fetch it directly from an ACME issuer. Letting the node handle its own secret material is infinitely better.

adam commented

@ananthb commented on GitHub (Aug 8, 2024):

As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain.

@ananthb commented on GitHub (Aug 8, 2024): As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain.

adam commented

@pavanbuzz commented on GitHub (Aug 9, 2024):

As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain.

@ananthb That is a good idea!

I am leaning towards using other provider for DNS due to several reasons (including but not limited to),

Not too familiar with setting up authoritative DNS server
Require a minimum of 2 NS
Security, performance and maintenance of DNS server

I found that using lego's Challenge.PreSolve will be able to add a TXT record. Since the ts-node is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response.

Because lego has support for multiple dns providers, it allows users to choose any DNS servers (including custom DNS servers). Values (or file reference) can be passed-in via Environment variables. More info can be found in their docs. We can take a similar approach to Traefik's dnsChallenge (although, headscale will not obtain the certificate, but just add DNS record and clean it up later).

@pavanbuzz commented on GitHub (Aug 9, 2024): > As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain. @ananthb That is a good idea! I am leaning towards using other provider for DNS due to several reasons (including but not limited to), * Not too familiar with setting up authoritative DNS server * Require a minimum of 2 NS * Security, performance and maintenance of DNS server I found that using lego's [Challenge.PreSolve](https://pkg.go.dev/github.com/go-acme/lego/v4@v4.17.4/challenge/dns01#Challenge.PreSolve) will be able to add a TXT record. Since the `ts-node` is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response. Because lego has support for multiple dns providers, it allows users to choose any DNS servers (including custom DNS servers). Values (or file reference) can be passed-in via Environment variables. More info can be found in their [docs](https://go-acme.github.io/lego/dns/). We can take a similar approach to [Traefik's dnsChallenge](https://doc.traefik.io/traefik/https/acme/#dnschallenge) (although, headscale will not obtain the certificate, but just add DNS record and clean it up later).

adam commented

@ananthb commented on GitHub (Aug 9, 2024):

Leaning on lego for challenge providers sounds promising.

@ananthb commented on GitHub (Aug 9, 2024): Leaning on lego for challenge providers sounds promising.

adam commented

@pavanbuzz commented on GitHub (Aug 10, 2024):

I found that using lego's Challenge.PreSolve will be able to add a TXT record. Since the ts-node is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response.

I did a bit more analysis and found it shouldn't be Challenge.PreSolve, but instead Provider.Present. However there is an issue using Present method. Based on many provider's implementation, it hashes the value before adding an entry into the dns provider. Tailscale client already sends a hashed value (solved the challenge). So this won't work. There is an issue opened in their repo to allow adding the value without hashing. Until this functionality is added, we cannot use lego to achieve this feature.

Libdns seems promising and can do what we require. But they are still WIP. Would it be okay to add that dependency in headscale?

@ananthb Your idea of DNS server would be better solution as well. Do you have any suggestion for embedded dns? This would require a detailed instruction of how to get headscale to become an authoritative server for a subdomain.

Any idea how do we proceed?

@pavanbuzz commented on GitHub (Aug 10, 2024): > I found that using lego's [Challenge.PreSolve][challenge-pre-solve] will be able to add a TXT record. Since the `ts-node` is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response. I did a bit more analysis and found it shouldn't be [Challenge.PreSolve][challenge-pre-solve], but instead [Provider.Present][provider-present]. However there is an issue using Present method. Based on many provider's implementation, it hashes the value before adding an entry into the dns provider. Tailscale client already sends a hashed value (solved the challenge). So this won't work. There is an [issue](https://github.com/go-acme/lego/issues/720) opened in their repo to allow adding the value without hashing. Until this functionality is added, we cannot use lego to achieve this feature. [Libdns][libdns-repo] seems promising and can do what we require. But they are still WIP. Would it be okay to add that dependency in headscale? @ananthb Your idea of DNS server would be better solution as well. Do you have any suggestion for embedded dns? This would require a detailed instruction of how to get headscale to become an authoritative server for a subdomain. Any idea how do we proceed? [challenge-pre-solve]: https://pkg.go.dev/github.com/go-acme/lego/v4@v4.17.4/challenge/dns01#Challenge.PreSolve [provider-present]: https://github.com/go-acme/lego/blob/master/challenge/provider.go#L6 [libdns-repo]: https://github.com/libdns/libdns

adam commented

@ananthb commented on GitHub (Aug 10, 2024):

It was @teleclimber's idea to embed an authoritative DNS server in headscale. They've even linked to one we can use.

But, the more I think about it, the less this sounds like a good idea. Headscale cannot be hosted in a high availability configuration, so any DNS server hosted by it will suffer from reliability issues.

That Lego issue is moving frustratingly slowly for what should be a straightforward change.

I haven't looked at libdns in depth, but it looks promising.

We should be able to write our own interface similar to the lego one that we are unable to use currently.

I'd strongly recommend against hosting our own DNS. There be dragons.

@ananthb commented on GitHub (Aug 10, 2024): It was @teleclimber's idea to embed an authoritative DNS server in headscale. They've even linked to one we can use. But, the more I think about it, the less this sounds like a good idea. Headscale cannot be hosted in a high availability configuration, so any DNS server hosted by it will suffer from reliability issues. That Lego issue is moving frustratingly slowly for what should be a straightforward change. I haven't looked at libdns in depth, but it looks promising. We should be able to write our own interface similar to the lego one that we are unable to use currently. I'd strongly recommend against hosting our own DNS. There be dragons.

adam commented

@pavanbuzz commented on GitHub (Aug 10, 2024):

@teleclimber I think your idea to use libdns would be the way to go. I can help coding this feature. Kindly let me know if you have started working on this. Would be happy to help.

@pavanbuzz commented on GitHub (Aug 10, 2024): @teleclimber I think your idea to use [libdns](https://github.com/libdns/libdns) would be the way to go. I can help coding this feature. Kindly let me know if you have started working on this. Would be happy to help.

adam commented

@teleclimber commented on GitHub (Aug 11, 2024):

Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments.

One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions

Some DNS providers don't even offer any API access to change DNS records. So going with 3rd party nameservers implies that Headscale users may have to change their domain's nameservers to use one that has an API. Hopefully that's not too high a burden.

Of course, if headscale is the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end.

Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same.

@pavanbuzz I haven't started working on this yet. If we agree that we should go with 3rd party and something like libdns, I would like to hear from maintainers whether they would accept libdns as a dependency. Also I need to dig into libdns and headscale code a lot more.

@teleclimber commented on GitHub (Aug 11, 2024): Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments. One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions Some DNS providers don't even offer any API access to change DNS records. So going with 3rd party nameservers implies that Headscale users may have to change their domain's nameservers to use one that has an API. Hopefully that's not too high a burden. Of course, if headscale *is* the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end. Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same. @pavanbuzz I haven't started working on this yet. If we agree that we should go with 3rd party and something like libdns, I would like to hear from maintainers whether they would accept libdns as a dependency. Also I need to dig into libdns and headscale code a lot more.

adam commented

@pavanbuzz commented on GitHub (Aug 12, 2024):

Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments.

One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions

@teleclimber - Thanks for this info. I didn't know that other providers did not provide granular control. Though Cloudflare provides creation of api token for specific resource (domain). This token can edit DNS records only for this zone (step-6 gives instruction as to how to select this). I am using this setup currently.

But you are right, its a security implication that needs to be carefully considered.

Of course, if headscale is the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end.

I don't think it will be an issue. We could design a solution similar to joohoi/acme-dns, by using a DNS server as an authority server for a subdomain, instead of the main domain. This way, only DNS requests for this subdomain and subdomains of this subdomain will be served. This is achieved by creating a NS record for the subdomain on the main DNS provider along with a A record that points to the DNS server (info provided in dns-records section of joohoi/acme-dns).

Note - We might not be able to use joohoi/acme-dns. It requires a way to add a CNAME redirection (ACME magic) into the main DNS provider for each challenge (goes back to the same issue of security).

Let me try to explain the above logic with an example. If the main domain is example.com & hs.example.com is the subdomain. This new DNS server will become an authoritative server for hs.example.com & *.hs.example.com. If this server is ever compromised, impact is restricted only to the hs.example.com and *.hs.example.com.

Note - I think remediation could also be as easy as users logging into their main DNS provider and disabling the NS & A record for this subdomain.

There are two main hurdles for users that I can think of.

Users have to change their dns_config.base_domain to a subdomain like hs.example.com. So all the hostnames for magic_dns will become myhost.hs.example.com.
Create a NS & A record (one-time setup) in their main DNS provider.

There are few of things for implementing this feature -
We require a dns server that provides a mechanism (API/RPC/etc) to create/update/delete (TXT,CNAME,A,AAAA) records.

Leverage existing DNS servers like CoreDNS with file/redis/grpc plugin. I can't think of any small dns servers other than CoreDNS that we can leverage.
Pros
- Faster feature implementation
- Less code maintenance
- Can be used for both acme_challenge (TXT) & Funnel DNS (CNAME/A/AAAA) records out of the box
- With grpc plugin used by CoreDNS, we could request it to fetch data from Headscale. Headscale could save the details (like TXT, CNAME, A, AAAA) in the DB (sqlite/postgres).
Cons
- Feature would rely explicitly on users to setup the stack properly. Especially if we plan to use CoreDNS along with redis plugin.
- Detailed guide and on-going support to questions related to setting up this stack like the community maintained guides.
Write a micro-dns embedded within Headscale server explicitly for acme_challenges and DNS requests for Funnel.
Pros
- Everything works out of the box without additional setup required from users
Cons
- More code & more maintenance

Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same.

I tried libdns, and its actually pretty easy. Though the custom dns server is more secure, it also means more work.

@teleclimber / @ananthb - let me know what do think. Hope I didn't confuse.

@pavanbuzz commented on GitHub (Aug 12, 2024): > Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments. > > One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions @teleclimber - Thanks for this info. I didn't know that other providers did not provide granular control. Though Cloudflare provides creation of api token for specific resource (domain). This token can edit DNS records only for this zone [(step-6 gives instruction as to how to select this)](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/). I am using this setup currently. But you are right, its a security implication that needs to be carefully considered. > Of course, if headscale _is_ the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end. I don't think it will be an issue. We could design a solution similar to `joohoi/acme-dns`, by using a DNS server as an authority server for a subdomain, instead of the main domain. This way, only DNS requests for this subdomain and subdomains of this subdomain will be served. This is achieved by creating a `NS` record for the subdomain on the main DNS provider along with a `A` record that points to the DNS server ([info provided in dns-records section of `joohoi/acme-dns`](https://github.com/joohoi/acme-dns?tab=readme-ov-file#dns-records)). **Note** - We might not be able to use `joohoi/acme-dns`. It requires a way to add a CNAME redirection (ACME magic) into the main DNS provider for each challenge (goes back to the same issue of security). Let me try to explain the above logic with an example. If the main domain is `example.com` & `hs.example.com` is the subdomain. This new DNS server will become an authoritative server for `hs.example.com` & `*.hs.example.com`. If this server is ever compromised, impact is restricted only to the `hs.example.com` and `*.hs.example.com`. **Note** - I think remediation could also be as easy as users logging into their main DNS provider and disabling the `NS` & `A` record for this subdomain. There are two main hurdles for users that I can think of. 1. Users have to change their `dns_config.base_domain` to a subdomain like `hs.example.com`. So all the hostnames for magic_dns will become `myhost.hs.example.com`. 2. Create a `NS` & `A` record (one-time setup) in their main DNS provider. There are few of things for implementing this feature - We require a dns server that provides a mechanism (API/RPC/etc) to create/update/delete (TXT,CNAME,A,AAAA) records. * Leverage existing DNS servers like CoreDNS with file/redis/grpc plugin. I can't think of any small dns servers other than CoreDNS that we can leverage. Pros * Faster feature implementation * Less code maintenance * Can be used for both acme_challenge (TXT) & Funnel DNS (CNAME/A/AAAA) records out of the box * With grpc plugin used by CoreDNS, we could request it to fetch data from Headscale. Headscale could save the details (like TXT, CNAME, A, AAAA) in the DB (sqlite/postgres). Cons * Feature would rely explicitly on users to setup the stack properly. Especially if we plan to use CoreDNS along with redis plugin. * Detailed guide and on-going support to questions related to setting up this stack like the community maintained guides. * Write a micro-dns embedded within Headscale server explicitly for acme_challenges and DNS requests for Funnel. Pros * Everything works out of the box without additional setup required from users Cons * More code & more maintenance > Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same. I tried libdns, and its actually pretty easy. Though the custom dns server is more secure, it also means more work. @teleclimber / @ananthb - let me know what do think. Hope I didn't confuse.

adam commented

@ananthb commented on GitHub (Aug 12, 2024):

As to the question of DNS zone security, the blast radius is the same whether headscale can manipulate a third-party hosted zone or whether its hosting the zone.

Self-hosting reliable DNS means at least two servers for failover and a whole other can of worms besides.

My vote is resoundingly for third-party DNS server support.

@ananthb commented on GitHub (Aug 12, 2024): As to the question of DNS zone security, the blast radius is the same whether headscale can manipulate a third-party hosted zone or whether its hosting the zone. Self-hosting reliable DNS means at least two servers for failover and a whole other can of worms besides. My vote is resoundingly for third-party DNS server support.

adam commented

@teleclimber commented on GitHub (Aug 13, 2024):

Let me try to explain the above logic with an example. If the main domain is example.com & hs.example.com is the subdomain. This new DNS server will become an authoritative server for hs.example.com & *.hs.example.com. If this server is ever compromised, impact is restricted only to the hs.example.com and *.hs.example.com.

Yes this is how I was imagining we would do things. I may have been too loose with terminology, using "domain" instead of subdomain and zone. Sorry for the confusion.

My vote is resoundingly for third-party DNS server support.

Yes I think that's where I'm at as well.

Note that nothing prevents headscale from supporting other options down the line.

@teleclimber commented on GitHub (Aug 13, 2024): > Let me try to explain the above logic with an example. If the main domain is example.com & hs.example.com is the subdomain. This new DNS server will become an authoritative server for hs.example.com & *.hs.example.com. If this server is ever compromised, impact is restricted only to the hs.example.com and *.hs.example.com. Yes this is how I was imagining we would do things. I may have been too loose with terminology, using "domain" instead of subdomain and zone. Sorry for the confusion. > My vote is resoundingly for third-party DNS server support. Yes I think that's where I'm at as well. Note that nothing prevents headscale from supporting other options down the line.

adam commented

@mitchellkellett commented on GitHub (Aug 14, 2024):

I've been quietly following this in the background. I've previously taken a look at jsiebens/ionscale, and I can see that they are using libdns for their implementation of Serve. Looks like that might be the way to go for now at least.

@mitchellkellett commented on GitHub (Aug 14, 2024): I've been quietly following this in the background. I've previously taken a look at [jsiebens/ionscale](https://github.com/jsiebens/ionscale), and I can see that they are using libdns for their implementation of Serve. Looks like that might be the way to go for now at least.

adam commented

@pavanbuzz commented on GitHub (Aug 16, 2024):

If we all agree with libdns for now, should we involve the maintainers now? We can embed dns (with limited scope for acme challenge and funnel dns response) implemented later.

Sequence diagram with external DNS server using libdns

%%{init: {'sequence': {'rightAngles': true}} }%%

sequenceDiagram
title TLS certificate flow with Cloudflare/Other DNS Provider

participant node as ts-node
participant hs as Headscale Server
participant le as Let's Encrypt
participant dns as Cloudflare DNS Server

node->>+le: AuthorizeOrder with `DNS-01` Challenge
le-->>-node: Challenge value

node->>+hs: /machine/set-dns

hs->>+dns: using libdns to save _acme-challenge.subdomain.example.com

dns-->>-hs: Saved

hs-->>-node: OK

node->>+le: Challenge accepted

le->>+dns: lookup _acme-challenge.subdomain.example.com

dns-->>-le: TXT record

le->>-le: validate response

node->>+le: get status
le-->>-node: challenge verified
node->>+le: CreateOrderCert
le->>-node: Certificate

Sequence diagram with embedded DNS server in Headscale (future implementation - if maintainers are okay)

%%{init: {'sequence': {'rightAngles': true}} }%%

sequenceDiagram
title TLS certificate flow with Headscale as DNS server

participant node as ts-node
participant hs as Headscale Server
participant le as Let's Encrypt

node->>+le: AuthorizeOrder with `DNS-01` Challenge
le-->>-node: Challenge value

node->>+hs: /machine/set-dns
hs->>hs: save _acme-challenge.subdomain.hs.example.com
hs-->>-node: OK


node->>+le: Challenge accepted
critical DNS lookup on port 53/other port if redirected using rinetd

le->>+hs: lookup _acme-challenge.subdomain.hs.example.com

hs->>hs: fetch _acme-challenge.subdomain.hs.example.com from db
hs-->>-le: TXT record

end


le->>-le: validate response

node->>+le: get status
le-->>-node: challenge verified
node->>+le: CreateOrderCert
le-->>-node: Certificate

@pavanbuzz commented on GitHub (Aug 16, 2024): If we all agree with libdns for now, should we involve the maintainers now? We can embed dns (with limited scope for acme challenge and funnel dns response) implemented later. <details> <summary>Sequence diagram with external DNS server using libdns</summary> ```mermaid %%{init: {'sequence': {'rightAngles': true}} }%% sequenceDiagram title TLS certificate flow with Cloudflare/Other DNS Provider participant node as ts-node participant hs as Headscale Server participant le as Let's Encrypt participant dns as Cloudflare DNS Server node->>+le: AuthorizeOrder with `DNS-01` Challenge le-->>-node: Challenge value node->>+hs: /machine/set-dns hs->>+dns: using libdns to save _acme-challenge.subdomain.example.com dns-->>-hs: Saved hs-->>-node: OK node->>+le: Challenge accepted le->>+dns: lookup _acme-challenge.subdomain.example.com dns-->>-le: TXT record le->>-le: validate response node->>+le: get status le-->>-node: challenge verified node->>+le: CreateOrderCert le->>-node: Certificate ``` </details> <details> <summary>Sequence diagram with embedded DNS server in Headscale (future implementation - if maintainers are okay)</summary> ```mermaid %%{init: {'sequence': {'rightAngles': true}} }%% sequenceDiagram title TLS certificate flow with Headscale as DNS server participant node as ts-node participant hs as Headscale Server participant le as Let's Encrypt node->>+le: AuthorizeOrder with `DNS-01` Challenge le-->>-node: Challenge value node->>+hs: /machine/set-dns hs->>hs: save _acme-challenge.subdomain.hs.example.com hs-->>-node: OK node->>+le: Challenge accepted critical DNS lookup on port 53/other port if redirected using rinetd le->>+hs: lookup _acme-challenge.subdomain.hs.example.com hs->>hs: fetch _acme-challenge.subdomain.hs.example.com from db hs-->>-le: TXT record end le->>-le: validate response node->>+le: get status le-->>-node: challenge verified node->>+le: CreateOrderCert le-->>-node: Certificate ``` </details>

adam commented

@teleclimber commented on GitHub (Aug 20, 2024):

Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended.

@teleclimber commented on GitHub (Aug 20, 2024): Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended.

adam commented

@pavanbuzz commented on GitHub (Aug 21, 2024):

Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended.

@teleclimber thanks! This is my first experience with Go. So might take a bit longer, but i will get this up and running.

@pavanbuzz commented on GitHub (Aug 21, 2024): > Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended. @teleclimber thanks! This is my first experience with Go. So might take a bit longer, but i will get this up and running.

adam commented

@pavanbuzz commented on GitHub (Aug 21, 2024):

@juanfont / @kradalby - Our objective is to incorporate the tailscale serve feature into the Headscale server. To achieve this, @teleclimber has proposed two options detailed below:

Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain.

It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https

Additional fact: DNS-01 is the only LetsEncrypt challenge that the tailscale client can solve. See this line.

The following options are ruled out unless Tailscale make changes to their clients:

Using a wildcard certificate is not possible. There is currently no way to tell tailscale serve to use that cert, AFAIK.

Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too.

With that out of the way the only path forwards is to have Headscale implement DNS-01. I know of two approaches to this:

Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and plenty of Go Code for this.

Embed something like acme-dns into headscale.

I'd be interested to know maintainer's thoughts on this at this point. Thanks.

We believe that leveraging libdns would be the optimal approach, given its compatibility with various external DNS providers such as Cloudflare. This choice also sets the stage for the future integration of the Tailscale Funnel feature.

Corresponding sequence diagrams can be found here https://github.com/juanfont/headscale/issues/1921#issuecomment-2293382400 .

We would like to get your opinion so we can move forward with the implementation.

@pavanbuzz commented on GitHub (Aug 21, 2024): @juanfont / @kradalby - Our objective is to incorporate the tailscale serve feature into the Headscale server. To achieve this, @teleclimber has proposed two options detailed below: > Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain. > > It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https > > Additional fact: `DNS-01` is the only LetsEncrypt challenge that the tailscale client can solve. See [this line](https://github.com/tailscale/tailscale/blob/f97d0ac99418256ab4971d002ee807cfdf778453/ipn/ipnlocal/cert.go#L454). > > The following options are ruled out unless Tailscale make changes to their clients: > > * Using a wildcard certificate is not possible. There is currently no way to tell `tailscale serve` to use that cert, AFAIK. > * Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too. > > With that out of the way the only path forwards is to have Headscale implement `DNS-01`. I know of two approaches to this: > > * Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and [plenty of Go Code for this](https://github.com/libdns). > * Embed something like [acme-dns](https://github.com/joohoi/acme-dns) into headscale. > > I'd be interested to know maintainer's thoughts on this at this point. Thanks. We believe that leveraging [libdns](https://github.com/libdns/libdns) would be the optimal approach, given its compatibility with various external DNS providers such as Cloudflare. This choice also sets the stage for the future integration of the Tailscale Funnel feature. Corresponding sequence diagrams can be found here https://github.com/juanfont/headscale/issues/1921#issuecomment-2293382400 . We would like to get your opinion so we can move forward with the implementation.

adam commented

@kradalby commented on GitHub (Aug 21, 2024):

I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it.

I think the work should be split into dns+serve standalone, and then potentially funnel in the future.

My main concern with all user contributed code is outlined in our contribution guidelines.

I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be.

Summarised, it needs to be:

Very well tested (integration mostly for this I would assume)
Well documented, code + docs
Any external dependencies need to be vetted.

@kradalby commented on GitHub (Aug 21, 2024): I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it. I think the work should be split into dns+serve standalone, and then potentially funnel in the future. My main concern with all user contributed code is outlined in our [contribution guidelines](https://github.com/juanfont/headscale/blob/main/CONTRIBUTING.md). I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be. Summarised, it needs to be: - Very well tested (integration mostly for this I would assume) - Well documented, code + docs - Any external dependencies need to be vetted.

adam commented

@pavanbuzz commented on GitHub (Aug 21, 2024):

I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it.

I believe I have an idea on how to achieve this. Though this would require building a separate funnel ingress server like a derp server and should be self-hosted separately by users. Funnel can be dealt later once Serve is implemented.

I think the work should be split into dns+serve standalone, and then potentially funnel in the future.

I don't understand this part. Do you mean separate PRs for dns+serve standalone?

I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be.

I understand the concern. And would stick to the contribution guidelines.

Summarised, it needs to be:

Very well tested (integration mostly for this I would assume)

I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable.

Any external dependencies need to be vetted.

@kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records.

@pavanbuzz commented on GitHub (Aug 21, 2024): > I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it. I believe I have an idea on how to achieve this. Though this would require building a separate funnel ingress server like a derp server and should be self-hosted separately by users. Funnel can be dealt later once Serve is implemented. > I think the work should be split into dns+serve standalone, and then potentially funnel in the future. I don't understand this part. Do you mean separate PRs for dns+serve standalone? > I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be. I understand the concern. And would stick to the contribution guidelines. > Summarised, it needs to be: > > * Very well tested (integration mostly for this I would assume) I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable. > * Any external dependencies need to be vetted. @kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records.

adam commented

@kradalby commented on GitHub (Aug 21, 2024):

I don't understand this part. Do you mean separate PRs for dns+serve standalone?

Do what is needed for serve, and just dont start on funnel, I would be comfortable with giving a thumbs up for serve, but not funnel.

I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable.

Yes, I think the logic of what and how it is set should be tested, but not necessarily the upstream.

@kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records.

libdns looks fine, I think it is the one I looked at last time this came up.

A nice exercise for using libdns would be to replace/add to the current configuration and logic to set up headscale itself with HTTPS, the config is old and yankee and could use some love and nicer configuration.

@kradalby commented on GitHub (Aug 21, 2024): > I don't understand this part. Do you mean separate PRs for dns+serve standalone? Do what is needed for serve, and just dont start on funnel, I would be comfortable with giving a thumbs up for serve, but not funnel. > I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable. Yes, I think the logic of what and how it is set should be tested, but not necessarily the upstream. > @kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records. libdns looks fine, I think it is the one I looked at last time this came up. A nice exercise for using libdns would be to replace/add to the current configuration and logic to set up headscale itself with HTTPS, the config is old and yankee and could use some love and nicer configuration.

adam commented

@ananthb commented on GitHub (Aug 21, 2024):

Funnel definitely needs more from the community than I think we can ask of it/ourselves for now.

I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you.

@ananthb commented on GitHub (Aug 21, 2024): Funnel definitely needs more from the community than I think we can ask of it/ourselves for now. I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you.

adam commented

@pavanbuzz commented on GitHub (Aug 21, 2024):

Funnel definitely needs more from the community than I think we can ask of it/ourselves for now.

I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you.

That would be great @ananthb, lets have a chat to see how we can split the work and get started!

@pavanbuzz commented on GitHub (Aug 21, 2024): > Funnel definitely needs more from the community than I think we can ask of it/ourselves for now. > > I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you. That would be great @ananthb, lets have a chat to see how we can split the work and get started!

adam commented

@ananthb commented on GitHub (Aug 21, 2024):

My email and matrix links are on my GitHub profile.

@ananthb commented on GitHub (Aug 21, 2024): My email and matrix links are on my GitHub profile.

adam commented

@imft-debug commented on GitHub (Sep 20, 2024):

I would also like to contribute on the issue as its quite good feature to serve https servers on opensource headscale server

@imft-debug commented on GitHub (Sep 20, 2024): I would also like to contribute on the issue as its quite good feature to serve https servers on opensource headscale server

adam commented

@ananthb commented on GitHub (Sep 20, 2024):

@pavanbuzz do you want to get started?

@ananthb commented on GitHub (Sep 20, 2024): @pavanbuzz do you want to get started?

adam commented

@xzzpig commented on GitHub (Sep 21, 2024):

Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates?
I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS.
As for the certificate content, the administrator can provide a generic domain name certificate (*.) or multiple specific domain name certificates.
At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates.

@xzzpig commented on GitHub (Sep 21, 2024): Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates? I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS. As for the certificate content, the administrator can provide a generic domain name certificate (*.<my headscale domain>) or multiple specific domain name certificates. At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates.

adam commented

@pavanbuzz commented on GitHub (Sep 21, 2024):

@pavanbuzz do you want to get started?

Hey @ananthb , I will not be work on it for a while as am busy with other work. Please feel free to get started with this feature.

@pavanbuzz commented on GitHub (Sep 21, 2024): > @pavanbuzz do you want to get started? Hey @ananthb , I will not be work on it for a while as am busy with other work. Please feel free to get started with this feature.

adam commented

@madejackson commented on GitHub (Sep 21, 2024):

Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates? I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS. As for the certificate content, the administrator can provide a generic domain name certificate (*.) or multiple specific domain name certificates. At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates.

When you read the earlier comments in this issue, you'll realize that it has been established already how tailscale serve works on the client side: It requests an SSL-Cert via DNS Challenge. From headscales perspective, this is given and cannot be changed. Headscale has no control over the clients functionality so it can only respond to the client's requests accordingly.

For the client to be able to do a DNS-Challenge, it relies on the coordination server (headscale) to enter in the necessary DNS-entries in to the authorative DNS-server. This is the part that has to be implemented into headscale.

@madejackson commented on GitHub (Sep 21, 2024): > Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates? I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS. As for the certificate content, the administrator can provide a generic domain name certificate (*.) or multiple specific domain name certificates. At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates. When you read the earlier comments in this issue, you'll realize that it has been established already how tailscale serve works on the client side: It requests an SSL-Cert via DNS Challenge. From headscales perspective, this is given and cannot be changed. Headscale has no control over the clients functionality so it can only respond to the client's requests accordingly. For the client to be able to do a DNS-Challenge, it relies on the coordination server (headscale) to enter in the necessary DNS-entries in to the authorative DNS-server. This is the part that has to be implemented into headscale.

adam commented

@p3lim commented on GitHub (Nov 12, 2024):

I am not sure how we can test the part where the DNS records are updated.

Could support RFC2136 and set up a minimal BIND9 server to facilitate DNS-01 during the tests, if libdns (if it ends up using that) supports this method.

@p3lim commented on GitHub (Nov 12, 2024): > I am not sure how we can test the part where the DNS records are updated. Could support RFC2136 and set up a minimal BIND9 server to facilitate DNS-01 during the tests, if libdns (if it ends up using that) supports this method.

adam commented

@almereyda commented on GitHub (Nov 12, 2024):

For running a self-hosted DNS API for integration testing, libdns also has a PowerDNS provider, next to its older RFC2136 provider the more up to date DNS UPDATE provider.

This way a PowerDNS container could be spun up and a zone served. Together with a custom ACME endpoint, such as one offered by a Smallstep CA, and its certificate loaded into the test system's trust store, plus a reverse proxy and ACME client (Caddy, Traefik, etc.), it is possible to build a complete DNS-01 ACME test environment. I would be happy to provide example boilerplate for such a setup, when needed.

@almereyda commented on GitHub (Nov 12, 2024): For running a self-hosted DNS API for integration testing, libdns also has a [PowerDNS provider](https://github.com/libdns/powerdns), next to its older [RFC2136 provider](https://github.com/libdns/rfc2136) the more up to date [DNS UPDATE provider](https://github.com/libdns/dnsupdate). This way a PowerDNS container could be spun up and a zone served. Together with a custom ACME endpoint, such as one offered by a Smallstep CA, and its certificate loaded into the test system's trust store, plus a reverse proxy and ACME client (Caddy, Traefik, etc.), it is possible to build a complete DNS-01 ACME test environment. I would be happy to provide example boilerplate for such a setup, when needed.

adam commented

2025-12-29 02:22:35 +01:00

@mitchellkellett commented on GitHub (Nov 13, 2024):

Could take a look at how the folks over at DNSControl do it as well, they integrate with a number of providers and have tests for each one.

@mitchellkellett commented on GitHub (Nov 13, 2024): Could take a look at how the folks over at [DNSControl](https://github.com/StackExchange/dnscontrol) do it as well, they integrate with a number of providers and have tests for each one.

adam commented

@e-zk commented on GitHub (Nov 23, 2024):

Maybe I'm confusing things but I am able to run tailscale serve on my windows machine to expose a local service to my Headscale tailnet and it just works.

Edit: I see now this discussion here is more about serve defaulting to https and how to support that my bad.

@e-zk commented on GitHub (Nov 23, 2024): Maybe I'm confusing things but I am able to run `tailscale serve` on my windows machine to expose a local service to my Headscale tailnet and it just works. Edit: I see now this discussion here is more about serve defaulting to https and how to support that my bad.

adam commented

2025-12-29 02:22:35 +01:00

@imft-debug commented on GitHub (Nov 23, 2024):

@e-zk Tailscale serve works but for headscale the opensource version of it doesnt allow serve to expose internal services to the internet. Try web server to expose the website hosted on interal pods / docker to expose to internet as well

@imft-debug commented on GitHub (Nov 23, 2024): @e-zk Tailscale serve works but for headscale the opensource version of it doesnt allow serve to expose internal services to the internet. Try web server to expose the website hosted on interal pods / docker to expose to internet as well

adam commented

2025-12-29 02:22:35 +01:00

@kradalby commented on GitHub (Nov 25, 2024):

ah, I was under the understanding that serve didnt work, and from what I understand from the last comment, is this issue actually trying to implement Tailscale Funnel? If so thats cool, should we rename this issue and consolidate other questions about funnel?

@kradalby commented on GitHub (Nov 25, 2024): ah, I was under the understanding that `serve` didnt work, and from what I understand from the last comment, is this issue actually trying to implement [Tailscale Funnel](https://tailscale.com/kb/1223/funnel)? If so thats cool, should we rename this issue and consolidate other questions about funnel?

adam commented

@mitchellkellett commented on GitHub (Nov 25, 2024):

Last time I tried to use serve (about a month ago) I had issues and had to use --http <port> flag.

My understanding of this issue is that serve by default will still want certificates for HTTPS, and this will resolve it.

@mitchellkellett commented on GitHub (Nov 25, 2024): Last time I tried to use `serve` (about a month ago) I had issues and had to use `--http <port>` flag. My understanding of this issue is that serve by default will still want certificates for HTTPS, and this will resolve it.

adam commented

@teleclimber commented on GitHub (Nov 25, 2024):

serve works but not over HTTPS. The problem is headscale is not wired to fetch TLS certs, which is what the tailscale CLI expects. This issue is about adding support for automatic TLS cert acquisition.

$ tailscale serve

..doesn't work because by default tailscale tries to use :443 and get a TLS cert. So it fails.

$ tailscale serve --bg --http 80 http://localhost:3003

..works because no cert is being fetched.

Funnel support is not part of this issue, but funnel support will be much easier when this issue is resolved.

@teleclimber commented on GitHub (Nov 25, 2024): serve works but not over HTTPS. The problem is headscale is not wired to fetch TLS certs, which is what the tailscale CLI expects. This issue is about adding support for automatic TLS cert acquisition. ``` $ tailscale serve ``` ..doesn't work because by default tailscale tries to use :443 and get a TLS cert. So it fails. ``` $ tailscale serve --bg --http 80 http://localhost:3003 ``` ..works because no cert is being fetched. Funnel support is not part of this issue, but funnel support will be much easier when this issue is resolved.

adam commented

@korpa commented on GitHub (Dec 16, 2024):

Is anyone working on this feature?

@korpa commented on GitHub (Dec 16, 2024): Is anyone working on this feature?

adam commented

@nqnminh commented on GitHub (Dec 17, 2024):

Is this feature similar to caddy?

@nqnminh commented on GitHub (Dec 17, 2024): Is this feature similar to [caddy](https://tailscale.com/blog/caddy)?

adam commented

@nom3ad commented on GitHub (Dec 17, 2024):

Is anyone working on this feature?

I've been having a working setup on my local fork for quite some time. That codebase is a mess due to personal customizations I made, which doesn't make any sense to be in upstream. I think I even have a setup for "funnel" that's extremely hacky and very close to working.
I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out.
FWIW, my approach to DNS update was to allow headscale to run a command, I did something like follows.

serve:
	enable: true
	set-dns-command: "/path/to/route53-update-script"

@nom3ad commented on GitHub (Dec 17, 2024): > Is anyone working on this feature? I've been having a working setup on my local fork for quite some time. That codebase is a mess due to personal customizations I made, which doesn't make any sense to be in upstream. I think I even have a setup for "funnel" that's extremely hacky and very close to working. I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out. FWIW, my approach to DNS update was to allow headscale to run a command, I did something like follows. ```yaml serve: enable: true set-dns-command: "/path/to/route53-update-script" ```

adam commented

@korpa commented on GitHub (Dec 17, 2024):

... I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out. ...

That would be great

@korpa commented on GitHub (Dec 17, 2024): > > ... I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out. ... > That would be great

adam commented

@ananthb commented on GitHub (Dec 17, 2024):

I'm available to test and to clean up code.

@ananthb commented on GitHub (Dec 17, 2024): I'm available to test and to clean up code.

adam commented

@korpa commented on GitHub (Dec 21, 2024):

I started to look into this and have a POC with all my personal values hardcoded into the code which already works. This helped me to understand, that tailscaled does the heavy-lifting of cert creation. Headscale "just" has to forward the DNS challenge to the DNS provider.

The conclusion from previous comments was to use libdns. But this means very much work to add all DNS providers as every API needs different credentials and config values. With all that knowledge and @nom3ad's solution to use a script I was thinking of an hybrid solution: Use libdns for the 5 to 10 most common DNS providers and as an alternative to provide the script/command solution. Then anyone who uses a not so common DNS provider could create a small binary with libdns or even create a shell script with an API call via curl.

What do you think?

@korpa commented on GitHub (Dec 21, 2024): I started to look into this and have a POC with all my personal values hardcoded into the code which already works. This helped me to understand, that `tailscaled` does the heavy-lifting of cert creation. Headscale "just" has to forward the DNS challenge to the DNS provider. The conclusion from previous comments was to use libdns. But this means very much work to add all DNS providers as every API needs different credentials and config values. With all that knowledge and @nom3ad's solution to use a script I was thinking of an hybrid solution: Use libdns for the 5 to 10 most common DNS providers and as an alternative to provide the script/command solution. Then anyone who uses a not so common DNS provider could create a small binary with libdns or even create a shell script with an API call via curl. What do you think?

adam commented

@nom3ad commented on GitHub (Dec 21, 2024):

FWIW, I have placed a draft PR to showcase how I did it this on my setup.
In my case, it's Route53 and I have a simple AWS CLI wrapper to set DNS.

@nom3ad commented on GitHub (Dec 21, 2024): FWIW, I have placed a draft PR to showcase how I did it this on my setup. In my case, it's Route53 and I have a simple AWS CLI wrapper to set DNS.

adam commented

@korpa commented on GitHub (Dec 21, 2024):

@nom3ad Thank you very much. I had a look at your code. In general my code looks similar except of the exec part.

While cleaning up my code and testing I encountered a few things which are not so easy to cover with the "command" solution:

If anything goes wrong, the TXT record lives forever in DNS and prevents to set a new record. This could be solved that the command removes all TXT records for that entry. But still the TXT record stays in there very long and clutters the zone.
After the record was set it could take a couple of seconds before it can be found by Lets-Encrypt. Therefore we should test every couple of seconds if the record is accessible before returning a success to tailscaled. In case of a timeout or error we have to remove the TXT entry
After everything went ok, we want to remove the TXT record as well to unclutter the zone. As we do not get informed by tailscaled after it got an cert (or ran into an error e.g. Lets-Encrypt rate limit - don't ask how I know :) ), I added a sleep of 30s before removing the record again.

All in all this could be done by a script as well, but it is not straight forward.

@korpa commented on GitHub (Dec 21, 2024): @nom3ad Thank you very much. I had a look at your code. In general my code looks similar except of the `exec` part. While cleaning up my code and testing I encountered a few things which are not so easy to cover with the "command" solution: * If anything goes wrong, the TXT record lives forever in DNS and prevents to set a new record. This could be solved that the command removes all TXT records for that entry. But still the TXT record stays in there very long and clutters the zone. * After the record was set it could take a couple of seconds before it can be found by Lets-Encrypt. Therefore we should test every couple of seconds if the record is accessible before returning a success to `tailscaled`. In case of a timeout or error we have to remove the TXT entry * After everything went ok, we want to remove the TXT record as well to unclutter the zone. As we do not get informed by `tailscaled` after it got an cert (or ran into an error e.g. Lets-Encrypt rate limit - don't ask how I know :) ), I added a sleep of 30s before removing the record again. All in all this could be done by a script as well, but it is not straight forward.

adam commented

@ananthb commented on GitHub (Dec 22, 2024):

The command solution seems brittle to me. I would much rather keep this in Go if at all possible. If headscale starts depending on external commands then this breaks the current model where everything that an installation needs is inside the single binary.

@ananthb commented on GitHub (Dec 22, 2024): The command solution seems brittle to me. I would much rather keep this in Go if at all possible. If headscale starts depending on external commands then this breaks the current model where everything that an installation needs is inside the single binary.

adam commented

@renne commented on GitHub (Dec 23, 2024):

LEGO seems to have the most extensive support for DNS providers (unfortunately I wasn't able to persuade the IETF ACME working group to make an already existing standard like RFC2136 mandatory).
Are the problems with the LEGO-API solved meanwhile?

PowerDNS seems to be a good choice as it is the only authoritative DNS server supported in Proxmox Virtual Environment Software Defined Network zones.

@renne commented on GitHub (Dec 23, 2024): LEGO seems to have the most extensive support for DNS providers (unfortunately I wasn't able to persuade the IETF ACME working group to make an already existing standard like RFC2136 mandatory). Are the problems with the LEGO-API solved meanwhile? PowerDNS seems to be a good choice as it is the only authoritative DNS server supported in Proxmox Virtual Environment Software Defined Network zones.

adam commented

@nblock commented on GitHub (Dec 24, 2024):

The command solution seems brittle to me. I would much rather keep this in Go if at all possible.

DNS setups are site-specific and I doubt that a single library can cope with all kinds of DNS configurations. Some kind of custom command/script support - be it in headscale or in a library itself - may be needed to support such custom setups.

@nblock commented on GitHub (Dec 24, 2024): > The command solution seems brittle to me. I would much rather keep this in Go if at all possible. DNS setups are site-specific and I doubt that a single library can cope with all kinds of DNS configurations. Some kind of custom command/script support - be it in headscale or in a library itself - may be needed to support such custom setups.

adam commented

@ananthb commented on GitHub (Dec 24, 2024):

I personally cannot acquiesce to a solution that is built on the hope that a reliable command with wide ranging DNS configuration support will materialise. Headscale requires very specific configuration to DNS records and I cannot see a universal way to do that on the back of a command runner.

I'm also not sure that a solution involving shelling out to arbitrary commands will be merged.

We agreed on an implementation direction with the maintainer of this project earlier in this same thread. The quick summary is that we planned to implement serve using libdns.

@ananthb commented on GitHub (Dec 24, 2024): I personally cannot acquiesce to a solution that is built on the hope that a reliable command with wide ranging DNS configuration support will materialise. Headscale requires very specific configuration to DNS records and I cannot see a universal way to do that on the back of a command runner. I'm also not sure that a solution involving shelling out to arbitrary commands will be merged. We agreed on an implementation direction with the maintainer of this project earlier in this same thread. The quick summary is that we planned to implement serve using [`libdns`](https://github.com/libdns/libdns).

adam commented

@ananthb commented on GitHub (Dec 24, 2024):

@nom3ad I will work on adapting your PR to use libdns.

@ananthb commented on GitHub (Dec 24, 2024): @nom3ad I will work on adapting your PR to use libdns.

adam commented

@valkum commented on GitHub (Feb 1, 2025):

How about extension binaries e.g. everything called headscale-dns-{provider} that are called with a fixed API. This way, the community can provide their own plugins for the various DNS providers out there.

@valkum commented on GitHub (Feb 1, 2025): How about extension binaries e.g. everything called `headscale-dns-{provider}` that are called with a fixed API. This way, the community can provide their own plugins for the various DNS providers out there.

adam commented

@jack-indaboks commented on GitHub (Feb 25, 2025):

I’ve been following this discussion and was wondering if the idea of an internal CA for Headscale has been considered. Instead of relying on external ACME providers and their challenges, Headscale could potentially generate and manage its own CA for issuing certificates to nodes.

It seems to me that this could simplify certificate management in self-hosted environments where external validation isn’t practical and the headscale server itself is considered a sufficient source of trust within the tailnet.

Has there been any discussion around this approach? I’d love to hear thoughts on its feasibility and potential challenges.

@jack-indaboks commented on GitHub (Feb 25, 2025): I’ve been following this discussion and was wondering if the idea of an internal CA for Headscale has been considered. Instead of relying on external ACME providers and their challenges, Headscale could potentially generate and manage its own CA for issuing certificates to nodes. It seems to me that this could simplify certificate management in self-hosted environments where external validation isn’t practical and the headscale server itself is considered a sufficient source of trust within the tailnet. Has there been any discussion around this approach? I’d love to hear thoughts on its feasibility and potential challenges.

adam commented

@jhnnsrs commented on GitHub (Mar 15, 2025):

Is there an update on this? I would love an tailscale cert integration just so that browsers shut up :D . I would not resort to a self-hosted CA Authority because it would still mean clients need to put the root certificate in their certstore. :D Glad to help out testing everything!

@jhnnsrs commented on GitHub (Mar 15, 2025): Is there an update on this? I would love an tailscale cert integration just so that browsers shut up :D . I would not resort to a self-hosted CA Authority because it would still mean clients need to put the root certificate in their certstore. :D Glad to help out testing everything!

adam commented

@SpiderUnderUrBed commented on GitHub (Mar 16, 2025):

@jhnnsrs there is a PR:
#2312

@SpiderUnderUrBed commented on GitHub (Mar 16, 2025): @jhnnsrs there is a PR: #2312

adam commented

@peskyAdmin commented on GitHub (Apr 7, 2025):

hey all I am surprised no one has commented this yet in the thread. while i would love to see funnel work there are some hurdles (DNS) to be overcame for that to work. there is a simple reliable configuration to do that makes serve though work. atleast for linux and tailscale sidecar containers which is what i cared about, and its really quite simple. i will use a docker as an example but it should work for other clients. this allows a persistent configuration and no need to fiddle with cli.

i create a volume - ./tailscale/config:/config

Create a `config.json` in the tailscale config directory

{
    "TCP": {
      "80": {
        "HTTP": true
      }
    },
    "Web": {
      {{YOUR_TAILNET_DOMAIN}}:80": {
        "Handlers": {
          "/": {
            "Proxy": "http://127.0.0.1:3000"
          }
        }
      }
    }
  }

This config serves port 3000 on port 80 with http. verified with gitea. this works because gitea is using network_mode: service:gitea-ts in the compose file. since gitea is listening on 3000 and using tailscale container network stack this is effective.

to achieve a tailscale funnel type experience i guess you could run a reverse proxy with tailscale sidecare and configure all your routes there to "funnel" the traffic into the tailnet. not sure how you could get the same "magic" of tailscale funnel where everything just works because you need to manage your public DNS records. for me if i want to expose something to the internet. i expose the ports on the host, ie - "3000:3000" and have a reverse proxy handling the traffic. but in that case the traffic is not funneled through tailscale its just being used like you would in the old internet ;).

@peskyAdmin commented on GitHub (Apr 7, 2025): hey all I am surprised no one has commented this yet in the thread. while i would love to see funnel work there are some hurdles (DNS) to be overcame for that to work. there is a simple reliable configuration to do that makes serve though work. atleast for linux and tailscale sidecar containers which is what i cared about, and its really quite simple. i will use a docker as an example but it should work for other clients. this allows a persistent configuration and no need to fiddle with cli. i create a volume ` - ./tailscale/config:/config` ## Create a `config.json` in the tailscale config directory ``` { "TCP": { "80": { "HTTP": true } }, "Web": { {{YOUR_TAILNET_DOMAIN}}:80": { "Handlers": { "/": { "Proxy": "http://127.0.0.1:3000" } } } } } ``` This config serves port 3000 on port 80 with http. verified with gitea. this works because gitea is using `network_mode: service:gitea-ts` in the compose file. since gitea is listening on 3000 and using tailscale container network stack this is effective. to achieve a tailscale funnel type experience i guess you could run a reverse proxy with tailscale sidecare and configure all your routes there to "funnel" the traffic into the tailnet. not sure how you could get the same "magic" of tailscale funnel where everything just works because you need to manage your public DNS records. for me if i want to expose something to the internet. i expose the ports on the host, ie ` - "3000:3000"` and have a reverse proxy handling the traffic. but in that case the traffic is not funneled through tailscale its just being used like you would in the old internet ;).

adam commented

@Daniel15 commented on GitHub (Sep 1, 2025):

One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions

Some DNS providers let you restrict an API key to a particular zone, in which case you could have a separate zone just for Headscale (i.e. example.com and vpn.example.com would be two separate zones rather than having everything in one zone).

In terms of self-hosting, PowerDNS with PowerDNS-Admin supports this.

@Daniel15 commented on GitHub (Sep 1, 2025): > One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions Some DNS providers let you restrict an API key to a particular zone, in which case you could have a separate zone just for Headscale (i.e. `example.com` and `vpn.example.com` would be two separate zones rather than having everything in one zone). In terms of self-hosting, PowerDNS with PowerDNS-Admin supports this.

adam commented