starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-12 04:10:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

TLS handshake remote error: tls: bad certificate #494

New Issue
Closed
opened 2025-12-29 02:19:05 +01:00 by adam · 10 comments
adam commented 2025-12-29 02:19:05 +01:00
Owner
Copy Link

Originally created by @atb00ker on GitHub (Apr 30, 2023).

Issue description

I am unable to use HTTPS for connecting with headscale with self-signed certificates.

I am seeing the following problem:

sudo headscale serve
2023-04-29T23:13:42Z WRN DERP map is empty, not a single DERP
map datasource was loaded correctly or contained a region
2023-04-29T23:13:42Z INF Setting up a DERPMap update worker frequency=86400000
2023-04-29T23:13:42Z INF Enabling remote gRPC at 127.0.0.1:50443
2023-04-29T23:13:42Z INF listening and serving gRPC on: 127.0.0.1:50443
2023-04-29T23:13:42Z INF listening and serving HTTP on: 0.0.0.0:8080
2023-04-29T23:13:42Z INF listening and serving metrics on: 127.0.0.1:9090

2023/04/29 23:13:52 http: TLS handshake error
 from <IP>:34202: remote error: tls: bad certificate

Where self-signed certificates where generated using command:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem 
-sha256 -days 3650 -nodes 
-subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"

Where I have default config.yaml with following changes:

server_url: https://scale.atb00ker.local:443
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
derp:
  server:
    enabled: enable
    region_id: 999
    region_code: "headscale"
    region_name: "Headscale Embedded DERP"
    stun_listen_addr: "0.0.0.0:3478"
  urls: []
  paths: []
  auto_update_enabled: true
  update_frequency: 24h
tls_cert_path: "/path/cert.pem"
tls_key_path: "/path/key.pem"
log:
  format: text
  level: debug
  • I think but it's a configuration problem, but I think I've read all the documentation I could find on it, and I am unable to solve it.
  • I found some users using tls_client_auth_mode: disabled, but that's doing nothing for me.
  • I tried using it with nginx (reverse proxy); and I still need to register node on HTTP! :-(
  • Everything works if I remove the tls_cert_path and tls_key_path; however, I do want to use HTTPs thru and thru for security reasons.

Could I please get any pointers for solving this?

(Sorry for asking here, cannot find mailing list and unable to join discord! 😓 )

Context info

Please add relevant information about your system. For example:

  • Version of headscale used: v0.22.1
  • Version of tailscale client: 1.40.0
  • OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version: debian 12 (bookworm)
  • Kernel version: 6.1.0-7-amd64
Originally created by @atb00ker on GitHub (Apr 30, 2023). **Issue description** I am unable to use HTTPS for connecting with headscale with self-signed certificates. I am seeing the following problem: ```bash sudo headscale serve 2023-04-29T23:13:42Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region 2023-04-29T23:13:42Z INF Setting up a DERPMap update worker frequency=86400000 2023-04-29T23:13:42Z INF Enabling remote gRPC at 127.0.0.1:50443 2023-04-29T23:13:42Z INF listening and serving gRPC on: 127.0.0.1:50443 2023-04-29T23:13:42Z INF listening and serving HTTP on: 0.0.0.0:8080 2023-04-29T23:13:42Z INF listening and serving metrics on: 127.0.0.1:9090 2023/04/29 23:13:52 http: TLS handshake error from <IP>:34202: remote error: tls: bad certificate ``` Where self-signed certificates where generated using command: ```bash openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname" ``` Where I have default `config.yaml` with following changes: ```yaml server_url: https://scale.atb00ker.local:443 listen_addr: 0.0.0.0:8080 metrics_listen_addr: 127.0.0.1:9090 grpc_listen_addr: 127.0.0.1:50443 derp: server: enabled: enable region_id: 999 region_code: "headscale" region_name: "Headscale Embedded DERP" stun_listen_addr: "0.0.0.0:3478" urls: [] paths: [] auto_update_enabled: true update_frequency: 24h tls_cert_path: "/path/cert.pem" tls_key_path: "/path/key.pem" log: format: text level: debug ``` - I think but it's a configuration problem, but I think I've read all the documentation I could find on it, and I am unable to solve it. - I found some users using `tls_client_auth_mode: disabled`, but that's doing nothing for me. - I tried using it with nginx (reverse proxy); and I still need to register node on HTTP! :-( - Everything works if I remove the `tls_cert_path` and `tls_key_path`; however, I do want to use HTTPs thru and thru for security reasons. Could I please get any pointers for solving this? (Sorry for asking here, cannot find mailing list and unable to join discord! 😓 ) **Context info** Please add relevant information about your system. For example: - Version of headscale used: v0.22.1 - Version of tailscale client: 1.40.0 - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version: debian 12 (bookworm) - Kernel version: 6.1.0-7-amd64
adam added the bug label 2025-12-29 02:19:05 +01:00
adam closed this issue 2025-12-29 02:19:05 +01:00
adam commented 2025-12-29 02:19:05 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Apr 30, 2023):

tls_client_auth_mode was deprecated, and would not have helped here.

If you want to use self-signed certificates, you need to put the cert in the ca-certificates store of the clients, so the cert is trusted by them.

Basically:

  • Copy the cert.pem to /usr/local/share/ca-certificates/headscale.crt or similar.
  • Run update-ca-certificates
@juanfont commented on GitHub (Apr 30, 2023): `tls_client_auth_mode` was deprecated, and would not have helped here. If you want to use self-signed certificates, you need to put the cert in the ca-certificates store of the clients, so the cert is trusted by them. Basically: - Copy the `cert.pem` to `/usr/local/share/ca-certificates/headscale.crt` or similar. - Run `update-ca-certificates`
adam commented 2025-12-29 02:19:05 +01:00
Author
Owner
Copy Link

@loprima-l commented on GitHub (Apr 30, 2023):

A more convenient way to use HTTPS is using Headscale behind a proxy

@loprima-l commented on GitHub (Apr 30, 2023): A more convenient way to use HTTPS is using Headscale behind a proxy
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@atb00ker commented on GitHub (May 4, 2023):

clients, so the cert is trusted by them.

Thanks a ton, it worked.
Closing because the main issue is solved.

HTTPS is using Headscale behind a proxy

@loprima-l
Ah, I have tried a lot to use nginx, and following this doc: https://headscale.net/reverse-proxy/#nginx
it'll be ideal if it works; but I am not able to make it work.

Nginx is forwarding the request to localhost:8080 correctly, but then headscale is silent.
I ran it in debug mode and all I see is:

2023-05-04T21:32:43Z INF ../home/runner/work/headscale/headscale/derp_server.go:92 > DERP region: {RegionID:1 RegionCode:headscale RegionName:Headscale Embedded DERP Avoid:false Nodes:[0x40004dc120]}
2023-05-04T21:32:43Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region
2023-05-04T21:32:43Z INF STUN server started at [::]:3478
2023-05-04T21:32:43Z WRN Listening without TLS but ServerURL does not start with http://
2023-05-04T21:32:43Z INF listening and serving HTTP on: 127.0.0.1:8080
2023-05-04T21:32:43Z INF listening and serving metrics on: 127.0.0.1:9090

headscale serve is running on debug and it outputs nothing.

Here is my full config yaml 
---
server_url: https://headscale.server.test:443
# listen_addr: 0.0.0.0:8080
listen_addr: 127.0.0.1:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false
private_key_path: /opt/headscale/private.key
noise:
  private_key_path: /opt/headscale/noise_private.key
ip_prefixes:
  - fd45:119d:ec64::/48
  - 10.80.0.0/16
derp:
  server:
    enabled: true
    region_id: 1
    region_code: "headscale"
    region_name: "Headscale Embedded DERP"
    stun_listen_addr: "0.0.0.0:3478"
  urls: []
  paths: []
  auto_update_enabled: false
  update_frequency: 24h
disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m
node_update_check_interval: 20s
randomize_client_port: false

# Postgres config
db_type: postgres
db_host: localhost
db_port: 5432
db_name: headscale
db_user: headscale
db_pass: headscale

# TLS configuration
acme_url: https://acme-v02.api.letsencrypt.org/directory
acme_email: ""
tls_letsencrypt_hostname: ""
tls_letsencrypt_cache_dir: /var/lib/headscale/cache
tls_letsencrypt_challenge_type: HTTP-01
tls_letsencrypt_listen: ":http"

# Use already defined certificates:
# tls_cert_path: "/etc/letsencrypt/live/headscale/fullchain.pem"
# tls_key_path: "/etc/letsencrypt/live/headscale/privkey.pem"

log:
  format: text
  level: debug

# DNS
dns_config:
  override_local_dns: true
  nameservers:
    - 1.1.1.1
    - 9.9.9.9
  domains: []
  magic_dns: true
  base_domain: example.com
unix_socket: /var/run/headscale.sock
unix_socket_permission: "0770"
logtail:
  enabled: false
acl_policy_path: ""
Here is my full headscale nginx conf 
map $http_upgrade $connection_upgrade {
    default      keep-alive;
    'websocket'  upgrade;
    ''           close;
}

server {
    listen [::]:443 ssl http2;
    listen 443 ssl http2;

    server_name headscale.server.test;

    # SSL parameters
    ssl_certificate      /etc/letsencrypt/live/headscale/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/headscale/privkey.pem;
    ssl_session_cache shared:SSL:20m;
    ssl_session_timeout 10m;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_dhparam /etc/letsencrypt/live/headscale/dhparam.pem;

    # log files
    access_log /var/log/nginx/headscale.access.log;
    error_log /var/log/nginx/headscale.error.log;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $server_name;
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    }
}

server {
    listen 80;
    server_name headscale.server.test;
    return 301 https://headscale.server.test$request_uri;
}
  • I can guarantee connection is established between client and server.
  • I know nginx is doing proxy_pass to headscale, because I see the blank 404 page when I open the url on browser.
  • But when I do sudo tailscale up --login-server https://headscale.server.test:443, absolutely nothing happens; headscale doesn't even print any debug log.

If I change listen_addr: 0.0.0.0:8080 and do sudo tailscale up --login-server https://headscale.server.test:8080; everything starts to work.

If you see something silly I am doing please let me know, thanks!

@atb00ker commented on GitHub (May 4, 2023): > clients, so the cert is trusted by them. Thanks a ton, it worked. Closing because the main issue is solved. > HTTPS is using Headscale behind a proxy @loprima-l Ah, I have tried a lot to use nginx, and following this doc: https://headscale.net/reverse-proxy/#nginx it'll be ideal if it works; but I am not able to make it work. Nginx is forwarding the request to `localhost:8080` correctly, but then `headscale` is silent. I ran it in debug mode and all I see is: ``` 2023-05-04T21:32:43Z INF ../home/runner/work/headscale/headscale/derp_server.go:92 > DERP region: {RegionID:1 RegionCode:headscale RegionName:Headscale Embedded DERP Avoid:false Nodes:[0x40004dc120]} 2023-05-04T21:32:43Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region 2023-05-04T21:32:43Z INF STUN server started at [::]:3478 2023-05-04T21:32:43Z WRN Listening without TLS but ServerURL does not start with http:// 2023-05-04T21:32:43Z INF listening and serving HTTP on: 127.0.0.1:8080 2023-05-04T21:32:43Z INF listening and serving metrics on: 127.0.0.1:9090 ``` `headscale serve` is running on debug and it outputs nothing. <details> <summary>Here is my full config yaml</summary> ```yaml --- server_url: https://headscale.server.test:443 # listen_addr: 0.0.0.0:8080 listen_addr: 127.0.0.1:8080 metrics_listen_addr: 127.0.0.1:9090 grpc_listen_addr: 127.0.0.1:50443 grpc_allow_insecure: false private_key_path: /opt/headscale/private.key noise: private_key_path: /opt/headscale/noise_private.key ip_prefixes: - fd45:119d:ec64::/48 - 10.80.0.0/16 derp: server: enabled: true region_id: 1 region_code: "headscale" region_name: "Headscale Embedded DERP" stun_listen_addr: "0.0.0.0:3478" urls: [] paths: [] auto_update_enabled: false update_frequency: 24h disable_check_updates: false ephemeral_node_inactivity_timeout: 30m node_update_check_interval: 20s randomize_client_port: false # Postgres config db_type: postgres db_host: localhost db_port: 5432 db_name: headscale db_user: headscale db_pass: headscale # TLS configuration acme_url: https://acme-v02.api.letsencrypt.org/directory acme_email: "" tls_letsencrypt_hostname: "" tls_letsencrypt_cache_dir: /var/lib/headscale/cache tls_letsencrypt_challenge_type: HTTP-01 tls_letsencrypt_listen: ":http" # Use already defined certificates: # tls_cert_path: "/etc/letsencrypt/live/headscale/fullchain.pem" # tls_key_path: "/etc/letsencrypt/live/headscale/privkey.pem" log: format: text level: debug # DNS dns_config: override_local_dns: true nameservers: - 1.1.1.1 - 9.9.9.9 domains: [] magic_dns: true base_domain: example.com unix_socket: /var/run/headscale.sock unix_socket_permission: "0770" logtail: enabled: false acl_policy_path: "" ``` </details> <details> <summary>Here is my full headscale nginx conf</summary> ```conf map $http_upgrade $connection_upgrade { default keep-alive; 'websocket' upgrade; '' close; } server { listen [::]:443 ssl http2; listen 443 ssl http2; server_name headscale.server.test; # SSL parameters ssl_certificate /etc/letsencrypt/live/headscale/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/headscale/privkey.pem; ssl_session_cache shared:SSL:20m; ssl_session_timeout 10m; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; ssl_dhparam /etc/letsencrypt/live/headscale/dhparam.pem; # log files access_log /var/log/nginx/headscale.access.log; error_log /var/log/nginx/headscale.error.log; location / { proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header Host $server_name; proxy_redirect http:// https://; proxy_buffering off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto; add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always; } } server { listen 80; server_name headscale.server.test; return 301 https://headscale.server.test$request_uri; } ``` </details> - I can guarantee connection is established between client and server. - I know nginx is doing `proxy_pass` to `headscale`, because I see the blank 404 page when I open the url on browser. - But when I do `sudo tailscale up --login-server https://headscale.server.test:443`, absolutely nothing happens; headscale doesn't even print any debug log. If I change ` listen_addr: 0.0.0.0:8080` and do `sudo tailscale up --login-server https://headscale.server.test:8080`; everything starts to work. If you see something silly I am doing please let me know, thanks!
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@loprima-l commented on GitHub (May 6, 2023):

I can't check it know but when I came back on Monday, I'll get my eyes on it !

@loprima-l commented on GitHub (May 6, 2023): I can't check it know but when I came back on Monday, I'll get my eyes on it !
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@atb00ker commented on GitHub (May 7, 2023):

@juanfont While the solution worked on Linux boxes; it's not working on Android.
I added the certificate in "User CAs" on Android and verified it's working using Firefox.

Yet, it seems, the Tailscale app ignores the user added CA and only checks default CAs and the handshake fails again! 😭

@atb00ker commented on GitHub (May 7, 2023): @juanfont While the solution worked on Linux boxes; it's not working on Android. I added the certificate in "User CAs" on Android and verified it's working using Firefox. Yet, it seems, the Tailscale app ignores the user added CA and only checks default CAs and the handshake fails again! 😭
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (May 7, 2023):

Then I am afraid the issue is not in Headscale side, but on the Tailscale client.

That being said, you can easily get a signed certificate from Let's Encrypt if you own a domain.

@juanfont commented on GitHub (May 7, 2023): Then I am afraid the issue is not in Headscale side, but on the Tailscale client. That being said, you can easily get a signed certificate from Let's Encrypt if you own a domain.
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@Pr0pHesyer commented on GitHub (Dec 24, 2023):

Android & Linux have the same question with letsencrypt certificate

@Pr0pHesyer commented on GitHub (Dec 24, 2023): Android & Linux have the same question with letsencrypt certificate
adam commented 2025-12-29 02:19:06 +01:00
Author
Owner
Copy Link

@Pr0pHesyer commented on GitHub (Dec 29, 2023):

I see,it was caused by my cert file.
Working after using chain.pem/fullchain.pem instead of cert.pem

@Pr0pHesyer commented on GitHub (Dec 29, 2023): I see,it was caused by my cert file. Working after using chain.pem/fullchain.pem instead of cert.pem
adam commented 2025-12-29 02:19:07 +01:00
Author
Owner
Copy Link

@zeusraman commented on GitHub (Jun 17, 2024):

hi all
I am in the same boat today .. can we use TLS with certs in config file without nginx proxy ?
Would someone be kind enough to let me know what config i need to use ?
i have tried using this
tls_cert_path: "/path/cert.pem"
tls_key_path: "/path/key.pem"

and server as https://mydomain.com:8081

cheers all

@zeusraman commented on GitHub (Jun 17, 2024): hi all I am in the same boat today .. can we use TLS with certs in config file without nginx proxy ? Would someone be kind enough to let me know what config i need to use ? i have tried using this tls_cert_path: "/path/cert.pem" tls_key_path: "/path/key.pem" and server as https://mydomain.com:8081 cheers all
adam commented 2025-12-29 02:19:07 +01:00
Author
Owner
Copy Link

@Pr0pHesyer commented on GitHub (Jun 17, 2024):

Use chain.pem/fullchain.pem instead of cert.pem

@Pr0pHesyer commented on GitHub (Jun 17, 2024): Use chain.pem/fullchain.pem instead of cert.pem
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#494
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?