starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] Headscale panic when a registered tailscale node is expired and is trying to bootstrap using an already used or expired preauthkeys #946

New Issue
Closed
opened 2025-12-29 02:26:33 +01:00 by adam · 2 comments
adam commented 2025-12-29 02:26:33 +01:00
Owner
Copy Link

Originally created by @lepichon on GitHub (Feb 18, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

headscale logs with tailscale node bootstrap using an already used authkey

2025-02-17T12:38:06Z ERR home/runner/work/headscale/headscale/hscontrol/noise.go:268 > Error handling registration error="handling register with auth key: http error[401]: authkey already used, %!s(<nil>)"
2025/02/17 12:38:06 http2: panic serving 10.240.1.24:60088: runtime error: invalid memory address or nil pointer dereference
goroutine 73037 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x145
panic({0x1e7d120?, 0x369aae0?})
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0x132
github.com/juanfont/headscale/hscontrol.(*noiseServer).NoiseRegistrationHandler(0xc000982d80, {0x2569ed0, 0xc0005ab0b0}, 0x2110fa0?)
	/home/runner/work/headscale/headscale/hscontrol/noise.go:275 +0x144
net/http.HandlerFunc.ServeHTTP(0x4a4d78?, {0x2569ed0?, 0xc0005ab0b0?}, 0xc0005ab080?)
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29
github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x256b480, 0xc000080418}, 0xc000bee280)
	/home/runner/work/headscale/headscale/hscontrol/metrics.go:89 +0x293
net/http.HandlerFunc.ServeHTTP(0xc000bee140?, {0x256b480?, 0xc000080418?}, 0x7fb5c7d41f18?)
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc001149740, {0x256b480, 0xc000080418}, 0xc0000bc8c0)
	/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
golang.org/x/net/http2.(*serverConn).runHandler(0x44791b?, 0xc0004a4fd0?, 0xae7385?, 0xc001310880?)
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xf5
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 71746
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d

Corresponding tailscale node logs

2025/02/17 12:38:05 control: LoginInteractive -> regen=true
2025/02/17 12:38:05 control: doLogin(regen=true, hasUrl=false)
2025/02/17 12:38:06 control: control server key from https://headscale.xxxxxxxxx.xxx: ts2021=[QyJoF], legacy=
2025/02/17 12:38:06 control: Generating a new nodekey.
2025/02/17 12:38:06 control: RegisterReq: onode= node=[WTvrU] fup=false nks=false
2025/02/17 12:38:06 health(warnable=login-state): error: You are logged out. The last login error was: register request: Post "https://headscale.xxxxxxxxx.xxx/machine/register": stream error: stream ID 53; INTERNAL_ERROR; received from peer
2025/02/17 12:38:06 Received error: register request: Post "https://headscale.xxxxxxxxx.xxx/machine/register": stream error: stream ID 53; INTERNAL_ERROR; received from peer
boot: 2025/02/17 12:38:09 Error waiting for consistent state on shutdown: context deadline exceeded
boot: 2025/02/17 12:38:09 Sending SIGTERM to tailscaled
boot: 2025/02/17 12:38:09 failed to auth tailscale: failed to auth tailscale: tailscale up failed: signal: killed

Same headscale logs with tailscale node bootstrap using an expired authkey

2025-02-18T09:05:09Z ERR home/runner/work/headscale/headscale/hscontrol/noise.go:268 > Error handling registration error="handling register with auth key: http error[401]: authkey expired, %!s(<nil>)"
2025/02/18 09:05:09 http2: panic serving 10.240.1.24:47666: runtime error: invalid memory address or nil pointer dereference
goroutine 879560 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x145
panic({0x1e7d120?, 0x369aae0?})
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0x132
github.com/juanfont/headscale/hscontrol.(*noiseServer).NoiseRegistrationHandler(0xc00143a360, {0x2569ed0, 0xc0004687e0}, 0xc000e3fd88?)
	/home/runner/work/headscale/headscale/hscontrol/noise.go:275 +0x144
net/http.HandlerFunc.ServeHTTP(0xe3fd78?, {0x2569ed0?, 0xc0004687e0?}, 0xc0004687b0?)
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29
github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x256b480, 0xc0004399c0}, 0xc000ebde00)
	/home/runner/work/headscale/headscale/hscontrol/metrics.go:89 +0x293
net/http.HandlerFunc.ServeHTTP(0xc000ebdcc0?, {0x256b480?, 0xc0004399c0?}, 0xc000e3fe30?)
	/nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc001139bc0, {0x256b480, 0xc0004399c0}, 0xc000359040)
	/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
golang.org/x/net/http2.(*serverConn).runHandler(0x44791b?, 0xc000e3ffd0?, 0x8476c5?, 0xc000b19980?)
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xf5
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 878624
	/home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d

Expected Behavior

no panic when tailscale node is trying to bootstrap with an already used or expired authkey

Steps To Reproduce

on the tailscale node kube cluster, removing the self generated node key secret (called headscale in our deployment) and letting only the TS_AUTHKEY secret with an already used or expired authkey is enough to trigger the headscale panic when the node is trying to join the headscale tailnet

kubectl delete -n tailscale secret headscale

Environment

- OS: Kubernetes v1.31.3
- Headscale version: docker.io/headscale/headscale:v0.25.0-debug
- Tailscale version: docker.io/tailscale/tailscale:v1.80.0

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

Headscale configuration

# headscale will look for a configuration file named `config.yaml` (or `config.json`) in the following order:
#
# - `/etc/headscale`
# - `~/.headscale`
# - current working directory

# The url clients will connect to.
# Typically this will be a domain like:
#
# https://myheadscale.example.com:443
#
server_url: https://headscale.xxxxxxxxx.xxx:443

# Address to listen to / bind to on the server
#
# For production:
# listen_addr: 0.0.0.0:8080
listen_addr: 0.0.0.0:8442

# Address to listen to /metrics, you may want
# to keep this endpoint private to your internal
# network
#
metrics_listen_addr: 0.0.0.0:9090

# Address to listen for gRPC.
# gRPC is used for controlling a headscale server
# remotely with the CLI
# Note: Remote access _only_ works if you have
# valid certificates.
#
# For production:
# grpc_listen_addr: 0.0.0.0:50443
grpc_listen_addr: 0.0.0.0:50443

# Allow the gRPC admin interface to run in INSECURE
# mode. This is not recommended as the traffic will
# be unencrypted. Only enable if you know what you
# are doing.
grpc_allow_insecure: false

# The Noise section includes specific configuration for the
# TS2021 Noise protocol
noise:
  # The Noise private key is used to encrypt the
  # traffic between headscale and Tailscale clients when
  # using the new Noise-based protocol.
  private_key_path: /var/lib/headscale/noise_private.key

# List of IP prefixes to allocate tailaddresses from.
# Each prefix consists of either an IPv4 or IPv6 address,
# and the associated prefix length, delimited by a slash.
# It must be within IP ranges supported by the Tailscale
# client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48.
# See below:
# IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
# IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33
# Any other range is NOT supported, and it will cause unexpected issues.
prefixes:
  v6: fd7a:115c:a1e0::/51
  v4: 100.64.0.0/13

  # Strategy used for allocation of IPs to nodes, available options:
  # - sequential (default): assigns the next free IP from the previous given IP.
  # - random: assigns the next free IP from a pseudo-random IP generator (crypto/rand).
  allocation: sequential

# DERP is a relay system that Tailscale uses when a direct
# connection cannot be established.
# https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp
#
# headscale needs a list of DERP servers that can be presented
# to the clients.
derp:
  server:
    # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config
    # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place
    enabled: true

    # Region ID to use for the embedded DERP server.
    # The local DERP prevails if the region ID collides with other region ID coming from
    # the regular DERP config.
    region_id: 999

    # Region code and name are displayed in the Tailscale UI to identify a DERP region
    region_code: "headscale-east-us"
    region_name: "Headscale east-us embedded DERP"

    # Listens over UDP at the configured address for STUN connections - to help with NAT traversal.
    # When the embedded DERP server is enabled stun_listen_addr MUST be defined.
    #
    # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/
    stun_listen_addr: "0.0.0.0:3478"

    # Private key used to encrypt the traffic between headscale DERP
    # and Tailscale clients.
    # The private key file will be autogenerated if it's missing.
    #
    private_key_path: /var/lib/headscale/derp_server_private.key

    # This flag can be used, so the DERP map entry for the embedded DERP server is not written automatically,
    # it enables the creation of your very own DERP map entry using a locally available file with the parameter DERP.paths
    # If you enable the DERP server and set this to false, it is required to add the DERP server to the DERP map using DERP.paths
    automatically_add_embedded_derp_region: true

  paths:
    - /etc/headscale/derp/derp-map.yaml

  # If enabled, a worker will be set up to periodically
  # refresh the given sources and update the derpmap
  # will be set up.
  auto_update_enabled: true

  # How often should we check for DERP updates?
  update_frequency: 24h

# Disables the automatic check for headscale updates on startup
disable_check_updates: true

# Time before an inactive ephemeral node is deleted?
ephemeral_node_inactivity_timeout: 30m

database:
  # Database type. Available options: sqlite, postgres
  # Please note that using Postgres is highly discouraged as it is only supported for legacy reasons.
  # All new development, testing and optimisations are done with SQLite in mind.
  type: sqlite

  # Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".
  debug: false

  # GORM configuration settings.
  gorm:
    # Enable prepared statements.
    prepare_stmt: true

    # Enable parameterized queries.
    parameterized_queries: true

    # Skip logging "record not found" errors.
    skip_err_record_not_found: true

    # Threshold for slow queries in milliseconds.
    slow_threshold: 1000

  # SQLite config
  sqlite:
    path: /var/lib/headscale/db.sqlite

    # Enable WAL mode for SQLite. This is recommended for production environments.
    # https://www.sqlite.org/wal.html
    write_ahead_log: true

### TLS configuration
#
# Even if tailscale is exposed through Haproxy, we keep
# the TLS configuration for private service access on headscale GRPC/REST
## Use already defined certificates:
tls_cert_path: "/etc/certs/headscale/tls.crt"
tls_key_path: "/etc/certs/headscale/tls.key"

log:
  # Output formatting for logs: text or json
  format: json
  level: debug

## Policy
# headscale supports Tailscale's ACL policies.
# Please have a look to their KB to better
# understand the concepts: https://tailscale.com/kb/1018/acls/
policy:
  # The mode can be "file" or "database" that defines
  # where the ACL policies are stored and read from.
  mode: file
  # If the mode is set to "file", the path to a
  # HuJSON file containing ACL policies.
  path: "/etc/headscale/acls/headscale-acl-policies.hjson"

## DNS
#
# If you want stop Headscale from managing the DNS configuration
# all the fields under `dns` should be set to empty values.
dns:
  # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/).
  # Only works if there is at least a nameserver defined.
  magic_dns: true

  # Defines the base domain to create the hostnames for MagicDNS.
  # This domain _must_ be different from the server_url domain.
  # `base_domain` must be a FQDN, without the trailing dot.
  # The FQDN of the hosts will be
  # `hostname.base_domain` (e.g., _myhost.example.com_).
  base_domain: hsnet.xxxxxxxxx.xxx

  # List of DNS servers to expose to clients.
  nameservers:
    global:
      - 1.1.1.1
      - 1.0.0.1
      - 2606:4700:4700::1111
      - 2606:4700:4700::1001

      # NextDNS (see https://tailscale.com/kb/1218/nextdns/).
      # "abc123" is example NextDNS ID, replace with yours.
      # - https://dns.nextdns.io/abc123

    # Split DNS (see https://tailscale.com/kb/1054/dns/),
    # a map of domains and which DNS server to use for each.
    split:
      {}
      # foo.bar.com:
      #   - 1.1.1.1
      # darp.headscale.net:
      #   - 1.1.1.1
      #   - 8.8.8.8

  # Set custom DNS search domains. With MagicDNS enabled,
  # your tailnet base_domain is always the first search domain.
  search_domains: []

# Unix socket used for the CLI to connect without authentication
# Note: for production you will want to set this to something like:
unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

#
# headscale supports experimental OpenID connect support,
# it is still being tested and might have some bugs, please
# help us test it.
# OpenID Connect
oidc:
  only_start_if_oidc_is_available: false
  issuer: "https://dex.xxxxxxxxx.xxx/dex"
  client_id: "headscale"
  client_secret: "XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXx"
  scope: ["openid", "groups", "profile", "email"]

# Logtail configuration
# Logtail is Tailscales logging and auditing infrastructure, it allows the control panel
# to instruct tailscale nodes to log their activity to a remote server.
logtail:
  # Enable logtail for this headscales clients.
  # As there is currently no support for overriding the log server in headscale, this is
  # disabled by default. Enabling this will make your clients send logs to Tailscale Inc.
  enabled: false

# Enabling this option makes devices prefer a random port for WireGuard traffic over the
# default static port 41641. This option is intended as a workaround for some buggy
# firewall devices. See https://tailscale.com/kb/1181/firewalls/ for more information.
randomize_client_port: false
Originally created by @lepichon on GitHub (Feb 18, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior headscale logs with tailscale node bootstrap using an already used authkey ``` 2025-02-17T12:38:06Z ERR home/runner/work/headscale/headscale/hscontrol/noise.go:268 > Error handling registration error="handling register with auth key: http error[401]: authkey already used, %!s(<nil>)" 2025/02/17 12:38:06 http2: panic serving 10.240.1.24:60088: runtime error: invalid memory address or nil pointer dereference goroutine 73037 [running]: golang.org/x/net/http2.(*serverConn).runHandler.func1() /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x145 panic({0x1e7d120?, 0x369aae0?}) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0x132 github.com/juanfont/headscale/hscontrol.(*noiseServer).NoiseRegistrationHandler(0xc000982d80, {0x2569ed0, 0xc0005ab0b0}, 0x2110fa0?) /home/runner/work/headscale/headscale/hscontrol/noise.go:275 +0x144 net/http.HandlerFunc.ServeHTTP(0x4a4d78?, {0x2569ed0?, 0xc0005ab0b0?}, 0xc0005ab080?) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29 github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x256b480, 0xc000080418}, 0xc000bee280) /home/runner/work/headscale/headscale/hscontrol/metrics.go:89 +0x293 net/http.HandlerFunc.ServeHTTP(0xc000bee140?, {0x256b480?, 0xc000080418?}, 0x7fb5c7d41f18?) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29 github.com/gorilla/mux.(*Router).ServeHTTP(0xc001149740, {0x256b480, 0xc000080418}, 0xc0000bc8c0) /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 golang.org/x/net/http2.(*serverConn).runHandler(0x44791b?, 0xc0004a4fd0?, 0xae7385?, 0xc001310880?) /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xf5 created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 71746 /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d ``` Corresponding tailscale node logs ``` 2025/02/17 12:38:05 control: LoginInteractive -> regen=true 2025/02/17 12:38:05 control: doLogin(regen=true, hasUrl=false) 2025/02/17 12:38:06 control: control server key from https://headscale.xxxxxxxxx.xxx: ts2021=[QyJoF], legacy= 2025/02/17 12:38:06 control: Generating a new nodekey. 2025/02/17 12:38:06 control: RegisterReq: onode= node=[WTvrU] fup=false nks=false 2025/02/17 12:38:06 health(warnable=login-state): error: You are logged out. The last login error was: register request: Post "https://headscale.xxxxxxxxx.xxx/machine/register": stream error: stream ID 53; INTERNAL_ERROR; received from peer 2025/02/17 12:38:06 Received error: register request: Post "https://headscale.xxxxxxxxx.xxx/machine/register": stream error: stream ID 53; INTERNAL_ERROR; received from peer boot: 2025/02/17 12:38:09 Error waiting for consistent state on shutdown: context deadline exceeded boot: 2025/02/17 12:38:09 Sending SIGTERM to tailscaled boot: 2025/02/17 12:38:09 failed to auth tailscale: failed to auth tailscale: tailscale up failed: signal: killed ``` Same headscale logs with tailscale node bootstrap using an expired authkey ``` 2025-02-18T09:05:09Z ERR home/runner/work/headscale/headscale/hscontrol/noise.go:268 > Error handling registration error="handling register with auth key: http error[401]: authkey expired, %!s(<nil>)" 2025/02/18 09:05:09 http2: panic serving 10.240.1.24:47666: runtime error: invalid memory address or nil pointer dereference goroutine 879560 [running]: golang.org/x/net/http2.(*serverConn).runHandler.func1() /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2468 +0x145 panic({0x1e7d120?, 0x369aae0?}) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0x132 github.com/juanfont/headscale/hscontrol.(*noiseServer).NoiseRegistrationHandler(0xc00143a360, {0x2569ed0, 0xc0004687e0}, 0xc000e3fd88?) /home/runner/work/headscale/headscale/hscontrol/noise.go:275 +0x144 net/http.HandlerFunc.ServeHTTP(0xe3fd78?, {0x2569ed0?, 0xc0004687e0?}, 0xc0004687b0?) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29 github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x256b480, 0xc0004399c0}, 0xc000ebde00) /home/runner/work/headscale/headscale/hscontrol/metrics.go:89 +0x293 net/http.HandlerFunc.ServeHTTP(0xc000ebdcc0?, {0x256b480?, 0xc0004399c0?}, 0xc000e3fe30?) /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x29 github.com/gorilla/mux.(*Router).ServeHTTP(0xc001139bc0, {0x256b480, 0xc0004399c0}, 0xc000359040) /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2 golang.org/x/net/http2.(*serverConn).runHandler(0x44791b?, 0xc000e3ffd0?, 0x8476c5?, 0xc000b19980?) /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2475 +0xf5 created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 878624 /home/runner/go/pkg/mod/golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d ``` ### Expected Behavior no panic when tailscale node is trying to bootstrap with an already used or expired authkey ### Steps To Reproduce on the tailscale node kube cluster, removing the self generated node key secret (called `headscale` in our deployment) and letting only the `TS_AUTHKEY` secret with an already used or expired authkey is enough to trigger the headscale panic when the node is trying to join the headscale tailnet `kubectl delete -n tailscale secret headscale` ### Environment ```markdown - OS: Kubernetes v1.31.3 - Headscale version: docker.io/headscale/headscale:v0.25.0-debug - Tailscale version: docker.io/tailscale/tailscale:v1.80.0 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Anything else? Headscale configuration ``` # headscale will look for a configuration file named `config.yaml` (or `config.json`) in the following order: # # - `/etc/headscale` # - `~/.headscale` # - current working directory # The url clients will connect to. # Typically this will be a domain like: # # https://myheadscale.example.com:443 # server_url: https://headscale.xxxxxxxxx.xxx:443 # Address to listen to / bind to on the server # # For production: # listen_addr: 0.0.0.0:8080 listen_addr: 0.0.0.0:8442 # Address to listen to /metrics, you may want # to keep this endpoint private to your internal # network # metrics_listen_addr: 0.0.0.0:9090 # Address to listen for gRPC. # gRPC is used for controlling a headscale server # remotely with the CLI # Note: Remote access _only_ works if you have # valid certificates. # # For production: # grpc_listen_addr: 0.0.0.0:50443 grpc_listen_addr: 0.0.0.0:50443 # Allow the gRPC admin interface to run in INSECURE # mode. This is not recommended as the traffic will # be unencrypted. Only enable if you know what you # are doing. grpc_allow_insecure: false # The Noise section includes specific configuration for the # TS2021 Noise protocol noise: # The Noise private key is used to encrypt the # traffic between headscale and Tailscale clients when # using the new Noise-based protocol. private_key_path: /var/lib/headscale/noise_private.key # List of IP prefixes to allocate tailaddresses from. # Each prefix consists of either an IPv4 or IPv6 address, # and the associated prefix length, delimited by a slash. # It must be within IP ranges supported by the Tailscale # client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48. # See below: # IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71 # IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33 # Any other range is NOT supported, and it will cause unexpected issues. prefixes: v6: fd7a:115c:a1e0::/51 v4: 100.64.0.0/13 # Strategy used for allocation of IPs to nodes, available options: # - sequential (default): assigns the next free IP from the previous given IP. # - random: assigns the next free IP from a pseudo-random IP generator (crypto/rand). allocation: sequential # DERP is a relay system that Tailscale uses when a direct # connection cannot be established. # https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp # # headscale needs a list of DERP servers that can be presented # to the clients. derp: server: # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place enabled: true # Region ID to use for the embedded DERP server. # The local DERP prevails if the region ID collides with other region ID coming from # the regular DERP config. region_id: 999 # Region code and name are displayed in the Tailscale UI to identify a DERP region region_code: "headscale-east-us" region_name: "Headscale east-us embedded DERP" # Listens over UDP at the configured address for STUN connections - to help with NAT traversal. # When the embedded DERP server is enabled stun_listen_addr MUST be defined. # # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/ stun_listen_addr: "0.0.0.0:3478" # Private key used to encrypt the traffic between headscale DERP # and Tailscale clients. # The private key file will be autogenerated if it's missing. # private_key_path: /var/lib/headscale/derp_server_private.key # This flag can be used, so the DERP map entry for the embedded DERP server is not written automatically, # it enables the creation of your very own DERP map entry using a locally available file with the parameter DERP.paths # If you enable the DERP server and set this to false, it is required to add the DERP server to the DERP map using DERP.paths automatically_add_embedded_derp_region: true paths: - /etc/headscale/derp/derp-map.yaml # If enabled, a worker will be set up to periodically # refresh the given sources and update the derpmap # will be set up. auto_update_enabled: true # How often should we check for DERP updates? update_frequency: 24h # Disables the automatic check for headscale updates on startup disable_check_updates: true # Time before an inactive ephemeral node is deleted? ephemeral_node_inactivity_timeout: 30m database: # Database type. Available options: sqlite, postgres # Please note that using Postgres is highly discouraged as it is only supported for legacy reasons. # All new development, testing and optimisations are done with SQLite in mind. type: sqlite # Enable debug mode. This setting requires the log.level to be set to "debug" or "trace". debug: false # GORM configuration settings. gorm: # Enable prepared statements. prepare_stmt: true # Enable parameterized queries. parameterized_queries: true # Skip logging "record not found" errors. skip_err_record_not_found: true # Threshold for slow queries in milliseconds. slow_threshold: 1000 # SQLite config sqlite: path: /var/lib/headscale/db.sqlite # Enable WAL mode for SQLite. This is recommended for production environments. # https://www.sqlite.org/wal.html write_ahead_log: true ### TLS configuration # # Even if tailscale is exposed through Haproxy, we keep # the TLS configuration for private service access on headscale GRPC/REST ## Use already defined certificates: tls_cert_path: "/etc/certs/headscale/tls.crt" tls_key_path: "/etc/certs/headscale/tls.key" log: # Output formatting for logs: text or json format: json level: debug ## Policy # headscale supports Tailscale's ACL policies. # Please have a look to their KB to better # understand the concepts: https://tailscale.com/kb/1018/acls/ policy: # The mode can be "file" or "database" that defines # where the ACL policies are stored and read from. mode: file # If the mode is set to "file", the path to a # HuJSON file containing ACL policies. path: "/etc/headscale/acls/headscale-acl-policies.hjson" ## DNS # # If you want stop Headscale from managing the DNS configuration # all the fields under `dns` should be set to empty values. dns: # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/). # Only works if there is at least a nameserver defined. magic_dns: true # Defines the base domain to create the hostnames for MagicDNS. # This domain _must_ be different from the server_url domain. # `base_domain` must be a FQDN, without the trailing dot. # The FQDN of the hosts will be # `hostname.base_domain` (e.g., _myhost.example.com_). base_domain: hsnet.xxxxxxxxx.xxx # List of DNS servers to expose to clients. nameservers: global: - 1.1.1.1 - 1.0.0.1 - 2606:4700:4700::1111 - 2606:4700:4700::1001 # NextDNS (see https://tailscale.com/kb/1218/nextdns/). # "abc123" is example NextDNS ID, replace with yours. # - https://dns.nextdns.io/abc123 # Split DNS (see https://tailscale.com/kb/1054/dns/), # a map of domains and which DNS server to use for each. split: {} # foo.bar.com: # - 1.1.1.1 # darp.headscale.net: # - 1.1.1.1 # - 8.8.8.8 # Set custom DNS search domains. With MagicDNS enabled, # your tailnet base_domain is always the first search domain. search_domains: [] # Unix socket used for the CLI to connect without authentication # Note: for production you will want to set this to something like: unix_socket: /var/run/headscale/headscale.sock unix_socket_permission: "0770" # # headscale supports experimental OpenID connect support, # it is still being tested and might have some bugs, please # help us test it. # OpenID Connect oidc: only_start_if_oidc_is_available: false issuer: "https://dex.xxxxxxxxx.xxx/dex" client_id: "headscale" client_secret: "XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXx" scope: ["openid", "groups", "profile", "email"] # Logtail configuration # Logtail is Tailscales logging and auditing infrastructure, it allows the control panel # to instruct tailscale nodes to log their activity to a remote server. logtail: # Enable logtail for this headscales clients. # As there is currently no support for overriding the log server in headscale, this is # disabled by default. Enabling this will make your clients send logs to Tailscale Inc. enabled: false # Enabling this option makes devices prefer a random port for WireGuard traffic over the # default static port 41641. This option is intended as a workaround for some buggy # firewall devices. See https://tailscale.com/kb/1181/firewalls/ for more information. randomize_client_port: false ```
adam added the bug label 2025-12-29 02:26:33 +01:00
adam closed this issue 2025-12-29 02:26:33 +01:00
adam commented 2025-12-29 02:26:34 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Feb 18, 2025):

Opened #2435, thanks for highlighting, I discovered that the error register responses wasnt wired up as the client expected too, so fixed both.

Please take a look.

@kradalby commented on GitHub (Feb 18, 2025): Opened #2435, thanks for highlighting, I discovered that the error register responses wasnt wired up as the client expected too, so fixed both. Please take a look.
adam commented 2025-12-29 02:26:34 +01:00
Author
Owner
Copy Link

@sjansen1 commented on GitHub (Feb 20, 2025):

We just updated from 0.23 to 0.25. We are using Entra ID for authentication and migration didnt run so well, usernames changed, email_verify does not exist in Entra ID. We just decided to delete all users and client machines and let them register again. In this process we had around 5/50 machines that got "internal server error" from headscale right after OpenID authentication with Entra ID. In Headscale we got "handling register with auth key: http error[401]: authkey already used, %!s(" error in the logs.

We fixed it by deleting all Tailscale data on the the client and let them register again as a new account.

@sjansen1 commented on GitHub (Feb 20, 2025): We just updated from 0.23 to 0.25. We are using Entra ID for authentication and migration didnt run so well, usernames changed, email_verify does not exist in Entra ID. We just decided to delete all users and client machines and let them register again. In this process we had around 5/50 machines that got "internal server error" from headscale right after OpenID authentication with Entra ID. In Headscale we got "handling register with auth key: http error[401]: authkey already used, %!s(<nil>" error in the logs. We fixed it by deleting all Tailscale data on the the client and let them register again as a new account.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#946
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?