starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-13 12:50:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] DERP: derpport is ignored #1126

New Issue
Closed
opened 2025-12-29 02:28:25 +01:00 by adam · 9 comments
adam commented 2025-12-29 02:28:25 +01:00
Owner
Copy Link

Originally created by @parasiteoflife on GitHub (Oct 29, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I'm using

regions:
  900:
    regionid: 900
    regioncode: tz
    regionname: test
    nodes:
      - name: 900a
        regionid: 900
        hostname: https://derp.domain.com
        ipv4: xxx.xxx.xxx.xxx
        ipv6: none
        stunport: 0
        stunonly: false
        derpport: 30000
        canport80: false

but Headscale simply ignores that and keeps using 443
the cached map from a client shows it:

        "900": {
            "RegionID": 900,
            "RegionCode": "tz",
            "RegionName": "test",
            "Nodes": [
                {
                    "Name": "900",
                    "RegionID": 900,
                    "HostName": "headscale.domain.com",
                    "STUNPort": 3478,
                    "DERPPort": 443
                }
            ]
        }

It also seems to be ignoring ipv4 and ipv6

Expected Behavior

DERP uses the specified port and ips

Steps To Reproduce

  1. Configure derpport
  2. Restart headscale container
  3. logs will print that DERP is using 443

Environment

- OS: Almalinux8
- Headscale version: 0.27.0

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

headscale  | 2025-10-29T05:26:12Z INF home/runner/work/headscale/headscale/hscontrol/derp/server/derp_server.go:138 > DERP region: {RegionID:900 RegionCode:tz RegionName:test Latitude:0 Longitude:0 Avoid:false NoMeasureNoHome:false Nodes:[0xc0005c3050]}
headscale  | 2025-10-29T05:26:12Z INF home/runner/work/headscale/headscale/hscontrol/derp/server/derp_server.go:139 > DERP Nodes[0]: &{Name:900 RegionID:900 HostName:headscale.domain.com CertName: IPv4: IPv6: STUNPort:3478 STUNOnly:false DERPPort:443 InsecureForTests:false STUNTestIP: CanPort80:false}
Originally created by @parasiteoflife on GitHub (Oct 29, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior I'm using ```yaml regions: 900: regionid: 900 regioncode: tz regionname: test nodes: - name: 900a regionid: 900 hostname: https://derp.domain.com ipv4: xxx.xxx.xxx.xxx ipv6: none stunport: 0 stunonly: false derpport: 30000 canport80: false ``` but Headscale simply ignores that and keeps using 443 the cached map from a client shows it: ```json "900": { "RegionID": 900, "RegionCode": "tz", "RegionName": "test", "Nodes": [ { "Name": "900", "RegionID": 900, "HostName": "headscale.domain.com", "STUNPort": 3478, "DERPPort": 443 } ] } ``` It also seems to be ignoring ipv4 and ipv6 ### Expected Behavior DERP uses the specified port and ips ### Steps To Reproduce 1. Configure `derpport` 2. Restart headscale container 3. logs will print that DERP is using 443 ### Environment ```markdown - OS: Almalinux8 - Headscale version: 0.27.0 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information ``` headscale | 2025-10-29T05:26:12Z INF home/runner/work/headscale/headscale/hscontrol/derp/server/derp_server.go:138 > DERP region: {RegionID:900 RegionCode:tz RegionName:test Latitude:0 Longitude:0 Avoid:false NoMeasureNoHome:false Nodes:[0xc0005c3050]} headscale | 2025-10-29T05:26:12Z INF home/runner/work/headscale/headscale/hscontrol/derp/server/derp_server.go:139 > DERP Nodes[0]: &{Name:900 RegionID:900 HostName:headscale.domain.com CertName: IPv4: IPv6: STUNPort:3478 STUNOnly:false DERPPort:443 InsecureForTests:false STUNTestIP: CanPort80:false} ```
adam added the questionbugDERP labels 2025-12-29 02:28:25 +01:00
adam closed this issue 2025-12-29 02:28:25 +01:00
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@ArcticLampyrid commented on GitHub (Oct 30, 2025):

Is this a regression problem? I am still using v0.26 and have not encountered these issues.

@ArcticLampyrid commented on GitHub (Oct 30, 2025): Is this a regression problem? I am still using v0.26 and have not encountered these issues.
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Oct 31, 2025):

I can't reproduce this on 0.27.0; tested an identical config with 0.26.1 and 0.27.0 and there's no difference for tailscale debug derp-map. Setup according to our DERP docs.

Relevant part of config.yaml

derp:
  server:
    enabled: true
    ipv4: 198.51.100.1
    ipv6: 2001:db8::1
  urls: []
  paths:
    - /path/to/derp.yaml

And derp.yaml:

regions:
  900:
    regionid: 900
    regioncode: tz
    regionname: test
    nodes:
      - name: 900a
        regionid: 900
        hostname: https://derp.example.com
        ipv4: 198.51.100.2
        ipv6: null
        stunport: 0
        stunonly: false
        derpport: 30000
        canport80: false

Result of tailscale debug derp-map (identical for 0.26.1 and 0.27.0) - this looks OK to me:

{
  "Regions": {
    "900": {
      "RegionID": 900,
      "RegionCode": "tz",
      "RegionName": "test",
      "Nodes": [
        {
          "Name": "900a",
          "RegionID": 900,
          "HostName": "https://derp.example.com",
          "IPv4": "198.51.100.2",
          "DERPPort": 30000
        }
      ]
    },
    "999": {
      "RegionID": 999,
      "RegionCode": "headscale",
      "RegionName": "Headscale Embedded DERP",
      "Nodes": [
        {
          "Name": "999",
          "RegionID": 999,
          "HostName": "derp.internal",
          "IPv4": "198.51.100.1",
          "IPv6": "2001:db8::1",
          "STUNPort": 3478,
          "DERPPort": 8443
        }
      ]
    }
  }
}

EDIT: removed wrong comment about disabling IPv6.

Please check again - and if this still happens to you please provide:

  • Headscale configuration (config.yaml and derp.yaml)
  • Output of tailscale debug derp-map, preferably for 0.26.1 and 0.27.0
  • Output of tailscale debug derp $NAME for each DERP
@nblock commented on GitHub (Oct 31, 2025): I can't reproduce this on 0.27.0; tested an identical config with 0.26.1 and 0.27.0 and there's no difference for `tailscale debug derp-map`. Setup according to our [DERP docs](https://headscale.net/development/ref/derp/#customize-derp-map). Relevant part of `config.yaml` ```yaml derp: server: enabled: true ipv4: 198.51.100.1 ipv6: 2001:db8::1 urls: [] paths: - /path/to/derp.yaml ``` And `derp.yaml`: ```yaml regions: 900: regionid: 900 regioncode: tz regionname: test nodes: - name: 900a regionid: 900 hostname: https://derp.example.com ipv4: 198.51.100.2 ipv6: null stunport: 0 stunonly: false derpport: 30000 canport80: false ``` Result of `tailscale debug derp-map` (identical for 0.26.1 and 0.27.0) - this looks OK to me: ```json { "Regions": { "900": { "RegionID": 900, "RegionCode": "tz", "RegionName": "test", "Nodes": [ { "Name": "900a", "RegionID": 900, "HostName": "https://derp.example.com", "IPv4": "198.51.100.2", "DERPPort": 30000 } ] }, "999": { "RegionID": 999, "RegionCode": "headscale", "RegionName": "Headscale Embedded DERP", "Nodes": [ { "Name": "999", "RegionID": 999, "HostName": "derp.internal", "IPv4": "198.51.100.1", "IPv6": "2001:db8::1", "STUNPort": 3478, "DERPPort": 8443 } ] } } } ``` EDIT: removed wrong comment about disabling IPv6. --- Please check again - and if this still happens to you please provide: - Headscale configuration (`config.yaml` and `derp.yaml`) - Output of `tailscale debug derp-map`, preferably for 0.26.1 and 0.27.0 - Output of `tailscale debug derp $NAME` for each DERP
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@ArcticLampyrid commented on GitHub (Oct 31, 2025):

If you want to disable IPv6, you'd have to use ipv6: null.

No, it's official documented way to set ipv6 to string "none".

061e6266cf/tailcfg/derpmap.go (L170-L182)

@ArcticLampyrid commented on GitHub (Oct 31, 2025): > If you want to disable IPv6, you'd have to use ipv6: null. No, it's official documented way to set ipv6 to string "none". https://github.com/tailscale/tailscale/blob/061e6266cf4e9c9a0f06b0d60d4d7840f6b7678d/tailcfg/derpmap.go#L170-L182
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@ArcticLampyrid commented on GitHub (Oct 31, 2025):

If you set it to null, it means that IPv6 addresses should be looked up from DNS.

@ArcticLampyrid commented on GitHub (Oct 31, 2025): If you set it to null, it means that IPv6 addresses should be looked up from DNS.
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Oct 31, 2025):

If you set it to null, it means that IPv6 addresses should be looked up from DNS.

Ah, I wasn't aware of this. Thx!

@nblock commented on GitHub (Oct 31, 2025): > If you set it to null, it means that IPv6 addresses should be looked up from DNS. Ah, I wasn't aware of this. Thx!
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@parasiteoflife commented on GitHub (Nov 5, 2025):

ok I'm doing something wrong because I never got that Headscale Embedded DERP

config.yml

---

server_url: https://headscale.domain.com

listen_addr: 0.0.0.0:8080

metrics_listen_addr: 127.0.0.1:9090

grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v6: fd7a:115c:a1e0::/48
  v4: 100.64.0.0/10
  allocation: sequential

derp:
  server:
    enabled: true

    region_id: 900
    region_code: "tz"
    region_name: "test"

    stun_listen_addr: "0.0.0.0:3478"
    private_key_path: /var/lib/headscale/derp_server_private.key
    automatically_add_embedded_derp_region: true

    ipv4: xxx.xxx.xxx.xxx
    # ipv6:

    verify_clients: true

  urls:
    - https://controlplane.tailscale.com/derpmap/default

  paths:
    - /etc/headscale/derp.yaml

  auto_update_enabled: true
  update_frequency: 3h

disable_check_updates: false

ephemeral_node_inactivity_timeout: 30m

database:
  type: sqlite
  debug: false
  gorm:
    prepare_stmt: true
    parameterized_queries: true
    skip_err_record_not_found: true
    slow_threshold: 1000
  sqlite:
    path: /etc/headscale/db.sqlite
    write_ahead_log: true
    wal_autocheckpoint: 1000
tls_cert_path: ""
tls_key_path: ""

log:
  format: text
  level: info

policy:
  mode: database
  path: ""

dns:
  magic_dns: false
  base_domain: tailscale.arpa
  override_local_dns: false
  nameservers:
    global:
      - 100.64.0.2
      - 100.64.0.3
      - 100.64.0.5
    split: {}
  search_domains: []
  extra_records: []
unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"


logtail:
  enabled: false

randomize_client_port: false

derp.yml

regions:
  900:
    regionid: 900
    regioncode: tz
    regionname: test
    nodes:
      - name: 900a
        regionid: 900
        hostname: https://derp.domain.com
        ipv4: xxx.xxx.xxx.xxx
        ipv6: none
        stunport: 0
        stunonly: false
        derpport: 30000
        canport80: false

tailscale debug derp-map (I removed all the other tailscale regions)

{
    "Regions": {
        "900": {
            "RegionID": 900,
            "RegionCode": "tz",
            "RegionName": "test",
            "Nodes": [
                {
                    "Name": "900",
                    "RegionID": 900,
                    "HostName": "headscale.domain.com",
                    "IPv4": "xxx.xxx.xxx.xxx",
                    "STUNPort": 3478,
                    "DERPPort": 443
                }
            ]
        }
    }
}

tailscale debug derp 900

{
 "Info": [
  "Region 900 == \"tz\"",
  "Successfully established a DERP connection with node \"headscale.domain.com\"",
  "Node \"headscale.domain.com\" returned IPv4 STUN response: xxx.xxx.xxx.xxx:34636"
 ],
 "Warnings": null,
 "Errors": [
  "Error connecting to node \"headscale.domain.com\" @ \"headscale.domain.com:443\" over IPv6: dial tcp6: lookup headscale.domain.com on 1.1.1.1:53: no such host"
 ]
}

Also which ports I should be exposing? I'm only exposing 50443 and 3478 (and 30000) but there appears port 34636 which I don't get where it comes from, also in the config.yml comments it says that 41641 is also used?

@parasiteoflife commented on GitHub (Nov 5, 2025): ok I'm doing something wrong because I never got that `Headscale Embedded DERP` config.yml ```yaml --- server_url: https://headscale.domain.com listen_addr: 0.0.0.0:8080 metrics_listen_addr: 127.0.0.1:9090 grpc_listen_addr: 0.0.0.0:50443 grpc_allow_insecure: false noise: private_key_path: /var/lib/headscale/noise_private.key prefixes: v6: fd7a:115c:a1e0::/48 v4: 100.64.0.0/10 allocation: sequential derp: server: enabled: true region_id: 900 region_code: "tz" region_name: "test" stun_listen_addr: "0.0.0.0:3478" private_key_path: /var/lib/headscale/derp_server_private.key automatically_add_embedded_derp_region: true ipv4: xxx.xxx.xxx.xxx # ipv6: verify_clients: true urls: - https://controlplane.tailscale.com/derpmap/default paths: - /etc/headscale/derp.yaml auto_update_enabled: true update_frequency: 3h disable_check_updates: false ephemeral_node_inactivity_timeout: 30m database: type: sqlite debug: false gorm: prepare_stmt: true parameterized_queries: true skip_err_record_not_found: true slow_threshold: 1000 sqlite: path: /etc/headscale/db.sqlite write_ahead_log: true wal_autocheckpoint: 1000 tls_cert_path: "" tls_key_path: "" log: format: text level: info policy: mode: database path: "" dns: magic_dns: false base_domain: tailscale.arpa override_local_dns: false nameservers: global: - 100.64.0.2 - 100.64.0.3 - 100.64.0.5 split: {} search_domains: [] extra_records: [] unix_socket: /var/run/headscale/headscale.sock unix_socket_permission: "0770" logtail: enabled: false randomize_client_port: false ``` derp.yml ```yaml regions: 900: regionid: 900 regioncode: tz regionname: test nodes: - name: 900a regionid: 900 hostname: https://derp.domain.com ipv4: xxx.xxx.xxx.xxx ipv6: none stunport: 0 stunonly: false derpport: 30000 canport80: false ``` `tailscale debug derp-map` (I removed all the other tailscale regions) ```json { "Regions": { "900": { "RegionID": 900, "RegionCode": "tz", "RegionName": "test", "Nodes": [ { "Name": "900", "RegionID": 900, "HostName": "headscale.domain.com", "IPv4": "xxx.xxx.xxx.xxx", "STUNPort": 3478, "DERPPort": 443 } ] } } } ``` `tailscale debug derp 900` ```json { "Info": [ "Region 900 == \"tz\"", "Successfully established a DERP connection with node \"headscale.domain.com\"", "Node \"headscale.domain.com\" returned IPv4 STUN response: xxx.xxx.xxx.xxx:34636" ], "Warnings": null, "Errors": [ "Error connecting to node \"headscale.domain.com\" @ \"headscale.domain.com:443\" over IPv6: dial tcp6: lookup headscale.domain.com on 1.1.1.1:53: no such host" ] } ``` Also which ports I should be exposing? I'm only exposing `50443` and `3478` (and `30000`) but there appears port `34636` which I don't get where it comes from, also in the config.yml comments it says that `41641` is also used?
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Nov 5, 2025):

It seems you're using the embedded DERP and configure it via config.yaml and also load essentially the same via derp.yaml (same ID, name, …). Can you please try to disable the internal derp and just load it from derp.yaml and rerun the checks from above. Should be doable via this change to config.yaml:

derp:
  server:
    enabled: false
  • You're running an external DERP at derp.domain.com, right?
  • Docs for a similar setup (2 external DERP, no builtin DERP)
@nblock commented on GitHub (Nov 5, 2025): It seems you're using the embedded DERP and configure it via `config.yaml` and also load essentially the same via `derp.yaml` (same ID, name, …). Can you please try to disable the internal derp and just load it from `derp.yaml` and rerun the checks from above. Should be doable via this change to `config.yaml`: ```yaml derp: server: enabled: false ``` - You're running an external DERP at `derp.domain.com`, right? - [Docs for a similar setup](https://headscale.net/stable/ref/derp/#__tabbed_1_2) (2 external DERP, no builtin DERP)
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@parasiteoflife commented on GitHub (Nov 5, 2025):

Yes, if I disable that it works. We can't specify a DERP port for the embedded one?

@parasiteoflife commented on GitHub (Nov 5, 2025): Yes, if I disable that it works. We can't specify a DERP port for the embedded one?
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Nov 6, 2025):

We can't specify a DERP port for the embedded one?

No, it reuses the server URL.

@nblock commented on GitHub (Nov 6, 2025): > We can't specify a DERP port for the embedded one? No, it reuses the server URL.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label DERP bug question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1126
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?