starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

Cannot connect to the embeded DERP when running headscale behind a reverse proxy #354

New Issue
Closed
opened 2025-12-29 01:27:32 +01:00 by adam · 8 comments
adam commented 2025-12-29 01:27:32 +01:00
Owner
Copy Link

Originally created by @Hyffer on GitHub (Oct 22, 2022).

Issue description

I am using headscale behind a reverse proxy. Headscale itself works fine, but I cannot connect to the embeded derp server.

$ tailscale netstate
Report:
	* UDP: true
	* IPv4: yes, ...:50888
	* IPv6: yes, [...]:48850
	* MappingVariesByDestIP: false
	* HairPinning: false
	* PortMapping: 
	* Nearest DERP: Hong Kong
	* DERP latency:
		- hkg: 66.5ms  (Hong Kong)
		- ...
		- headscale:         (Headscale Embedded DERP)

$ tailscale status
fd7a:115c:a1e0::1 hyffer-thinkpad- [namespace]       linux   -
fd7a:115c:a1e0::2 hyffers-huawei- [namespace]       android active; relay "headscale"; offline, tx 324024 rx 271116

$ tailscale status
fd7a:115c:a1e0::1 hyffer-thinkpad- [namespace]       linux   -
fd7a:115c:a1e0::2 hyffers-huawei- [namespace]       android active; relay "hkg", tx 358756 rx 254004

Context info

  • Version of headscale used
    v0.17.0-alpha4

  • The relevant config parameters you used
    Basically I just follow the "Running headscale on Linux" and "Running headscale behind a reverse proxy"

    server_url: https://[my domain]:8443
listen_addr: 0.0.0.0:8081
tls_cert_path: ""
tls_key_path: ""
derp:
    server:
        enabled: true

    And I use nginx as reverse proxy, it handles https encryption. All headscale functions are working fine so far.

    UDP port 3478 is set in firewall and I have tested that

  • Log

Oct 22 11:42:49 VM-4-13-ubuntu systemd[1]: Started headscale.
Oct 22 11:42:50 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:50+08:00 INF home/runner/work/headscale/headscale/derp_server.go:92 > DERP region: {RegionID:999 RegionCode:headscale RegionName:Headscale Embedded DERP Avoid:false Nodes:[0xc0001ee090]}
Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 WRN Listening without TLS but ServerURL does not start with http://
Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF listening and serving HTTP on: 0.0.0.0:8081
Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF listening and serving metrics on: 127.0.0.1:9090
Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF STUN server started at [::]:3478
Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF Setting up a DERPMap update worker frequency=86400000
Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF The HTTP2 server was closed error=EOF
Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Sending initial map handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Notifying peers handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:43:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:43:57+08:00 INF The client has closed the connection handler=PollNetMapStream machine=Hyffer-ThinkPad
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF The HTTP2 server was closed error=EOF
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF home/runner/work/headscale/headscale/protocol_common.go:633 > Machine successfully authorized machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Sending initial map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Notifying peers handler=PollNetMap machine="Hyffer's HUAWEI" noise=true
Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF The HTTP2 server was closed error=EOF
Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Sending initial map handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Notifying peers handler=PollNetMap machine=Hyffer-ThinkPad noise=true
Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF The client has closed the connection handler=PollNetMapStream machine="Hyffer's HUAWEI"
Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF derp client 127.0.0.1:33994/6e6f64656b65793a64643661663166336432333530656539613862363536386334613863303638303134323939386534393632376634356138303331626565396363303434333238: read EOF
Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF derp client 127.0.0.1:33994/6e6f64656b65793a64643661663166336432333530656539613862363536386334613863303638303134323939386534393632376634356138303331626565396363303434333238: read EOF

(Probably I should ask this on discord. I just registered an account, but get disabled immediately. I am trying to get my account back.)

Originally created by @Hyffer on GitHub (Oct 22, 2022). <!-- Headscale is a multinational community across the globe. Our common language is English. Please consider raising the issue in this language. --> <!-- If you have a question, please consider using our Discord for asking questions --> **Issue description** <!-- Please add your issue description. --> I am using headscale behind a reverse proxy. Headscale itself works fine, but I cannot connect to the embeded derp server. ``` $ tailscale netstate Report: * UDP: true * IPv4: yes, ...:50888 * IPv6: yes, [...]:48850 * MappingVariesByDestIP: false * HairPinning: false * PortMapping: * Nearest DERP: Hong Kong * DERP latency: - hkg: 66.5ms (Hong Kong) - ... - headscale: (Headscale Embedded DERP) $ tailscale status fd7a:115c:a1e0::1 hyffer-thinkpad- [namespace] linux - fd7a:115c:a1e0::2 hyffers-huawei- [namespace] android active; relay "headscale"; offline, tx 324024 rx 271116 $ tailscale status fd7a:115c:a1e0::1 hyffer-thinkpad- [namespace] linux - fd7a:115c:a1e0::2 hyffers-huawei- [namespace] android active; relay "hkg", tx 358756 rx 254004 ``` **Context info** - Version of headscale used v0.17.0-alpha4 - The relevant config parameters you used Basically I just follow the "[Running headscale on Linux](https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md)" and "[Running headscale behind a reverse proxy](https://github.com/juanfont/headscale/blob/main/docs/reverse-proxy.md)" ``` server_url: https://[my domain]:8443 listen_addr: 0.0.0.0:8081 tls_cert_path: "" tls_key_path: "" derp: server: enabled: true ``` And I use nginx as reverse proxy, it handles https encryption. All headscale functions are working fine so far. UDP port 3478 is set in firewall and I have tested that - Log ``` Oct 22 11:42:49 VM-4-13-ubuntu systemd[1]: Started headscale. Oct 22 11:42:50 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:50+08:00 INF home/runner/work/headscale/headscale/derp_server.go:92 > DERP region: {RegionID:999 RegionCode:headscale RegionName:Headscale Embedded DERP Avoid:false Nodes:[0xc0001ee090]} Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 WRN Listening without TLS but ServerURL does not start with http:// Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF listening and serving HTTP on: 0.0.0.0:8081 Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF listening and serving metrics on: 127.0.0.1:9090 Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF STUN server started at [::]:3478 Oct 22 11:42:51 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:51+08:00 INF Setting up a DERPMap update worker frequency=86400000 Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF The HTTP2 server was closed error=EOF Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Sending initial map handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:42:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:42:57+08:00 INF Notifying peers handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:43:57 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:43:57+08:00 INF The client has closed the connection handler=PollNetMapStream machine=Hyffer-ThinkPad Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF The HTTP2 server was closed error=EOF Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF home/runner/work/headscale/headscale/protocol_common.go:633 > Machine successfully authorized machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Sending initial map handler=PollNetMap machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:36 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:36+08:00 INF Notifying peers handler=PollNetMap machine="Hyffer's HUAWEI" noise=true Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF The HTTP2 server was closed error=EOF Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Client is ready to access the tailnet handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Sending initial map handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:44:37 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:44:37+08:00 INF Notifying peers handler=PollNetMap machine=Hyffer-ThinkPad noise=true Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF The client has closed the connection handler=PollNetMapStream machine="Hyffer's HUAWEI" Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF derp client 127.0.0.1:33994/6e6f64656b65793a64643661663166336432333530656539613862363536386334613863303638303134323939386534393632376634356138303331626565396363303434333238: read EOF Oct 22 11:45:33 VM-4-13-ubuntu headscale[2151579]: 2022-10-22T11:45:33+08:00 INF derp client 127.0.0.1:33994/6e6f64656b65793a64643661663166336432333530656539613862363536386334613863303638303134323939386534393632376634356138303331626565396363303434333238: read EOF ``` (Probably I should ask this on discord. I just registered an account, but get disabled immediately. I am trying to get my account back.)
adam added the bug label 2025-12-29 01:27:32 +01:00
adam closed this issue 2025-12-29 01:27:33 +01:00
adam commented 2025-12-29 01:27:33 +01:00
Author
Owner
Copy Link

@Hyffer commented on GitHub (Oct 22, 2022):

nginx config: (if matters)

map $http_upgrade $connection_upgrade {
    default      keep-alive;
    'websocket'  upgrade;
    ''           close;
}

server {
    listen 8080;
        listen [::]:8080;

        listen 8443      ssl http2;
        listen [::]:8443 ssl http2;

    server_name [my domain];

    ssl_certificate ...;
    ssl_certificate_key ...;
    ssl_protocols TLSv1.2 TLSv1.3;

# headscale
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $server_name;
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    }
}
@Hyffer commented on GitHub (Oct 22, 2022): nginx config: (if matters) ``` map $http_upgrade $connection_upgrade { default keep-alive; 'websocket' upgrade; '' close; } server { listen 8080; listen [::]:8080; listen 8443 ssl http2; listen [::]:8443 ssl http2; server_name [my domain]; ssl_certificate ...; ssl_certificate_key ...; ssl_protocols TLSv1.2 TLSv1.3; # headscale location / { proxy_pass http://127.0.0.1:8081; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header Host $server_name; proxy_redirect http:// https://; proxy_buffering off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto; add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always; } } ```
adam commented 2025-12-29 01:27:33 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Jan 1, 2023):

Are you still having this?

@juanfont commented on GitHub (Jan 1, 2023): Are you still having this?
adam commented 2025-12-29 01:27:34 +01:00
Author
Owner
Copy Link

@Hyffer commented on GitHub (Jan 6, 2023):

I cannot reproduce this somehow. And I am at version 0.17.1 now, everything works fine.

Thanks for bringing such a great project.

@Hyffer commented on GitHub (Jan 6, 2023): I cannot reproduce this somehow. And I am at version 0.17.1 now, everything works fine. Thanks for bringing such a great project.
adam commented 2025-12-29 01:27:34 +01:00
Author
Owner
Copy Link

@javito1081 commented on GitHub (Apr 23, 2025):

Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it

@javito1081 commented on GitHub (Apr 23, 2025): Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it
adam commented 2025-12-29 01:27:34 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Apr 23, 2025):

Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it

Please ask support questions on Discord. Thanks.

@nblock commented on GitHub (Apr 23, 2025): > Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it Please [ask support questions on Discord](https://headscale.net/stable/about/help/). Thanks.
adam commented 2025-12-29 01:27:35 +01:00
Author
Owner
Copy Link

@javito1081 commented on GitHub (Apr 23, 2025):

Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it

Please ask support questions on Discord. Thanks.

I did, I have no solución yet :-(

@javito1081 commented on GitHub (Apr 23, 2025): > > Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it > > Please [ask support questions on Discord](https://headscale.net/stable/about/help/). Thanks. I did, I have no solución yet :-(
adam commented 2025-12-29 01:27:35 +01:00
Author
Owner
Copy Link

@Hyffer commented on GitHub (Apr 23, 2025):

Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it

No, it didn't happen ever since. Many things has had changed on my machine, and it works pretty well now for a long time.

@Hyffer commented on GitHub (Apr 23, 2025): > Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it No, it didn't happen ever since. Many things has had changed on my machine, and it works pretty well now for a long time.
adam commented 2025-12-29 01:27:35 +01:00
Author
Owner
Copy Link

@javito1081 commented on GitHub (Apr 23, 2025):

Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it

No, it didn't happen ever since. Many things has had changed on my machine, and it works pretty well now for a long time.

Well im experiencing something wierd, I have 3 Site to Site connections, all 3 sites are connected with a direct connection, but after a while, one of the sites switches to relay connection, I didn't have a derp server enable at the time, so I decided to enable it cause my latencynwent from 5 ms to 120+ ms because of the relay thingy.

When I turn it on, I get no latency reading from my server, none what so ever, BUT if I leave my derp server on without the list of servers in the headacale config file, I get latency on my derp server and also my clients can connect to it.

So I don't know if this is a bug or if I'm doing something wrong.

Other 2 sites don't experience this behavior, they are always using a direct connection, but only Site C experiences that issue.

I have Site A as main site, Site B and C both connect to Site A subnet router, Site A local network router has the routes to redirect traffic between Site B and C

@javito1081 commented on GitHub (Apr 23, 2025): > > Did this ever happened to ya again? im trying right now to get my derp enable but i get no connections to it > > No, it didn't happen ever since. Many things has had changed on my machine, and it works pretty well now for a long time. Well im experiencing something wierd, I have 3 Site to Site connections, all 3 sites are connected with a direct connection, but after a while, one of the sites switches to relay connection, I didn't have a derp server enable at the time, so I decided to enable it cause my latencynwent from 5 ms to 120+ ms because of the relay thingy. When I turn it on, I get no latency reading from my server, none what so ever, BUT if I leave my derp server on without the list of servers in the headacale config file, I get latency on my derp server and also my clients can connect to it. So I don't know if this is a bug or if I'm doing something wrong. Other 2 sites don't experience this behavior, they are always using a direct connection, but only Site C experiences that issue. I have Site A as main site, Site B and C both connect to Site A subnet router, Site A local network router has the routes to redirect traffic between Site B and C
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#354
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?