mirror of
https://github.com/juanfont/headscale.git
synced 2026-01-11 20:00:28 +01:00
Nodes connected to headscale fail to see each other if headscale is fronted by nginx #12
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @viq on GitHub (Jul 17, 2021).
Descriptions
Machines manage to authenticate to headscale, see each other's existence in
tailscale status, but cannot ping each other.They are able to discover and ping each other fine when using tailscale infrastructure, or when talking to headscale directly, without nginx in the way.
Situation
Two machines running OpenBSD-current, tailscale
1.10.2, headscale0.3.4headscaleis running headscale with postgres, tailscale, nginx in front of headscale for TLS termination; has public IP, tailscale told to listen on specific port and that port opened on firewallinnernet-testis behind NAT (aggressive). Runs only tailscaledConfiguration
headscale
nginx
Logs
Part 1
Attaching machines to headscale
nginx
headscale & tailscaled
headscale machine
innernet-test machine
At this point daemons were stopped. Situation at this point:
tailscale statusoninnernet-testshows both machines, onheadscaleshows only itself.headscale -n viqWORKS nodes listshows both machines.Part 2
At this point I'm rotating logs, starting daemons, and will run status and ping
Logs
headscale machine
nginx
headscale & tailscaled
innernet-test
tailscaled
@viq commented on GitHub (Jul 17, 2021):
In case it's relevant, registration has been done using authkeys.
@juanfont commented on GitHub (Jul 17, 2021):
Can you add the output for this:
headscale -n viqWORKS nodes list -o json@viq commented on GitHub (Jul 17, 2021):
@viq commented on GitHub (Jul 18, 2021):
How does the http connection work? Looking at tcpdump, I see a bunch of
Connection: closebetween nginx and headscale, but not when tailscaled and headscale are talking directly. Maybe nginx needs some tuning for long polling or whatnot?@viq commented on GitHub (Jul 18, 2021):
OK, so seems like some of the settings from https://help.hcltechsw.com/connections/v65/admin/install/inst_post_nginx.html may have helped...
@viq commented on GitHub (Jul 18, 2021):
Currently seems to work with the below settings, I'll poke at it some more at a later time.
@juanfont commented on GitHub (Jul 18, 2021):
I have not been able to replicate the issue in my Vagrant setup, aside the
longpolling connections being closed after 60 secs (nginx's default) as
expected
Would in be possible to have access to your environment?
On Sun, Jul 18, 2021, 20:16 viq @.***> wrote:
@viq commented on GitHub (Jul 22, 2021):
Update: currently 2 vhosts,
headscale.viq.vcwith "broken" config as previously, andtestscale.viq.vcwith I think working config:You still have access, feel free to poke around.
@viq commented on GitHub (Jul 22, 2021):
Hm, looks like with those settings I need to restart
headscalefor nodes (either new, or after being restarted) to see each other.@juanfont commented on GitHub (Jul 25, 2021):
I have been finally able to replicate this 😄
I will check nginx configs now...
@juanfont commented on GitHub (Jul 30, 2021):
@viq @bharathmsd7
Can you check with this config in nginx?
https://github.com/juanfont/headscale/wiki/nginx-configuration
@qbit commented on GitHub (Jul 30, 2021):
Switched my config over to nginx, this seems to resolve it for me - I am able to ping hosts and ssh to them!
@qbit commented on GitHub (Jul 30, 2021):
It seems that after a while though things break down. I was adding hosts to a namespace and now
tailscale up --login-server ....just hangs.I see
Client is registered and we have the current NodeKey. All clear to /mapand then a POST to/machine/ID, but nothing after that.@viq commented on GitHub (Jul 31, 2021):
With following config
(disabling websocket related things one by one; finally
proxy_http_versionseemed to make a difference) after a couple restarts of various things I was still able to register clients, without the issues @qbit is describing (I would see them sometimes otherwise; might still as it's not necessarily 100% repeatable). But adding a new (ephemeral) node to headscale, none of the two so far connected nodes see it without restarting things. Which I guess is the same state as we started with...@viq commented on GitHub (Jul 31, 2021):
I'm now retrying with exactly your set of options.
@viq commented on GitHub (Jul 31, 2021):
Specifically:
and I'm apparently seeing what you described, i.e.
headscaleseeing clients closting the connections, but nginx and tailscaled believing they are still open.@juanfont commented on GitHub (Jul 31, 2021):
Can you send us the logs from the tailscaled daemons failing to connect?
@qbit commented on GitHub (Aug 4, 2021):
Ya, i'll try and get some more details today - It looks like the same behavior as #50 though.
@juanfont commented on GitHub (Aug 6, 2021):
@qbit @viq Can you please try with v0.5.0? https://github.com/juanfont/headscale/releases/tag/v0.5.0
@Juliaria08 commented on GitHub (Aug 14, 2021):
It fixed the issue i had and it is now working, thanks for the work of this project
@juanfont commented on GitHub (Aug 24, 2021):
@viq @qbit can you check 0.7? This issue should be solved now.
@qbit commented on GitHub (Aug 24, 2021):
Can confirm! Currently running just fine!