starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-12 04:10:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

Machine waiting for OIDC auth is spamming the server with requests #265

New Issue
Closed
opened 2025-12-29 01:25:19 +01:00 by adam · 7 comments
adam commented 2025-12-29 01:25:19 +01:00
Owner
Copy Link

Originally created by @iSchluff on GitHub (Jun 1, 2022).

Bug description
After tailscale up the machine is spamming the control server with http-requests waiting for the user to complete the oidc auth.
This is causing considerable load on the server, with a constant 80 requests/s in one instance.
image

The logs look like follows

[GIN] 2022/06/01 - 13:53:22 | 200 |    1.479334ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.899759ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.678392ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.717471ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.594622ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.819499ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.620602ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.890737ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"
[GIN] 2022/06/01 - 13:53:22 | 200 |    1.807969ms | xx.xx.141.118 | POST     "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c"

I assume the official tailscale control server just holds the connection open in that case instead of closing with code 200? Maybe headscale should do something similar?

From 6f700925ce/control/controlclient/direct.go (L265-L271) I gather that headscale would need to implement long-poll for the login url aswell.

To Reproduce
Run tailscale up --login-server=https://....

Details

  • Headscale commit be25bbce92
  • Tailscale 1.22.2
  • OS: Linux
  • Kernel: 5.15.19
Originally created by @iSchluff on GitHub (Jun 1, 2022). **Bug description** After tailscale up the machine is spamming the control server with http-requests waiting for the user to complete the oidc auth. This is causing considerable load on the server, with a constant 80 requests/s in one instance. ![image](https://user-images.githubusercontent.com/1429641/171422471-17adde25-0b0c-4b66-9d4f-b67290ddac04.png) The logs look like follows ``` [GIN] 2022/06/01 - 13:53:22 | 200 | 1.479334ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.899759ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.678392ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.717471ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.594622ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.819499ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.620602ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.890737ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" [GIN] 2022/06/01 - 13:53:22 | 200 | 1.807969ms | xx.xx.141.118 | POST "/machine/67aa1e887ece306e52c5f8b27663042a3cd4421d0ef3d94d9a74f1e09761c92c" ``` I assume the official tailscale control server just holds the connection open in that case instead of closing with code 200? Maybe headscale should do something similar? From https://github.com/tailscale/tailscale/blob/6f700925cef22d8b2a100840c8d9eb084dadfece/control/controlclient/direct.go#L265-L271 I gather that headscale would need to implement long-poll for the login url aswell. **To Reproduce** Run tailscale up --login-server=https://.... **Details** - Headscale commit be25bbce92fb1be3c233ad286971bd738923c352 - Tailscale 1.22.2 - OS: Linux - Kernel: 5.15.19
adam added the bug label 2025-12-29 01:25:19 +01:00
adam closed this issue 2025-12-29 01:25:19 +01:00
adam commented 2025-12-29 01:25:20 +01:00
Author
Owner
Copy Link

@anton-livewyer commented on GitHub (Jul 18, 2022):

I have the same issue with Linux and Windows Tailscale clients. Only deletion of the node from Headscale and relogin resolves the issue.

UPD: steps described in troubleshooting section didn't help to resolve it on Windows host

@anton-livewyer commented on GitHub (Jul 18, 2022): I have the same issue with Linux and Windows Tailscale clients. Only deletion of the node from Headscale and relogin resolves the issue. UPD: steps described in [troubleshooting section](https://github.com/juanfont/headscale/blob/main/docs/windows-client.md#troubleshooting) didn't help to resolve it on Windows host
adam commented 2025-12-29 01:25:20 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Aug 11, 2022):

This is closely related to #706, and (very partially) addressed in #725.

Will remain open for the time being, because indeed we need to implement long polling.

@juanfont commented on GitHub (Aug 11, 2022): This is closely related to #706, and (very partially) addressed in #725. Will remain open for the time being, because indeed we need to implement long polling.
adam commented 2025-12-29 01:25:21 +01:00
Author
Owner
Copy Link

@juanfont commented on GitHub (Aug 12, 2022):

@iSchluff in 0.16.1 we have implemented a workaround to help on this. Can you give it a try?

@juanfont commented on GitHub (Aug 12, 2022): @iSchluff in 0.16.1 we have implemented a workaround to help on this. Can you give it a try?
adam commented 2025-12-29 01:25:22 +01:00
Author
Owner
Copy Link

@iSchluff commented on GitHub (Aug 18, 2022):

@iSchluff in 0.16.1 we have implemented a workaround to help on this. Can you give it a try?

unfortunately I still see the same behavior with 0.16.3
Screenshot 2022-08-18 at 18-17-48 Headscale - Grafana

$ nomad alloc exec -i -t -task headscale 459f7965-ab8b-7713-4b74-cde04ce541da /bin/headscale version
v0.16.3
@iSchluff commented on GitHub (Aug 18, 2022): > @iSchluff in 0.16.1 we have implemented a workaround to help on this. Can you give it a try? unfortunately I still see the same behavior with 0.16.3 ![Screenshot 2022-08-18 at 18-17-48 Headscale - Grafana](https://user-images.githubusercontent.com/1429641/185444755-c04afc67-5dec-4dc8-82be-9ab2f5013677.png) ``` $ nomad alloc exec -i -t -task headscale 459f7965-ab8b-7713-4b74-cde04ce541da /bin/headscale version v0.16.3 ```
adam commented 2025-12-29 01:25:22 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Oct 28, 2022):

Not sure if anything has happened in 0.17, can you try that?

@kradalby commented on GitHub (Oct 28, 2022): Not sure if anything has happened in 0.17, can you try that?
adam commented 2025-12-29 01:25:23 +01:00
Author
Owner
Copy Link

@iSchluff commented on GitHub (Oct 31, 2022):

@kradalby I think request rates are fine with 0.17.0, or is this all moved to the websocket with noise now? Client is tailscale 1.32.0

But I still see logging with high frequency so I doubt whether it improved.
The repeated log line is

Auth URL for reauthenticate successfully sent machine=xxx noise=true

This is the counter for just this single logline
image

@iSchluff commented on GitHub (Oct 31, 2022): @kradalby I think request rates are fine with 0.17.0, or is this all moved to the websocket with noise now? Client is tailscale 1.32.0 But I still see logging with high frequency so I doubt whether it improved. The repeated log line is ``` Auth URL for reauthenticate successfully sent machine=xxx noise=true ``` This is the counter for just this single logline ![image](https://user-images.githubusercontent.com/1429641/199090911-fdd4c50f-b271-4ec8-bbc1-8a63bf70d2a8.png)
adam commented 2025-12-29 01:25:24 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (May 10, 2023):

This seems resolved, also might be part of #727

@kradalby commented on GitHub (May 10, 2023): This seems resolved, also might be part of #727
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#265
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?