starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-13 12:50:32 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

Headscale should retry gracefully when the OIDC provider is unavailable #691

New Issue
Open
opened 2025-12-29 02:22:08 +01:00 by adam · 6 comments
adam commented 2025-12-29 02:22:08 +01:00
Owner
Copy Link

Originally created by @korfuri on GitHub (Apr 16, 2024).

Why

In some setups, there is a cyclic dependency between headscale and its OIDC backend. This could be because e.g. the OIDC provider is located through the tailnet (e.g. the OIDC provider is configured via magicDNS). In those cases, if the system boots cold, the machine running headscale can't join the tailnet without headscale being up, and headscale can't come up without its OIDC provider, unless only_start_if_oidc_is_available is false. But setting this to false means that headscale will simply not use OIDC unless it is restarted later for some reason.

Description

When starting headscale, if only_start_if_oidc_is_available is false and the OIDC provider is not available, headscale gives up on OIDC until it is restarted. This shouldn't be the case: headscale should keep trying to connect to the OIDC backend so it is used once it becomes available.

This should probably extend to failures of the OIDC provider while serving: if the OIDC provider becomes unavailable and only_start_of_oidc_is_available is false, headscale should degrade gracefully to use non-OIDC authentication until it detects that the OIDC provider is back up.

Originally created by @korfuri on GitHub (Apr 16, 2024). ## Why In some setups, there is a cyclic dependency between headscale and its OIDC backend. This could be because e.g. the OIDC provider is located through the tailnet (e.g. the OIDC provider is configured via magicDNS). In those cases, if the system boots cold, the machine running headscale can't join the tailnet without headscale being up, and headscale can't come up without its OIDC provider, unless only_start_if_oidc_is_available is false. But setting this to false means that headscale will simply not use OIDC unless it is restarted later for some reason. ## Description When starting headscale, if only_start_if_oidc_is_available is false and the OIDC provider is not available, headscale [gives up on OIDC](https://github.com/juanfont/headscale/blob/60f0cf908ca630a98e087002e3c5e5a10e95d126/hscontrol/app.go#L162) until it is restarted. This shouldn't be the case: headscale should keep trying to connect to the OIDC backend so it is used once it becomes available. This should probably extend to failures of the OIDC provider while serving: if the OIDC provider becomes unavailable and only_start_of_oidc_is_available is false, headscale should degrade gracefully to use non-OIDC authentication until it detects that the OIDC provider is back up.
adam added the enhancementno-stale-botOIDC labels 2025-12-29 02:22:08 +01:00
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Dec 26, 2024):

This issue is stale because it has been open for 90 days with no activity.

@github-actions[bot] commented on GitHub (Dec 26, 2024): This issue is stale because it has been open for 90 days with no activity.
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@korfuri commented on GitHub (Dec 26, 2024):

Not stale :)

@korfuri commented on GitHub (Dec 26, 2024): Not stale :)
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@allenwoods commented on GitHub (Mar 10, 2025):

I encountered the same problem. My OIDC server is set up internally (100.64.0.1), and I can not connect it before starting headscale. Although it seems that using a CLI based authentication first and then switching to the OIDC service seems to be somewhat feasible, I noticed that according to Custom OIDC providers it is stated:

Self-hosted identity providers must be publicly accessible on the internet. IP block listing may interfere with the ability to sign up or authenticate to your tailnet.

Perhaps building OIDC provider within tailnet is not a good idea?

@allenwoods commented on GitHub (Mar 10, 2025): I encountered the same problem. My OIDC server is set up internally (100.64.0.1), and I can not connect it before starting headscale. Although it seems that using a CLI based authentication first and then switching to the OIDC service seems to be somewhat feasible, I noticed that according to [Custom OIDC providers](https://tailscale.com/kb/1240/sso-custom-oidc,) it is stated: > Self-hosted identity providers must be publicly accessible on the internet. IP block listing may interfere with the ability to sign up or authenticate to your tailnet. Perhaps building OIDC provider within tailnet is not a good idea?
adam commented 2025-12-29 02:22:09 +01:00
Author
Owner
Copy Link

@korfuri commented on GitHub (Jun 28, 2025):

Question for @kradalby - would it make sense to change the behavior (possibly behind an option) so that one could express "I never want headscale to allow registration by something other than OIDC"?

This could mean that on startup, if the OIDC provider is unavailable, we start in a mode that doesn't allow registration, with an infinite retry loop to contact the provider. If the provider becomes unavailable after that, this is mostly transparent for Headscale, but obviously registration attempts will fail as they'll redirect to a broken provider.

This is probably easier to implement than graceful degradation to non-OIDC. What do you think?

@korfuri commented on GitHub (Jun 28, 2025): Question for @kradalby - would it make sense to change the behavior (possibly behind an option) so that one could express "I never want headscale to allow registration by something other than OIDC"? This could mean that on startup, if the OIDC provider is unavailable, we start in a mode that doesn't allow registration, with an infinite retry loop to contact the provider. If the provider becomes unavailable after that, this is mostly transparent for Headscale, but obviously registration attempts will fail as they'll redirect to a broken provider. This is probably easier to implement than graceful degradation to non-OIDC. What do you think?
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Jun 28, 2025):

I think an option to disable cli users would be fine.

For the oidc retry, I feel like it would be something systemd and similar can handle for you, restarting if there is a problem which would look for the oidc

@kradalby commented on GitHub (Jun 28, 2025): I think an option to disable cli users would be fine. For the oidc retry, I feel like it would be something systemd and similar can handle for you, restarting if there is a problem which would look for the oidc
adam commented 2025-12-29 02:22:10 +01:00
Author
Owner
Copy Link

@Minion3665 commented on GitHub (Sep 13, 2025):

(I wrote a patch to defer auth setup until registration if it fails - at the moment I wanted this for my own installation in somewhat of a hurry... you, the reader, are welcome to use it (add the patch and enable option auth_setup_allow_defer in your config), but I provide no guarantees about whether it'll be good for your use-case and as per the contributing guidelines I won't be upstreaming it as-is. Anyone is welcome to do what you like with it (including upstream it, presumably provided you make a design doc, tests, etc.) - though as it wasn't discussed with the maintainers beforehand I'm sure I've made decisions that would need to be changed in that process 😃... maybe one day I'll get around to contributing this properly, no promises though)

@Minion3665 commented on GitHub (Sep 13, 2025): (I wrote [a patch to defer auth setup until registration if it fails](https://github.com/FreshlyBakedCake/headscale/commit/9803b8ce2460772ba0eaae7b245d10009ba6a00b) - at the moment I wanted this for my own installation in somewhat of a hurry... you, the reader, are welcome to use it (add the patch and enable option `auth_setup_allow_defer` in your config), but I provide no guarantees about whether it'll be good for your use-case and [as per the contributing guidelines](https://github.com/juanfont/headscale/blob/main/CONTRIBUTING.md) I won't be upstreaming it as-is. Anyone is welcome to do what you like with it (including upstream it, presumably provided you make a design doc, tests, etc.) - though as it wasn't discussed with the maintainers beforehand I'm sure I've made decisions that would need to be changed in that process 😃... maybe one day I'll get around to contributing this properly, no promises though)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label OIDC enhancement no-stale-bot
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#691
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?