[Bug] Internal server error when logging in using Google OIDC #970

Closed
opened 2025-12-29 02:26:52 +01:00 by adam · 23 comments
Owner

Originally created by @adriangoransson on GitHub (Mar 13, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Similar to https://github.com/juanfont/headscale/issues/2465 which was closed.

An existing (since Headscale 0.23) tried to log in for the first time since upgrading to 0.25, without success.

2025-03-13T08:09:14Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:09:24Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403
2025-03-13T08:09:34Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:10:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:11:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500

Even after destroying the user (along with the associated nodes and preauthkeys) and restarting headscale, the issue persists.

I tried to investigate how the OIDC handling has changed over the versions, but from what I could gather most issues seem to stem from the email having possibly changed? This has not happened in our case.

Expected Behavior

The current user should be migrated to the new format.

Steps To Reproduce

  1. Setup a user with Google OIDC on Headscale 0.23.0.
  2. Migrate to 0.24 and then 0.25
  3. Try to reauthenticate

Environment

- OS: macOS 15.3.1
- Headscale version: 0.25.1
- Tailscale version: 1.80.2

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

# config.yaml

map_legacy_users: true

oidc:
  only_start_if_oidc_is_available: true
  issuer: https://accounts.google.com
  client_id: <redacted>
  client_secret_path: /secrets/headscale/oidc_client_secret

  expiry: 180d
  use_expiry_from_token: false

  scope: ["openid", "profile", "email"]
  extra_params:
    domain_hint: <redacted>

  allowed_domains:
    - <redacted>

  strip_email_domain: true

Thank you for your dedicated work on Headscale!

Originally created by @adriangoransson on GitHub (Mar 13, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Similar to https://github.com/juanfont/headscale/issues/2465 which was closed. An existing (since Headscale 0.23) tried to log in for the first time since upgrading to 0.25, without success. ``` 2025-03-13T08:09:14Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 2025-03-13T08:09:24Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403 2025-03-13T08:09:34Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 2025-03-13T08:10:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 2025-03-13T08:11:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 ``` Even after destroying the user (along with the associated nodes and preauthkeys) and restarting headscale, the issue persists. I tried to investigate how the OIDC handling has changed over the versions, but from what I could gather most issues seem to stem from the email having possibly changed? This has not happened in our case. ### Expected Behavior The current user should be migrated to the new format. ### Steps To Reproduce 1. Setup a user with Google OIDC on Headscale 0.23.0. 2. Migrate to 0.24 and then 0.25 3. Try to reauthenticate ### Environment ```markdown - OS: macOS 15.3.1 - Headscale version: 0.25.1 - Tailscale version: 1.80.2 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Anything else? ```yaml # config.yaml map_legacy_users: true oidc: only_start_if_oidc_is_available: true issuer: https://accounts.google.com client_id: <redacted> client_secret_path: /secrets/headscale/oidc_client_secret expiry: 180d use_expiry_from_token: false scope: ["openid", "profile", "email"] extra_params: domain_hint: <redacted> allowed_domains: - <redacted> strip_email_domain: true ``` Thank you for your dedicated work on Headscale!
adam added the bugno-stale-botOIDCdatabase labels 2025-12-29 02:26:52 +01:00
adam closed this issue 2025-12-29 02:26:52 +01:00
Author
Owner

@kradalby commented on GitHub (Mar 14, 2025):

Can you provide a list of users in the database and their fields and the ones that try to login when you get the error?

I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24.

@kradalby commented on GitHub (Mar 14, 2025): Can you provide a list of users in the database and their fields and the ones that try to login when you get the error? I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24.
Author
Owner

@adriangoransson commented on GitHub (Mar 17, 2025):

Can you provide a list of users in the database and their fields and the ones that try to login when you get the error?

I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24.

Sure, here you go!
User 1 is our subnet router configured as a non-OIDC user. User 15 is an ordinary OIDC user, just with a special email.

sqlite> select * from users;
id | created_at                          | updated_at                          | deleted_at | name             | display_name     | email                     | provider_identifier | provider | profile_pic_url
 1 | 2024-03-08 22:00:05.052505772+00:00 | 2024-03-08 22:00:05.052505772+00:00 |            | pod-node         |                  |                           |                     |          |
 2 | 2024-03-08 22:01:43.602676781+00:00 | 2025-03-13 10:04:41.871021878+00:00 |            | adrian.goransson | Adrian Göransson | adrian.goransson@<domain> | <id_1>              | oidc     | <url_2>
 4 | 2024-03-14 13:19:43.923617972+00:00 | 2024-03-14 13:19:43.923617972+00:00 |            | <first>.<last>   |                  |                           |                     |          |
 5 | 2024-03-15 15:43:18.872487023+00:00 | 2024-03-15 15:43:18.872487023+00:00 |            | <first>.<last>   |                  |                           |                     |          |
 7 | 2024-05-30 07:32:14.825899374+00:00 | 2024-05-30 07:32:14.825899374+00:00 |            | <first>.<last>   |                  |                           |                     |          |
 8 | 2024-07-15 11:11:48.313729614+00:00 | 2024-07-15 11:11:48.313729614+00:00 |            | <first>.<last>   |                  |                           |                     |          |
 9 | 2024-08-26 07:41:05.647379266+00:00 | 2024-08-26 07:41:05.647379266+00:00 |            | <first>.<last>   |                  |                           |                     |          |
10 | 2024-08-27 09:37:05.68427993+00:00  | 2024-08-27 09:37:05.68427993+00:00  |            | <first>.<last>   |                  |                           |                     |          |
11 | 2024-09-25 08:05:59.81951107+00:00  | 2024-09-25 08:05:59.81951107+00:00  |            | <first>.<last>   |                  |                           |                     |          |
12 | 2024-10-16 12:02:24.530535501+00:00 | 2024-10-16 12:02:24.530535501+00:00 |            | <first>.<last>   |                  |                           |                     |          |
13 | 2024-11-05 05:32:40.270709176+00:00 | 2024-11-05 05:32:40.270709176+00:00 |            | <first>.<last>   |                  |                           |                     |          |
14 | 2024-12-17 09:32:19.148077663+00:00 | 2024-12-17 09:32:19.148077663+00:00 |            | <first>.<last>   |                  |                           |                     |          |
15 | 2024-12-20 08:11:36.352191756+00:00 | 2024-12-20 08:11:36.352191756+00:00 |            | <custom>         |                  |                           |                     |          |
16 | 2025-02-10 09:43:41.061294411+00:00 | 2025-02-10 09:43:41.061294411+00:00 |            | <first>.<last>   |                  |                           |                     |          |
17 | 2025-03-11 08:36:14.999339636+00:00 | 2025-03-11 08:36:14.999339636+00:00 |            |                  | <first> <last>   | <first>.<last>@<domain>   | <id_2>              | oidc     | <url_2>
  • My user seems to have been migrated correctly. I logged out and logged back in using the Tailscale client shortly after updating Headscale.
  • Users 4-16 are OIDC, but haven't reauthenticated yet.
  • User 17 used to be ID 3, but a new user was created upon reauthentication (after upgrading to 0.24 and then 0.25) so we destroyed the old one.

Note that the provider_identifier for me and user 17 are different. I don't know if they should be.

@adriangoransson commented on GitHub (Mar 17, 2025): > Can you provide a list of users in the database and their fields and the ones that try to login when you get the error? > > I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24. Sure, here you go! User 1 is our subnet router configured as a non-OIDC user. User 15 is an ordinary OIDC user, just with a special email. ``` sqlite> select * from users; id | created_at | updated_at | deleted_at | name | display_name | email | provider_identifier | provider | profile_pic_url 1 | 2024-03-08 22:00:05.052505772+00:00 | 2024-03-08 22:00:05.052505772+00:00 | | pod-node | | | | | 2 | 2024-03-08 22:01:43.602676781+00:00 | 2025-03-13 10:04:41.871021878+00:00 | | adrian.goransson | Adrian Göransson | adrian.goransson@<domain> | <id_1> | oidc | <url_2> 4 | 2024-03-14 13:19:43.923617972+00:00 | 2024-03-14 13:19:43.923617972+00:00 | | <first>.<last> | | | | | 5 | 2024-03-15 15:43:18.872487023+00:00 | 2024-03-15 15:43:18.872487023+00:00 | | <first>.<last> | | | | | 7 | 2024-05-30 07:32:14.825899374+00:00 | 2024-05-30 07:32:14.825899374+00:00 | | <first>.<last> | | | | | 8 | 2024-07-15 11:11:48.313729614+00:00 | 2024-07-15 11:11:48.313729614+00:00 | | <first>.<last> | | | | | 9 | 2024-08-26 07:41:05.647379266+00:00 | 2024-08-26 07:41:05.647379266+00:00 | | <first>.<last> | | | | | 10 | 2024-08-27 09:37:05.68427993+00:00 | 2024-08-27 09:37:05.68427993+00:00 | | <first>.<last> | | | | | 11 | 2024-09-25 08:05:59.81951107+00:00 | 2024-09-25 08:05:59.81951107+00:00 | | <first>.<last> | | | | | 12 | 2024-10-16 12:02:24.530535501+00:00 | 2024-10-16 12:02:24.530535501+00:00 | | <first>.<last> | | | | | 13 | 2024-11-05 05:32:40.270709176+00:00 | 2024-11-05 05:32:40.270709176+00:00 | | <first>.<last> | | | | | 14 | 2024-12-17 09:32:19.148077663+00:00 | 2024-12-17 09:32:19.148077663+00:00 | | <first>.<last> | | | | | 15 | 2024-12-20 08:11:36.352191756+00:00 | 2024-12-20 08:11:36.352191756+00:00 | | <custom> | | | | | 16 | 2025-02-10 09:43:41.061294411+00:00 | 2025-02-10 09:43:41.061294411+00:00 | | <first>.<last> | | | | | 17 | 2025-03-11 08:36:14.999339636+00:00 | 2025-03-11 08:36:14.999339636+00:00 | | | <first> <last> | <first>.<last>@<domain> | <id_2> | oidc | <url_2> ``` - _My_ user seems to have been migrated correctly. I logged out and logged back in using the Tailscale client shortly after updating Headscale. - Users 4-16 are OIDC, but haven't reauthenticated yet. - User 17 **used to be** ID 3, but a new user was created upon reauthentication (after upgrading to 0.24 and then 0.25) so we destroyed the old one. Note that the `provider_identifier` for me and user 17 are different. I don't know if they should be.
Author
Owner

@adriangoransson commented on GitHub (Mar 17, 2025):

Just tried two things:

Reauth

Reauthenticated user 4 (not that the id matters I guess 😅) and encountered an internal server error again.

2025-03-17T12:55:36Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0xc000564480, chan: 0xc0000af8f0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true
2025-03-17T12:55:38Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500

However, shutting down Tailscale and starting it again worked, so the previous key still seems valid. The output of headscale users list is unchanged, though I haven't verified the database.

2025-03-17T12:56:21Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0xc000003e00, chan: 0xc0000ae9a0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true

Registered new user

Registered a user who has never logged in to Headscale previously. They have a unique first and last name, so there should be no such collisions.

2025-03-17T12:59:15Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-17T12:59:23Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403
2025-03-17T12:59:36Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500```
@adriangoransson commented on GitHub (Mar 17, 2025): Just tried two things: ## Reauth Reauthenticated user 4 (not that the id matters I guess 😅) and encountered an internal server error again. ``` 2025-03-17T12:55:36Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0xc000564480, chan: 0xc0000af8f0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true 2025-03-17T12:55:38Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 ``` However, shutting down Tailscale and starting it again worked, so the previous key still seems valid. The output of `headscale users list` is unchanged, though I haven't verified the database. ``` 2025-03-17T12:56:21Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0xc000003e00, chan: 0xc0000ae9a0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true ``` ## Registered new user Registered a user who has never logged in to Headscale previously. They have a unique first and last name, so there should be no such collisions. ``` 2025-03-17T12:59:15Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 2025-03-17T12:59:23Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403 2025-03-17T12:59:36Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500``` ```
Author
Owner

@SeaweedbrainCY commented on GitHub (Apr 2, 2025):

I have the same issue. Even adding new users is creating this behavior. Does anyone has any hint on this ? Any workaround ?
In this state, headscale 0.25 is just unusable

@SeaweedbrainCY commented on GitHub (Apr 2, 2025): I have the same issue. Even adding new users is creating this behavior. Does anyone has any hint on this ? Any workaround ? In this state, headscale 0.25 is just unusable
Author
Owner

@SeaweedbrainCY commented on GitHub (Apr 2, 2025):

For those having the same issue, I managed to solve the user creation error :

My issue was that even with the migration flag on, my users had a user.name empty,

> headscale users list
ID | Name    | Username          | Email                                 | Created
1  |         | <redacted>        |                                       | 2024-04-24 03:37:43
2  |         | <redacted>        |                                       | 2024-04-25 19:20:32
3  |         | <redacted>        |                                       | 2024-04-28 16:01:08
5  |         | <redacted>        |                                       | 2024-05-04 15:11:28
6  |         | <redacted>        |                                       | 2024-05-04 16:30:55
7  |         | <redacted>        |                                       | 2024-12-13 12:36:51

and most importantly, my OIDC provider was returning an empty username to headscale (I am using authelia).
The UNIQUE constraint were then failing on creating a new user with users.name equals to ''.

After digging into the code of headscale, I found that the name (in the DB) is mapped with the username sent by the OIDC provider. And after adjusting the scopes in authelia, allowing profile and username I managed to create a new user account, with a valid Name, a Username and Email.

Bottom line : Destroying and recreating user is possible, but you have to ensure that you're allowing headscale to fetch the user username to avoid duplication of empty value.

It's working for v0.24 and 0.25

@SeaweedbrainCY commented on GitHub (Apr 2, 2025): For those having the same issue, I managed to solve the user creation error : My issue was that even with the migration flag on, my users had a user.name empty, ``` > headscale users list ID | Name | Username | Email | Created 1 | | <redacted> | | 2024-04-24 03:37:43 2 | | <redacted> | | 2024-04-25 19:20:32 3 | | <redacted> | | 2024-04-28 16:01:08 5 | | <redacted> | | 2024-05-04 15:11:28 6 | | <redacted> | | 2024-05-04 16:30:55 7 | | <redacted> | | 2024-12-13 12:36:51 ``` and **most importantly**, my OIDC provider was returning an empty **username** to headscale (I am using authelia). The UNIQUE constraint were then failing on creating a new user with users.name equals to `''`. After digging into the code of headscale, I found that the name (in the DB) is mapped with the `username` sent by the OIDC provider. And after adjusting the scopes in authelia, allowing `profile` and `username` I managed to create a new user account, with a valid `Name`, a `Username` and `Email`. Bottom line : Destroying and recreating user is possible, but you have to ensure that you're allowing headscale to fetch the user username to avoid duplication of empty value. It's working for v0.24 and 0.25
Author
Owner

@Subito commented on GitHub (Apr 8, 2025):

I'm facing the same problem. I tried the solution mentioned in #2505: https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims/#restore-functionality-prior-to-claims-parameter, but using the claims_policy just gives me the same error. Not using the claims_policy lets me login, but without a proper username.

After the successful login the username field is empty and I get successfully authenticated as https://auth.example.org/{long-uid} in the app. Afterwards it shows this URL instead of the username and none of my other devices.

The resulting "user" in the table looks like this:

id  created_at                          updated_at                           deleted_at  name  display_name  email  provider_identifier                                           provider  profile_pic_url
--  ----------------------------------  -----------------------------------  ----------  ----  ------------  -----  ------------------------------------------------------------  --------  ---------------
12  2025-04-08 17:38:38.82871823+02:00  2025-04-08 17:42:38.837385594+02:00                                         https://auth.{redacted}/4735db7e-eaed-4722-a3da-44b9d65df5e3  oidc

None of the other users (all use oidc) have the provider-field set.

Using Authelia 4.39.1 and Headscale 0.25.1 might be related to #2516?

@Subito commented on GitHub (Apr 8, 2025): I'm facing the same problem. I tried the solution mentioned in #2505: https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims/#restore-functionality-prior-to-claims-parameter, but using the `claims_policy` just gives me the same error. Not using the `claims_policy` lets me login, but without a proper username. After the successful login the username field is empty and I get `successfully authenticated as https://auth.example.org/{long-uid}` in the app. Afterwards it shows this URL instead of the username and none of my other devices. The resulting "user" in the table looks like this: ``` id created_at updated_at deleted_at name display_name email provider_identifier provider profile_pic_url -- ---------------------------------- ----------------------------------- ---------- ---- ------------ ----- ------------------------------------------------------------ -------- --------------- 12 2025-04-08 17:38:38.82871823+02:00 2025-04-08 17:42:38.837385594+02:00 https://auth.{redacted}/4735db7e-eaed-4722-a3da-44b9d65df5e3 oidc ``` None of the other users (all use oidc) have the `provider`-field set. Using Authelia 4.39.1 and Headscale 0.25.1 might be related to #2516?
Author
Owner

@ventsislav-georgiev commented on GitHub (May 5, 2025):

We are experiencing the same error: creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067) since updating to v0.24 for creation of new users / new OIDC logins. This error is returned to the user.

Updating to v0.25 does not help. The only thing that changes is the error returned to the user is now internal server error. Headscale still logs:

ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500

The only way we bypass it is to manually create the user with the cli: headscale users create .. and use v0.24, since this workaround does not work in v0.25..

This is using Google's OAuth 2.0: https://developers.google.com/identity/openid-connect/openid-connect, default scopes (openid, profile, email).

@ventsislav-georgiev commented on GitHub (May 5, 2025): We are experiencing the same error: `creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)` since updating to v0.24 for creation of new users / new OIDC logins. This error is returned to the user. Updating to v0.25 does not help. The only thing that changes is the error returned to the user is now internal server error. Headscale still logs: ``` ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500 ``` The only way we bypass it is to manually create the user with the cli: `headscale users create ..` and use v0.24, since this workaround does not work in v0.25.. This is using Google's OAuth 2.0: https://developers.google.com/identity/openid-connect/openid-connect, default scopes (openid, profile, email).
Author
Owner

@aronmolnar commented on GitHub (May 8, 2025):

We had a similar issue in 0.24.0-beta.2 with Entra ID. The login caused a new user to be created, however, without a user name.

> headscale users list
ID | Name             | Username    | Email | Created
5  | John Doe         |             |       | 2025-05-08 08:21:21

The next user who authenticated using SSO violated the UNIQUE constraint because of the empty user name. We could resolve it temporarily by adding a username to the existing user.

> headscale users rename -i 5 -r john
@aronmolnar commented on GitHub (May 8, 2025): We had a similar issue in `0.24.0-beta.2` with Entra ID. The login caused a new user to be created, however, without a user name. ``` > headscale users list ID | Name | Username | Email | Created 5 | John Doe | | | 2025-05-08 08:21:21 ``` The next user who authenticated using SSO violated the UNIQUE constraint because of the empty user name. We could resolve it temporarily by adding a username to the existing user. ``` > headscale users rename -i 5 -r john ```
Author
Owner

@cymonkey commented on GitHub (May 13, 2025):

Same problem here with Google OIDC, I can't even rename or update user since the error cannot edit OIDC user. Does anyone know any workaround? It's blocking my user registration.

@cymonkey commented on GitHub (May 13, 2025): Same problem here with Google OIDC, I can't even rename or update user since the error `cannot edit OIDC user`. Does anyone know any workaround? It's blocking my user registration.
Author
Owner

@joscdk commented on GitHub (May 14, 2025):

@cymonkey i ended up going into the SQLite database, and update name and updated_at in the users table to unblock user creation 🙈

@joscdk commented on GitHub (May 14, 2025): @cymonkey i ended up going into the SQLite database, and update `name` and `updated_at` in the `users` table to unblock user creation 🙈
Author
Owner

@cymonkey commented on GitHub (May 14, 2025):

Thanks @joscdk, tried and it worked, aw the only way I can think of at the this time too

@cymonkey commented on GitHub (May 14, 2025): Thanks @joscdk, tried and it worked, aw the only way I can think of at the this time too
Author
Owner

@ventsislav-georgiev commented on GitHub (May 14, 2025):

@kradalby In v0.26.0 it doesn't even create the user in the DB. The server just logs:
ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500

@joscdk @cymonkey what exactly did you need to do to workaround the user creation?

@ventsislav-georgiev commented on GitHub (May 14, 2025): @kradalby In v0.26.0 it doesn't even create the user in the DB. The server just logs: `ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500` @joscdk @cymonkey what exactly did you need to do to workaround the user creation?
Author
Owner

@cymonkey commented on GitHub (May 15, 2025):

@ventsislav-georgiev The users.name unique constraint still exists and the OIDC authen function somehow doesn't update the name when user reauthen/register, so it actually can still create one new user via OIDC authen, the user will has a users.name = null but any other user creation attempts after that will raise the error as you've seen.

All you need to do is to find the first created user via OIDC auth and update their users.name:

$ sqlite3 /var/lib/headscale/db.sqlite
# Or if you're running in a docker: sudo sqlite3 keys/db.sqlite

sqlite> SELECT * FROM users;
# Look for the user with name empty and provider = OIDC, take note the ID
sqlite> UPDATE users SET name='<unique_username>' WHERE id='222';

Not really a workaround tbh.

@cymonkey commented on GitHub (May 15, 2025): @ventsislav-georgiev The `users.name` unique constraint still exists and the OIDC authen function somehow doesn't update the name when user reauthen/register, so it actually can still create one new user via OIDC authen, the user will has a `users.name = null` but any other user creation attempts after that will raise the error as you've seen. All you need to do is to find the first created user via OIDC auth and update their `users.name`: ```sh $ sqlite3 /var/lib/headscale/db.sqlite # Or if you're running in a docker: sudo sqlite3 keys/db.sqlite sqlite> SELECT * FROM users; # Look for the user with name empty and provider = OIDC, take note the ID sqlite> UPDATE users SET name='<unique_username>' WHERE id='222'; ``` Not really a workaround tbh.
Author
Owner

@ventsislav-georgiev commented on GitHub (May 18, 2025):

Possible solution, which I tested working on our setup: https://github.com/juanfont/headscale/pull/2611

@ventsislav-georgiev commented on GitHub (May 18, 2025): Possible solution, which I tested working on our setup: https://github.com/juanfont/headscale/pull/2611
Author
Owner

@ventsislav-georgiev commented on GitHub (May 21, 2025):

With the help of @kradalby I found that my DB indexes were not up to date with what was expected in v0.26.

I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance.

My issue occurs with existing DB which was gradually updated from version 0.22.1 onward.

These were the indexes:
image

And it appears this migration hasn't been run or failed silently: d2879b2b36/hscontrol/db/db.go (L544)

despite having it in the migrations table:
image

After manually fixing my indexes by dumping the DB and running the following script:

-- 1. Create a new table without the UNIQUE constraint on "name"
CREATE TABLE users_new (
  id integer PRIMARY KEY,
  created_at datetime,
  updated_at datetime,
  deleted_at datetime,
  name text,
  display_name text,
  email text,
  provider_identifier text,
  provider text,
  profile_pic_url text
);

-- 2. Copy data
INSERT INTO users_new
SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url
FROM users;

-- 3. Drop old table
DROP TABLE users;

-- 4. Rename new table
ALTER TABLE users_new RENAME TO users;

-- 5. Re-create indices
CREATE INDEX idx_users_deleted_at ON users(deleted_at);
CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;
CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier);
CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;

All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login.

Here are the new users without usernames as listed by the CLI headscale users list:
Image

@ventsislav-georgiev commented on GitHub (May 21, 2025): With the help of @kradalby I found that my DB indexes were not up to date with what was expected in v0.26. I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance. My issue occurs with existing DB which was gradually updated from version 0.22.1 onward. These were the indexes: <img width="855" alt="image" src="https://github.com/user-attachments/assets/aa7db9b9-526e-43d7-aa9c-e4f4f44fffe3" /> And it appears this migration hasn't been run or failed silently: https://github.com/juanfont/headscale/blob/d2879b2b3675c6c38b1a39dd7a7b44c1679287a9/hscontrol/db/db.go#L544 despite having it in the migrations table: <img width="737" alt="image" src="https://github.com/user-attachments/assets/b5521dfa-6fa2-4507-b74e-cfc1bce26e37" /> After manually fixing my indexes by dumping the DB and running the following script: ``` -- 1. Create a new table without the UNIQUE constraint on "name" CREATE TABLE users_new ( id integer PRIMARY KEY, created_at datetime, updated_at datetime, deleted_at datetime, name text, display_name text, email text, provider_identifier text, provider text, profile_pic_url text ); -- 2. Copy data INSERT INTO users_new SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url FROM users; -- 3. Drop old table DROP TABLE users; -- 4. Rename new table ALTER TABLE users_new RENAME TO users; -- 5. Re-create indices CREATE INDEX idx_users_deleted_at ON users(deleted_at); CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL; CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier); CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL; ``` All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login. Here are the new users without usernames as listed by the CLI headscale users list: <img width="635" alt="Image" src="https://github.com/user-attachments/assets/a1490b1d-aa33-4778-bda5-149dc7f08d59" />
Author
Owner

@Subito commented on GitHub (May 28, 2025):

Thats a super helpful hint, thank you! Digging into my schema I found that it had some migrations missing, too! Unfortunately its not easy to find out which those are missing and what the sideeffects may be if I apply them now. I'm probably going to wipe my installation and create everything from scratch and see if that helps.

@Subito commented on GitHub (May 28, 2025): Thats a super helpful hint, thank you! Digging into my schema I found that it had some migrations missing, too! Unfortunately its not easy to find out which those are missing and what the sideeffects may be if I apply them now. I'm probably going to wipe my installation and create everything from scratch and see if that helps.
Author
Owner

@panteparak commented on GitHub (Jun 1, 2025):

I am getting a similar error too, with Azure EntraID

2025-06-01T15:44:14Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"AADSTS54005: OAuth2 Authorization code was already redeemed, please retry with a new valid code or use an existing refresh token. Trace ID: <TRACE_ID> Correlation ID: <CORR_ID> Timestamp: 2025-06-01 15:44:14Z\"" code=403

I have not tried some of the suggested method mentioned above.

@panteparak commented on GitHub (Jun 1, 2025): I am getting a similar error too, with Azure EntraID ``` 2025-06-01T15:44:14Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"AADSTS54005: OAuth2 Authorization code was already redeemed, please retry with a new valid code or use an existing refresh token. Trace ID: <TRACE_ID> Correlation ID: <CORR_ID> Timestamp: 2025-06-01 15:44:14Z\"" code=403 ``` I have not tried some of the suggested method mentioned above.
Author
Owner

@Subito commented on GitHub (Jun 2, 2025):

I got it working again by manually fixing the entries in the users table. Setting provider_identifier to the identifier I found in the Authelia-Logs and provider to oidc like this:

UPDATE users SET provider_identifier = 'https://auth.example.com/{some-uid}' WHERE id = 1;
UPDATE users SET provider = 'oidc' WHERE id = 1;

worked for me. Users can login again.

@Subito commented on GitHub (Jun 2, 2025): I got it working again by manually fixing the entries in the `users` table. Setting `provider_identifier` to the identifier I found in the Authelia-Logs and `provider` to `oidc` like this: ``` UPDATE users SET provider_identifier = 'https://auth.example.com/{some-uid}' WHERE id = 1; UPDATE users SET provider = 'oidc' WHERE id = 1; ``` worked for me. Users can login again.
Author
Owner

@hafometh88 commented on GitHub (Jun 15, 2025):

With the help of @kradalby I found that my DB indexes were not up to date with what was expected in v0.26.

I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance.

My issue occurs with existing DB which was gradually updated from version 0.22.1 onward.

These were the indexes: image

And it appears this migration hasn't been run or failed silently:

headscale/hscontrol/db/db.go

Line 544 in d2879b2

"CREATE UNIQUE INDEX IF NOT EXISTS idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;",
despite having it in the migrations table: image

After manually fixing my indexes by dumping the DB and running the following script:

-- 1. Create a new table without the UNIQUE constraint on "name"
CREATE TABLE users_new (
  id integer PRIMARY KEY,
  created_at datetime,
  updated_at datetime,
  deleted_at datetime,
  name text,
  display_name text,
  email text,
  provider_identifier text,
  provider text,
  profile_pic_url text
);

-- 2. Copy data
INSERT INTO users_new
SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url
FROM users;

-- 3. Drop old table
DROP TABLE users;

-- 4. Rename new table
ALTER TABLE users_new RENAME TO users;

-- 5. Re-create indices
CREATE INDEX idx_users_deleted_at ON users(deleted_at);
CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;
CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier);
CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;

All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login.

Here are the new users without usernames as listed by the CLI headscale users list: Image

Thank you, worked for me!

@hafometh88 commented on GitHub (Jun 15, 2025): > With the help of [@kradalby](https://github.com/kradalby) I found that my DB indexes were not up to date with what was expected in v0.26. > > I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance. > > My issue occurs with existing DB which was gradually updated from version 0.22.1 onward. > > These were the indexes: <img alt="image" width="855" src="https://private-user-images.githubusercontent.com/5616486/446044575-aa7db9b9-526e-43d7-aa9c-e4f4f44fffe3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDk5NDM2MjIsIm5iZiI6MTc0OTk0MzMyMiwicGF0aCI6Ii81NjE2NDg2LzQ0NjA0NDU3NS1hYTdkYjliOS01MjZlLTQzZDctYWE5Yy1lNGY0ZjQ0ZmZmZTMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDYxNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA2MTRUMjMyMjAyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODlhYTlhNmFiZjQ5Mjc4YWU3MWYzYmEzYTk2NjRhMjI2NzM5ZDczNWJhMzcxYzc0YjI1NDU2ZWQyMjVkZTc3YSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.oKodFRwWcRHPs__1Tm_TMBsknaBhdaxRw4R8DKaN9yU"> > > And it appears this migration hasn't been run or failed silently: > > [headscale/hscontrol/db/db.go](https://github.com/juanfont/headscale/blob/d2879b2b3675c6c38b1a39dd7a7b44c1679287a9/hscontrol/db/db.go#L544) > > Line 544 in [d2879b2](/juanfont/headscale/commit/d2879b2b3675c6c38b1a39dd7a7b44c1679287a9) > > "CREATE UNIQUE INDEX IF NOT EXISTS idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;", > despite having it in the migrations table: <img alt="image" width="737" src="https://private-user-images.githubusercontent.com/5616486/446047079-b5521dfa-6fa2-4507-b74e-cfc1bce26e37.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDk5NDM2MjIsIm5iZiI6MTc0OTk0MzMyMiwicGF0aCI6Ii81NjE2NDg2LzQ0NjA0NzA3OS1iNTUyMWRmYS02ZmEyLTQ1MDctYjc0ZS1jZmMxYmNlMjZlMzcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDYxNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA2MTRUMjMyMjAyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Mzg2YzBmNmRkODI1MjJiMmVkZTAyNTQyNDNlOGE5MjNiNGE3YjQ3ZGJiZDFjZmM3ODZlMjAzOGNjMjc2N2IxYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.OuCnO3AbajdpvoYJPhCpx6-7xoUnVv1p7bk9OhnfF8U"> > > After manually fixing my indexes by dumping the DB and running the following script: > > ``` > -- 1. Create a new table without the UNIQUE constraint on "name" > CREATE TABLE users_new ( > id integer PRIMARY KEY, > created_at datetime, > updated_at datetime, > deleted_at datetime, > name text, > display_name text, > email text, > provider_identifier text, > provider text, > profile_pic_url text > ); > > -- 2. Copy data > INSERT INTO users_new > SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url > FROM users; > > -- 3. Drop old table > DROP TABLE users; > > -- 4. Rename new table > ALTER TABLE users_new RENAME TO users; > > -- 5. Re-create indices > CREATE INDEX idx_users_deleted_at ON users(deleted_at); > CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL; > CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier); > CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL; > ``` > > All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login. > > Here are the new users without usernames as listed by the CLI headscale users list: <img alt="Image" width="635" src="https://private-user-images.githubusercontent.com/5616486/446154892-a1490b1d-aa33-4778-bda5-149dc7f08d59.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDk5NDM2MjIsIm5iZiI6MTc0OTk0MzMyMiwicGF0aCI6Ii81NjE2NDg2LzQ0NjE1NDg5Mi1hMTQ5MGIxZC1hYTMzLTQ3NzgtYmRhNS0xNDlkYzdmMDhkNTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDYxNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTA2MTRUMjMyMjAyWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Yzc3ZGI2NGMwYjljZWMxY2QwYmVjOTdmMzYxODgxODRjN2VlYTI5NTUwZTQxMDVjNTgzMGJiZGMxOGJkODJjNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.cXWuV62qaWc2zA0qi4IMG8tFkVq0qQE8-BOyeG-yAlU"> Thank you, worked for me!
Author
Owner

@kradalby commented on GitHub (Jul 2, 2025):

I'm working on this over in #2617 , more databases to add to the tests would be greatly appreciated. only schema helps, but with data would be even better.

I think we (@nblock) have a script for helping us randomise the data.

@kradalby commented on GitHub (Jul 2, 2025): I'm working on this over in #2617 , more databases to add to the tests would be greatly appreciated. only schema helps, but with data would be even better. I think we (@nblock) have a script for helping us randomise the data.
Author
Owner

@kradalby commented on GitHub (Jul 2, 2025):

As this comment is a bit confusing, I posted a lengthier one here: https://github.com/juanfont/headscale/issues/2597#issuecomment-3028234284

@kradalby commented on GitHub (Jul 2, 2025): As this comment is a bit confusing, I posted a lengthier one here: https://github.com/juanfont/headscale/issues/2597#issuecomment-3028234284
Author
Owner

@appleimperio commented on GitHub (Jul 17, 2025):

I'm getting the same issue but with Pocket-id. ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
These are my users

      1  |      | lrivera  |       | 2023-10-18 23:34:43
     2  |      | rrivera  |       | 2024-02-08 15:48:01

I just try the @ventsislav-georgiev and it worked

@appleimperio commented on GitHub (Jul 17, 2025): I'm getting the same issue but with Pocket-id. ```ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500``` These are my users ``` ID | Name | Username | Email | Created 1 | | lrivera | | 2023-10-18 23:34:43 2 | | rrivera | | 2024-02-08 15:48:01 ``` I just try the @ventsislav-georgiev and it worked
Author
Owner

@kradalby commented on GitHub (Sep 10, 2025):

I'm going to close this as fixed in the upcoming database cleanup for SQLite, no further work will be done for Postgres.

@kradalby commented on GitHub (Sep 10, 2025): I'm going to close this as fixed in the upcoming database cleanup for SQLite, no further work will be done for Postgres.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#970