[Bug] EntraID OIDC - ACLs not being applied to OIDC registered users #922

@kradalby commented on GitHub (Jan 26, 2025):

@SysAdminSmith

Could you give me some example output of your users?

Something like the output som headscale users list --output json ?

In headscale version 0.24.0, ACL worked by the user's email address (the username has no effect). If no email was defined, it worked on Provider ID (which was horrendously ugly having an entire URL in the ACL entries, but it did work). Now in 0.24.1, the user.name field gets populated with the email address but nothing seems to work in the ACL.

Are you saying there is different behaviour in 0.24.0 and 0.24.1 ? from entra, which didnt seem to send emails based on another issue, it would not expect that.

@GoodiesHQ Could you give me the same output but from 0.24.0 and 0.24.1?

@kradalby commented on GitHub (Jan 26, 2025): @SysAdminSmith Could you give me some example output of your users? Something like the output som `headscale users list --output json` ? > In headscale version 0.24.0, ACL worked by the user's email address (the username has no effect). If no email was defined, it worked on Provider ID (which was horrendously ugly having an entire URL in the ACL entries, but it did work). Now in 0.24.1, the user.name field gets populated with the email address but nothing seems to work in the ACL. Are you saying there is different behaviour in 0.24.0 and 0.24.1 ? from entra, which didnt seem to send emails based on another issue, it would not expect that. @GoodiesHQ Could you give me the same output but from 0.24.0 and 0.24.1?

adam commented

@SysAdminSmith commented on GitHub (Jan 26, 2025):

I just posted this in Discord in response to a suggestion I downgrade to 0.24.0 and manually change usernames - which auto populate with email names (user.name@domain.name) - to just user.name.

Sadly this did not work for me.
I downgraded to 0.24.0
I deleted an existing user.
Restarted Headscale.
Had the user sign in via OIDC. It was assigned a user.name1@domain.name username. I manually changed the username to user.name1.
Tailscale status on client node "shows" access to all nodes they should have access to but:
Magic DNS seems to be working but DNS, generally, does not work.
Changed ACL to accept all to all and users are able to get to WAN via exitnodes and navigate the non-Tailscale network via subnet routers.

Manually created users do not have any issues irrespective of the ACLs in place. My one non-OIDC user "node.mgr" has the same access as "user.name1@doman.name".

@kradalby

Could you give me some example output of your users?

Something like the output som headscale users list --output json ?

(Name, Display Name, and Tenant ID and sub claim are randomized)

This is in 0.24.0

[
	{
		"id": 1,
		"name": "user.name1",
		"created_at": {
			"seconds": 1737743188,
			"nanos": 738930533
		},
		"display_name": "User Name1",
		"provider_id": "https://login.microsoftonline.com/abcd1234-5678-90ef-ghij-klmnopqrstuv/v2.0/wXYZ9876-5432-10fe-dcba-lmnopqrstuvw",
		"provider": "oidc"
	},
	{
		"id": 2,
		"name": "node.mgr",
		"created_at": {
			"seconds": 1737743579,
			"nanos": 362789820
		}
	}
]

@SysAdminSmith commented on GitHub (Jan 26, 2025): I just posted this in Discord in response to a suggestion I downgrade to 0.24.0 and manually change usernames - which auto populate with email names (user.name@domain.name) - to just user.name. > > Sadly this did not work for me. > I downgraded to 0.24.0 > I deleted an existing user. > Restarted Headscale. > Had the user sign in via OIDC. It was assigned a user.name1@domain.name username. I manually changed the username to user.name1. > Tailscale status on client node "shows" access to all nodes they should have access to but: > Magic DNS seems to be working but DNS, generally, does not work. > Changed ACL to accept all to all and users are able to get to WAN via exitnodes and navigate the non-Tailscale network via subnet routers. > > Manually created users do not have any issues irrespective of the ACLs in place. My one non-OIDC user "node.mgr" has the same access as "user.name1@doman.name". @kradalby > > Could you give me some example output of your users? > > Something like the output som headscale users list --output json ? (Name, Display Name, and Tenant ID and sub claim are randomized) This is in 0.24.0 ``` [ { "id": 1, "name": "user.name1", "created_at": { "seconds": 1737743188, "nanos": 738930533 }, "display_name": "User Name1", "provider_id": "https://login.microsoftonline.com/abcd1234-5678-90ef-ghij-klmnopqrstuv/v2.0/wXYZ9876-5432-10fe-dcba-lmnopqrstuvw", "provider": "oidc" }, { "id": 2, "name": "node.mgr", "created_at": { "seconds": 1737743579, "nanos": 362789820 } } ] ```

adam commented

@Codelica commented on GitHub (Jan 27, 2025):

Curious your OIDC user is missing an "email" property, perhaps your IDP isn't including email_verified.

@Codelica commented on GitHub (Jan 27, 2025): Curious your OIDC user is missing an "email" property, perhaps your IDP isn't including `email_verified`.

adam commented

@Codelica commented on GitHub (Jan 27, 2025):

I am also experiencing this. In headscale version 0.24.0, ACL worked by the user's email address (the username has no effect). If no email was defined, it worked on Provider ID (which was horrendously ugly having an entire URL in the ACL entries, but it did work). Now in 0.24.1, the user.name field gets populated with the email address but nothing seems to work in the ACL.

This would match my experience.

In 0.24.0 with ODIC users that have plain usernames (fred, mike, etc) and email addresses (fred@domain.com, mike@domain.com, etc) populated, I have working user ACLs based on email address.

In 0.24.1, usernames get migrated to email addresses (as users login again), but I can't seem to get user ACLs to work based on email address.

@Codelica commented on GitHub (Jan 27, 2025): > I am also experiencing this. In headscale version 0.24.0, ACL worked by the user's email address (the username has no effect). If no email was defined, it worked on Provider ID (which was horrendously ugly having an entire URL in the ACL entries, but it did work). Now in 0.24.1, the `user.name` field gets populated with the email address but nothing seems to work in the ACL. This would match my experience. In 0.24.0 with ODIC users that have plain usernames (fred, mike, etc) and email addresses (fred@domain.com, mike@domain.com, etc) populated, I have working user ACLs based on email address. In 0.24.1, usernames get migrated to email addresses (as users login again), but I can't seem to get user ACLs to work based on email address.

adam commented

@SysAdminSmith commented on GitHub (Jan 27, 2025):

Curious your OIDC user is missing an "email" property, perhaps your IDP isn't including email_verified.

Do you mind expanding a bit on this?

@SysAdminSmith commented on GitHub (Jan 27, 2025): > Curious your OIDC user is missing an "email" property, perhaps your IDP isn't including `email_verified`. Do you mind expanding a bit on this?

adam commented

@Codelica commented on GitHub (Jan 27, 2025):

In 0.24.0 at least, user ACLs seem to key off email address, but your user with username user.name1 doesn't have any email address property set. I believe that happens when the email_verified claim isn't being sent by your IDP based on this comment.

My OIDC users do have the email property set like:

...
	{
		"id": 2,
		"name": "testuser",
		"created_at": {
			"seconds": 1737933527,
			"nanos": 133145801
		},
		"display_name": "Test User",
=>              "email": "testuser@domain.com",
		"provider_id": "https://auth.domain.com/304400930772615424",
		"provider": "oidc"
	}

And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems.

@Codelica commented on GitHub (Jan 27, 2025): In 0.24.0 at least, user ACLs seem to key off email address, but your user with username `user.name1` doesn't have any email address property set. I believe that happens when the `email_verified` claim isn't being sent by your IDP based on [this comment](https://github.com/juanfont/headscale/issues/2333#issuecomment-2601905707). My OIDC users do have the email property set like: ```json ... { "id": 2, "name": "testuser", "created_at": { "seconds": 1737933527, "nanos": 133145801 }, "display_name": "Test User", => "email": "testuser@domain.com", "provider_id": "https://auth.domain.com/304400930772615424", "provider": "oidc" } ``` And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems.

adam commented

@SysAdminSmith commented on GitHub (Jan 27, 2025):

In 0.24.0 at least, user ACLs seem to key off email address, but your user with username user.name1 doesn't have any email address property set. I believe that happens when the email_verified claim isn't being sent by your IDP based on this comment.

My OIDC users do have the email property set like:

...
{
"id": 2,
"name": "testuser",
"created_at": {
"seconds": 1737933527,
"nanos": 133145801
},
"display_name": "Test User",
=> "email": "testuser@domain.com",
"provider_id": "https://auth.domain.com/304400930772615424",
"provider": "oidc"
}

And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems.

Thank you :) I did some further research an apparently Entra ID does not provide email_verified by default. I'm looking into a solution

@SysAdminSmith commented on GitHub (Jan 27, 2025): > In 0.24.0 at least, user ACLs seem to key off email address, but your user with username `user.name1` doesn't have any email address property set. I believe that happens when the `email_verified` claim isn't being sent by your IDP based on [this comment](https://github.com/juanfont/headscale/issues/2333#issuecomment-2601905707). > > My OIDC users do have the email property set like: > > ... > { > "id": 2, > "name": "testuser", > "created_at": { > "seconds": 1737933527, > "nanos": 133145801 > }, > "display_name": "Test User", > => "email": "testuser@domain.com", > "provider_id": "https://auth.domain.com/304400930772615424", > "provider": "oidc" > } > > And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems. Thank you :) I did some further research an apparently Entra ID does not provide `email_verified` by default. I'm looking into a solution

adam commented

@SysAdminSmith commented on GitHub (Jan 27, 2025):

In 0.24.0 at least, user ACLs seem to key off email address, but your user with username user.name1 doesn't have any email address property set. I believe that happens when the email_verified claim isn't being sent by your IDP based on this comment.

My OIDC users do have the email property set like:

...
{
"id": 2,
"name": "testuser",
"created_at": {
"seconds": 1737933527,
"nanos": 133145801
},
"display_name": "Test User",
=> "email": "testuser@domain.com",
"provider_id": "https://auth.domain.com/304400930772615424",
"provider": "oidc"
}

And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems.

In 0.24.0 at least, user ACLs seem to key off email address, but your user with username user.name1 doesn't have any email address property set. I believe that happens when the email_verified claim isn't being sent by your IDP based on this comment.
My OIDC users do have the email property set like:
...
{
"id": 2,
"name": "testuser",
"created_at": {
"seconds": 1737933527,
"nanos": 133145801
},
"display_name": "Test User",
=> "email": "testuser@domain.com",
"provider_id": "https://auth.domain.com/304400930772615424",
"provider": "oidc"
}
And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems.

Thank you :) I did some further research an apparently Entra ID does not provide email_verified by default. I'm looking into a solution

I downgraded to 0.24.0 and went the UPN route since EntraID provides it by default (i.e. in the headscale config I set email_claim: "upn").

When I logged in, just the Display Name was populated; neither the username nor the email address was captured. I manually modified the user to create a username (user.name) which is referenced in the ACL.

No difference, sadly.

What is so damn odd is that when I set the endpoint to --accept-routes, --accept-dns, and --exit-node=exitnode

=== 'Use Tailscale DNS' status ===

Tailscale DNS: enabled.

Tailscale is configured to handle DNS queries on this device.
Run 'tailscale set --accept-dns=false' to revert to your system default DNS resolver.

=== MagicDNS configuration ===

This is the DNS configuration provided by the coordination server to this device.

MagicDNS: enabled tailnet-wide (suffix = dauntless.tail)

Other devices in your tailnet can reach this device at dd-0121-001l.dauntless.tail

Resolvers (in preference order):
  - 100.64.0.2

Split DNS Routes:

Search Domains:
  - dauntless.tail

=== System DNS configuration ===

This is the DNS configuration that Tailscale believes your operating system is using.
Tailscale may use this configuration if 'Override Local DNS' is disabled in the admin console,
or if no resolvers are provided by the coordination server.

  (reading the system DNS configuration is not supported on this platform)

[this is a preliminary version of this command; the output format may change in the future]

tailscale dns query google.com
DNS query for "google.com" (A) using internal resolver:

failed to query DNS: 500 Internal Server Error: resolving using "http://100.64.0.3:34152/dns-query": 403 Forbidden

For whatever reason, it is forwarding dns requests to the exitnode (100.64.0.3). The dns is configured at 100.64.0.2.

Further:

Jan 27 11:56:38 DD-0121-001L tailscaled[87856]: [RATELIMIT] format("dns udp query: %v")
Jan 27 11:56:46 DD-0121-001L tailscaled[87856]: [RATELIMIT] format("open-conn-track: flow %v %v > %v rejected due to %v") (6 dropped)
Jan 27 11:56:46 DD-0121-001L tailscaled[87856]: open-conn-track: flow TCP 100.64.0.1:43420 > 192.168.128.1:53 rejected due to acl
Jan 27 11:56:47 DD-0121-001L tailscaled[87856]: open-conn-track: flow TCP 100.64.0.1:43420 > 192.168.128.1:53 rejected due to acl

I'm not sure what its trying to do by sending to port 53 on a private ip. That is a DNS server but I am not sure how tailscale even knows about it.

@SysAdminSmith commented on GitHub (Jan 27, 2025): > In 0.24.0 at least, user ACLs seem to key off email address, but your user with username `user.name1` doesn't have any email address property set. I believe that happens when the `email_verified` claim isn't being sent by your IDP based on [this comment](https://github.com/juanfont/headscale/issues/2333#issuecomment-2601905707). > > My OIDC users do have the email property set like: > > ... > { > "id": 2, > "name": "testuser", > "created_at": { > "seconds": 1737933527, > "nanos": 133145801 > }, > "display_name": "Test User", > => "email": "testuser@domain.com", > "provider_id": "https://auth.domain.com/304400930772615424", > "provider": "oidc" > } > > And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems. > > In 0.24.0 at least, user ACLs seem to key off email address, but your user with username `user.name1` doesn't have any email address property set. I believe that happens when the `email_verified` claim isn't being sent by your IDP based on [this comment](https://github.com/juanfont/headscale/issues/2333#issuecomment-2601905707). > > My OIDC users do have the email property set like: > > ... > > { > > "id": 2, > > "name": "testuser", > > "created_at": { > > "seconds": 1737933527, > > "nanos": 133145801 > > }, > > "display_name": "Test User", > > => "email": "[testuser@domain.com](mailto:testuser@domain.com)", > > "provider_id": "https://auth.domain.com/304400930772615424", > > "provider": "oidc" > > } > > And the email address can be used for user based ACLs in 0.24.0. But not in 0.24.1 it seems. > > Thank you :) I did some further research an apparently Entra ID does not provide `email_verified` by default. I'm looking into a solution I downgraded to 0.24.0 and went the UPN route since EntraID provides it by default (i.e. in the headscale config I set ` email_claim: "upn"`). When I logged in, just the Display Name was populated; neither the username nor the email address was captured. I manually modified the user to create a username (user.name) which is referenced in the ACL. No difference, sadly. What is so damn odd is that when I set the endpoint to `--accept-routes`, `--accept-dns`, and `--exit-node=exitnode` ``` === 'Use Tailscale DNS' status === Tailscale DNS: enabled. Tailscale is configured to handle DNS queries on this device. Run 'tailscale set --accept-dns=false' to revert to your system default DNS resolver. === MagicDNS configuration === This is the DNS configuration provided by the coordination server to this device. MagicDNS: enabled tailnet-wide (suffix = dauntless.tail) Other devices in your tailnet can reach this device at dd-0121-001l.dauntless.tail Resolvers (in preference order): - 100.64.0.2 Split DNS Routes: Search Domains: - dauntless.tail === System DNS configuration === This is the DNS configuration that Tailscale believes your operating system is using. Tailscale may use this configuration if 'Override Local DNS' is disabled in the admin console, or if no resolvers are provided by the coordination server. (reading the system DNS configuration is not supported on this platform) [this is a preliminary version of this command; the output format may change in the future] ``` ``` tailscale dns query google.com DNS query for "google.com" (A) using internal resolver: failed to query DNS: 500 Internal Server Error: resolving using "http://100.64.0.3:34152/dns-query": 403 Forbidden ``` For whatever reason, it is forwarding dns requests to the exitnode (100.64.0.3). The dns is configured at 100.64.0.2. Further: ``` Jan 27 11:56:38 DD-0121-001L tailscaled[87856]: [RATELIMIT] format("dns udp query: %v") Jan 27 11:56:46 DD-0121-001L tailscaled[87856]: [RATELIMIT] format("open-conn-track: flow %v %v > %v rejected due to %v") (6 dropped) Jan 27 11:56:46 DD-0121-001L tailscaled[87856]: open-conn-track: flow TCP 100.64.0.1:43420 > 192.168.128.1:53 rejected due to acl Jan 27 11:56:47 DD-0121-001L tailscaled[87856]: open-conn-track: flow TCP 100.64.0.1:43420 > 192.168.128.1:53 rejected due to acl ``` I'm not sure what its trying to do by sending to port 53 on a private ip. That is a DNS server but I am not sure how tailscale even knows about it.

adam commented

@Codelica commented on GitHub (Jan 27, 2025):

Well AFAIK email address has to be used for user ACLs in 0.24.0, so without that set you may be out of luck there. And 0.24.1 doesn't seem work with email or username as far as I can tell.

Not sure about the dns issue, but without a working ACL maybe it's trying whatever it can to get some dns back through your exit node. Pure guess, seems odd. I know Tailscale does try to transparently upgrade you to DNS over HTTPS (DoH).

@Codelica commented on GitHub (Jan 27, 2025): Well AFAIK email address has to be used for user ACLs in 0.24.0, so without that set you may be out of luck there. And 0.24.1 doesn't seem work with email or username as far as I can tell. Not sure about the dns issue, but without a working ACL maybe it's trying whatever it can to get some dns back through your exit node. Pure guess, seems odd. I know Tailscale does try to transparently upgrade you to DNS over HTTPS (DoH).

adam commented

@SlackingVeteran commented on GitHub (Jan 27, 2025):

I have been fighting this ACL issue about a week now. And I finally found a repro case on my end

If a User's user.name field is updated ACL no longer works for any user. On 0.24.0 I had manually assigned each user a username and it broke my ACL, Once I upgraded to 0.24.1 ACL started working soon as server was up and it broke again once a user logged out and logged-in, which caused the user.name to be updated to include email.
Then I renamed user's user.name back to what it was, and ACL stayed broken. I then downgraded to 0.24.0 and it magically started working again.

It looks like update of the user.name field for any user breaks the ACL and it cannot be fixed until a minor version upgrade is done? Is there any database migration that occurs during version upgrade/downgrade? Because it only seems to get fixed after that

@SlackingVeteran commented on GitHub (Jan 27, 2025): I have been fighting this ACL issue about a week now. And I finally found a repro case on my end If a User's user.name field is updated ACL no longer works for any user. On 0.24.0 I had manually assigned each user a username and it broke my ACL, Once I upgraded to 0.24.1 ACL started working soon as server was up and it broke again once a user logged out and logged-in, which caused the user.name to be updated to include email. Then I renamed user's user.name back to what it was, and ACL stayed broken. I then downgraded to 0.24.0 and it magically started working again. It looks like update of the user.name field for any user breaks the ACL and it cannot be fixed until a minor version upgrade is done? Is there any database migration that occurs during version upgrade/downgrade? Because it only seems to get fixed after that

adam commented

@kradalby commented on GitHub (Jan 28, 2025):

@SlackingVeteran this is useful, can you show me your ACL?

Do you use your "old" usernames in the ACL or have you migrated to the what ends up being populated in the ACL (likely emails)?

@kradalby commented on GitHub (Jan 28, 2025): @SlackingVeteran this is useful, can you show me your ACL? Do you use your "old" usernames in the ACL or have you migrated to the what ends up being populated in the ACL (likely emails)?

adam commented

@SlackingVeteran commented on GitHub (Jan 28, 2025):

@kradalby I was using usernames in ACL on v0.24.0 which was not working, upgrading to 0.24.1 fixed it to a point where username started working until username was updated by OIDC to be email. I swapped to use email in ACL after it stopped working, and it still did not apply appropriate ACL, i then changed username from email to custom username and used username on ACL and that did not work either, downgraded to 0.24.0 and username started working again.
Following is my ACL on 0.24.0:

{
  "groups": {},
  "tagOwners": {},
  "hosts": {},
  "acls": [
    {
      "action": "accept",
      "src": [
        "user"
      ],
      "dst": [
        "user1:*"
      ]
    },
    {
      "action": "accept",
      "src": [
        "user2"
      ],
      "dst": [
        "user2:*"
      ]
    },
    {
      "action": "accept",
      "src": [
        "user3"
      ],
      "dst": [
        "user3:*"
      ]
    }
  ],
  "ssh": []
}

@SlackingVeteran commented on GitHub (Jan 28, 2025): @kradalby I was using usernames in ACL on v0.24.0 which was not working, upgrading to 0.24.1 fixed it to a point where username started working until username was updated by OIDC to be email. I swapped to use email in ACL after it stopped working, and it still did not apply appropriate ACL, i then changed username from email to custom username and used username on ACL and that did not work either, downgraded to 0.24.0 and username started working again. Following is my ACL on 0.24.0: ``` { "groups": {}, "tagOwners": {}, "hosts": {}, "acls": [ { "action": "accept", "src": [ "user" ], "dst": [ "user1:*" ] }, { "action": "accept", "src": [ "user2" ], "dst": [ "user2:*" ] }, { "action": "accept", "src": [ "user3" ], "dst": [ "user3:*" ] } ], "ssh": [] } ```

adam commented

@kradalby commented on GitHub (Jan 29, 2025):

@SlackingVeteran, you have to use the exact value present in the user object, it will check the Email and then the Name (username) field in order. So if it used to be stripped down from email user2@example.com to user2, it will now be user2@example.com as we do not strip them anymore.

When using OIDC, rename should not be used as it will be overwritten every time the user logs in, I filed #2387 to block the usage of it when OIDC is used to avoid confusion.

@kradalby commented on GitHub (Jan 29, 2025): @SlackingVeteran, you have to use the exact value present in the user object, it will check the `Email` and then the `Name` (username) field in order. So if it used to be stripped down from email user2@example.com to user2, it will now be user2@example.com as we do not strip them anymore. When using OIDC, rename should not be used as it will be overwritten every time the user logs in, I filed #2387 to block the usage of it when OIDC is used to avoid confusion.

adam commented

@SlackingVeteran commented on GitHub (Jan 29, 2025):

Thanks @kradalby

@SlackingVeteran commented on GitHub (Jan 29, 2025): Thanks @kradalby

adam commented

@Codelica commented on GitHub (Jan 29, 2025):

@kradalby I started fresh again thinking I may have made some mistake, but I'm afraid I can't get OIDC user based ACLs to work as you describe in 0.24.1.

Starting in a working state in 0.24.0 (note not .1) where I have 1 user (which is stable between login/logout as usernames can't be email addresses yet in 0.24.0):

root@cloud:~ # headscale users list
ID | Name         | Username | Email                 | Created
1  | Fred Wilson  | fred     | fred@domain.com       | 2025-01-21 22:31:18

And a very simple ACL:

{
  "groups": {},
  "tagOwners": {},
  "hosts": {},
  "acls": [
    {
      "action": "accept",
      "src": [
        "fred@domain.com"
      ],
      "dst": [
        "*:*"
      ]
    }
  ],
  "ssh": []
}

It works as expected, the user fred (fred@domain.com) sees and can reach everything.

Then I update the system to 0.24.1 -- initially the ACL still works fine.

But the second fred (fred@domain.com) does a logout/login, his Headscale username is updated to email as expected:

root@cloud:~ # headscale users list
ID | Name         | Username           | Email                 | Created
1  | Fred Wilson  | fred@domain.com    | fred@domain.com       | 2025-01-21 22:31:18

But the ACL no longer works, with the following logged to the Headscale console:

2025-01-29T12:19:12-07:00 WRN No IPs found with the alias fred@domain.com

Which according to headscale nodes list isn't true. fred@domain.com is shown as the user on his 4 nodes.

If I try editing the ACL using his email fred@domain.com, Headscale will log WRN No IPs found with the alias fred@domain.com each time it's applied.

The only thing that seems to work for me to identify his user in the ACL is using his provider_id (which I tried based on some commits I was looking at).

Is there some specific output that would help investigate this?

@Codelica commented on GitHub (Jan 29, 2025): @kradalby I started fresh again thinking I may have made some mistake, but I'm afraid I can't get OIDC user based ACLs to work as you describe in 0.24.1. Starting in a working state in **0.24.0** (note not .1) where I have 1 user (which is stable between login/logout as usernames can't be email addresses yet in 0.24.0): ```bash root@cloud:~ # headscale users list ID | Name | Username | Email | Created 1 | Fred Wilson | fred | fred@domain.com | 2025-01-21 22:31:18 ``` And a very simple ACL: ```json { "groups": {}, "tagOwners": {}, "hosts": {}, "acls": [ { "action": "accept", "src": [ "fred@domain.com" ], "dst": [ "*:*" ] } ], "ssh": [] } ``` It works as expected, the user fred (fred@domain.com) sees and can reach everything. Then I update the system to **0.24.1** -- initially the ACL still works fine. But the second fred (fred@domain.com) does a logout/login, his Headscale username is updated to email as expected: ```bash root@cloud:~ # headscale users list ID | Name | Username | Email | Created 1 | Fred Wilson | fred@domain.com | fred@domain.com | 2025-01-21 22:31:18 ``` But the ACL no longer works, with the following logged to the Headscale console: `2025-01-29T12:19:12-07:00 WRN No IPs found with the alias fred@domain.com` Which according to `headscale nodes list` isn't true. fred@domain.com is shown as the user on his 4 nodes. If I try editing the ACL using his email fred@domain.com, Headscale will log `WRN No IPs found with the alias fred@domain.com` each time it's applied. The **only** thing that seems to work for me to identify his user in the ACL is using his `provider_id` (which I tried based on some commits I was looking at). Is there some specific output that would help investigate this?

adam commented

@Codelica commented on GitHub (Jan 29, 2025):

I think the problem may be here.

Once the user does a logout/login in 0.24.1 both user.Email and user.Name will match their email address, which would then hit the multiple user check every time?

Edit: I just ran with trace logging and that is what is hitting for me:

2025-01-29T13:16:37-07:00 TRC home/runner/work/headscale/headscale/hscontrol/policy/acls.go:1043 > could not determine user to filter nodes by error="multiple users with token \"fred@domain.com\" found: no user matching"

@Codelica commented on GitHub (Jan 29, 2025): I think the problem *may* be [here](https://github.com/juanfont/headscale/blob/2c279e0a7bd8eefba83169a3d98554819c47dc80/hscontrol/policy/acls.go#L1075). Once the user does a logout/login in 0.24.1 both `user.Email` and `user.Name` will match their email address, which would then hit the [multiple user check](https://github.com/juanfont/headscale/blob/2c279e0a7bd8eefba83169a3d98554819c47dc80/hscontrol/policy/acls.go#L1087) every time? ----- **Edit:** I just ran with trace logging and that is what is hitting for me: `2025-01-29T13:16:37-07:00 TRC home/runner/work/headscale/headscale/hscontrol/policy/acls.go:1043 > could not determine user to filter nodes by error="multiple users with token \"fred@domain.com\" found: no user matching"`

adam commented

@SlackingVeteran commented on GitHub (Jan 29, 2025):

Great find @Codelica, I didn't update to 0.24.1 yet because of this issue and looks like you found the cause, might be worth creating a PR. A else if there should fix the issue or continue inside first if block

@SlackingVeteran commented on GitHub (Jan 29, 2025): Great find @Codelica, I didn't update to 0.24.1 yet because of this issue and looks like you found the cause, might be worth creating a PR. A `else if` there should fix the issue or `continue` inside first if block

adam commented

@Codelica commented on GitHub (Jan 29, 2025):

Unfortunately I'm not a Go dev :(, although it would be the one language I'd love to pick up if there's ever time. (seems doubtful) I have a feeling he'll want to preserve a check for legitimate multiple matches though -- just checking based on multiple unique users ids instead.

@Codelica commented on GitHub (Jan 29, 2025): Unfortunately I'm not a Go dev :(, although it would be the one language I'd love to pick up if there's ever time. (seems doubtful) I have a feeling he'll want to preserve a check for legitimate multiple matches though -- just checking based on multiple unique users ids instead.

adam commented

@kradalby commented on GitHub (Jan 29, 2025):

Once the user does a logout/login in 0.24.1 both user.Email and user.Name will match their email address, which would then hit the multiple user check every time?

I think this is it!, good catch!

I have actually redone that function here https://github.com/juanfont/headscale/pull/2388, a bit by accident but it should fix that problem.
Can you test that branch?

@kradalby commented on GitHub (Jan 29, 2025): > Once the user does a logout/login in 0.24.1 both `user.Email` and `user.Name` will match their email address, which would then hit the [multiple user check](https://github.com/juanfont/headscale/blob/2c279e0a7bd8eefba83169a3d98554819c47dc80/hscontrol/policy/acls.go#L1087) every time? I think this is it!, good catch! I have actually redone that function here https://github.com/juanfont/headscale/pull/2388, a bit by accident but it should fix that problem. Can you test that branch?

adam commented

@SlackingVeteran commented on GitHub (Jan 30, 2025):

I would test it but my Kubernetes deployment relies on pre-built docker images. If there is an image for it I can test it out right away

@SlackingVeteran commented on GitHub (Jan 30, 2025): I would test it but my Kubernetes deployment relies on pre-built docker images. If there is an image for it I can test it out right away

adam commented

@SlackingVeteran commented on GitHub (Jan 30, 2025):

I built the docker container myself to test this and can verify email for OIDC logged in user is working as expected in ACL now

@SlackingVeteran commented on GitHub (Jan 30, 2025): I built the docker container myself to test this and can verify email for OIDC logged in user is working as expected in ACL now

adam commented

@SysAdminSmith commented on GitHub (Jan 30, 2025):

I updated to 0.24.2 and unforuntately I still am having issues. Am I right to believe that all users must have an email address, now? Because, as mentioned previously, EntraID does not provide email by default in a way that the email field is filled. The username will be filled by the email EntraID provides, but the email field will remain empty (user.name is retrieved via EntraID; node.mgr was a user created locally):

ID | Name          | Username                             | Email | Created            
2  |               | node.mgr                             |       | 2025-01-24 18:32:59
8  | User Name     | user.name@foo.bar                    |       | 2025-01-30 15:53:45

@SysAdminSmith commented on GitHub (Jan 30, 2025): I updated to 0.24.2 and unforuntately I still am having issues. Am I right to believe that all users must have an email address, now? Because, as mentioned previously, EntraID does not provide email by default in a way that the email field is filled. The username will be filled by the email EntraID provides, but the email field will remain empty (user.name is retrieved via EntraID; node.mgr was a user created locally): ``` ID | Name | Username | Email | Created 2 | | node.mgr | | 2025-01-24 18:32:59 8 | User Name | user.name@foo.bar | | 2025-01-30 15:53:45 ```

adam commented

@suyashFSG commented on GitHub (Jan 30, 2025):

user.name will have email once user logs out and logs back in with OIDC. Account stays unchanged until user re-logins

@suyashFSG commented on GitHub (Jan 30, 2025): user.name will have email once user logs out and logs back in with OIDC. Account stays unchanged until user re-logins

adam commented

@suyashFSG commented on GitHub (Jan 30, 2025):

As for email field, it will stay empty unless OIDC claim contains email_verified claim with true value

@suyashFSG commented on GitHub (Jan 30, 2025): As for email field, it will stay empty unless OIDC claim contains email_verified claim with `true` value

adam commented

@Codelica commented on GitHub (Jan 30, 2025):

@SysAdminSmith Code in 0.24.2 checks the Username and Email fields you see in that table (user.Email and user.Name in the actual user object).

So you should be able to reference those 2 users as "node.mgr" and "user.name@foo.bar" even with their Email field empty.

@Codelica commented on GitHub (Jan 30, 2025): @SysAdminSmith [Code](https://github.com/kradalby/headscale/blob/209093f5ab3b143b82b68abc81706f3a7af07211/hscontrol/policy/acls.go#L1073) in 0.24.2 checks the `Username` and `Email` fields you see in that table (user.Email and user.Name in the actual user object). So you _should_ be able to reference those 2 users as "node.mgr" and "user.name@foo.bar" even with their `Email` field empty.

adam commented

@suyashFSG commented on GitHub (Jan 30, 2025):

I use Entra as well and I have added additional claim on entra that has true value. You should be able to follow this instruction on Headscale discord to do so https://discord.com/channels/896711691637780480/1105842846386356294/1331037795900461106

@suyashFSG commented on GitHub (Jan 30, 2025): I use Entra as well and I have added additional claim on entra that has true value. You should be able to follow this instruction on Headscale discord to do so https://discord.com/channels/896711691637780480/1105842846386356294/1331037795900461106

adam commented

@SysAdminSmith commented on GitHub (Jan 30, 2025):

I use Entra as well and I have added additional claim on entra that has true value. You should be able to follow this instruction on Headscale discord to do so https://discord.com/channels/896711691637780480/1105842846386356294/1331037795900461106

Thank you for this! What "scopes" did you use in your config.yaml? Also, did you declare a separate "email_claim" in your config.yaml?

@SysAdminSmith commented on GitHub (Jan 30, 2025): > I use Entra as well and I have added additional claim on entra that has true value. You should be able to follow this instruction on Headscale discord to do so https://discord.com/channels/896711691637780480/1105842846386356294/1331037795900461106 Thank you for this! What "scopes" did you use in your config.yaml? Also, did you declare a separate "email_claim" in your config.yaml?

adam commented

@suyashFSG commented on GitHub (Jan 30, 2025):

My scope isopenid profile email and on Entra i had to fill every user’s Email field under Properties -> Contact Information, wish it was possible to define what claim to use for email instead. Because email for Entra is in UPN and preferred_username by default

@suyashFSG commented on GitHub (Jan 30, 2025): My scope is`openid profile email` and on Entra i had to fill every user’s Email field under `Properties -> Contact Information`, wish it was possible to define what claim to use for email instead. Because email for Entra is in UPN and preferred_username by default

adam commented