[Bug] &#34;headscale node ls --tags&#34; does not list tags after v0.24.0-beta.1 upgrade #889

Originally created by @mhahl on GitHub (Dec 17, 2024).

Is this a support request?

This is not a support request

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

When running the command.

headscale node ls --tags

Shows

ID | Hostname        | Name               | MachineKey | NodeKey | User   | IP addresses                                             | Ephemeral | Last seen           | Expiration          | Connected | Expired | ForcedTags | InvalidTags | ValidTags
2  | app-srv-01      | syd-ts-rou         | [0DU0p]    | [hp9cC] | xxxxx  | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a   | false     | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online    | no      |            |             | 
3  | server1         | ncl-ts-rou         | [q8PuG]    | [qZxRu] | xxxxx  | 100.98.33.13, fd7a:115c:a1e0:1364:5d:63fb:6e57:afb0      | false     | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online    | no      |            |             | 
6  | localhost       | localhost          | [qR0EX]    | [/aHEX] | xxxxx  | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false     | 2024-12-17 08:08:03 | 2025-06-15 08:07:29 | offline   | no      |            |             |

Nothing is listed in the ValidTags, InvalidTags, ForcedTags columns.

Expected Behavior

When running the command.

headscale node ls --tags

I expect the nodes tags to be displayed in the ValidTags, InvalidTags, ForcedTags columns.

ID | Hostname        | Name               | MachineKey | NodeKey | User   | IP addresses                                             | Ephemeral | Last seen           | Expiration          | Connected | Expired | ForcedTags | InvalidTags | ValidTags
2  | app-srv-01      | syd-ts-rou         | [0DU0p]    | [hp9cC] | xxxxx  | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a   | false     | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online    | no      |            |             | tag:prod-aus-rou
3  | server1         | ncl-ts-rou         | [q8PuG]    | [qZxRu] | xxxxx  | 100.98.33.13, fd7a:115c:a1e0:1364:5d:63fb:6e57:afb0      | false     | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online    | no      |            |             | tag:prod-lan-sec-nvr
6  | localhost       | localhost          | [qR0EX]    | [/aHEX] | xxxxx  | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false     | 2024-12-17 08:08:03 | 2025-06-15 08:07:29 | offline   | no      |            |             |

Steps To Reproduce

Upgrade to v0.24.0-beta.
migrate all oidc users
restart tailscaled on nodes
run headscale nodes ls --tags

Environment

- OS: FreeBSD 14.1-RELEASE-p5
- Headscale version: v0.24.0-beta.1
- Tailscale version: 1.78.1 go version: go1.23.3
- Reverse Proxy: caddy

Runtime environment

Headscale is behind a (reverse) proxy
Headscale runs in a container

Anything else?

Even tho nothing is displayed in the output, ACLs seems to be working as if the tags were set correctly.

When running the command tailscale debug netmap I can see RequestTags being returned.

"RequestTags": [
        "tag:prod-aus-rou"
],

Running the following for a machine which I installed recently also does not return any tags.

curl -X 'GET'   'https://tailscale.sigaint.au/api/v1/node/20'   -H 'accept: application/json' -H 'Authorization: Bearer xxxxx'

Output:

{
  "node": {
    "id": "20",
    "machineKey": "mkey:xxx",
    "nodeKey": "nodekey:xxx",
    "discoKey": "discokey:xxx",
    "ipAddresses": [
      "100.87.117.196",
      "fd7a:115c:a1e0:ff4c:5f4f:a0a5:394d:b483"
    ],
    "name": "lan-mon-01",
    "user": {
      "id": "1",
      "name": "mhahl",
      "createdAt": "2024-11-04T21:33:47.127833898Z",
      "displayName": "Mark Hahl",
      "email": "mark.hahl@sigaint.au",
      "providerId": "https://auth.sigaint.au/realms/SIGAINT/4e754011-75e3-4c65-8c71-0265ff4c6e4c",
      "provider": "oidc",
      "profilePicUrl": ""
    },
    "lastSeen": "2024-12-17T08:26:21.434232205Z",
    "expiry": "2025-06-09T22:43:17.888225301Z",
    "preAuthKey": null,
    "createdAt": "2024-12-11T22:43:17.895004534Z",
    "registerMethod": "REGISTER_METHOD_OIDC",
    "forcedTags": [],
    "invalidTags": [],
    "validTags": [],
    "givenName": "lan-mon-01",
    "online": true
  }
}

Originally created by @mhahl on GitHub (Dec 17, 2024). ### Is this a support request? - [X] This is not a support request ### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior When running the command. ``` headscale node ls --tags ``` Shows ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired | ForcedTags | InvalidTags | ValidTags 2 | app-srv-01 | syd-ts-rou | [0DU0p] | [hp9cC] | xxxxx | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a | false | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online | no | | | 3 | server1 | ncl-ts-rou | [q8PuG] | [qZxRu] | xxxxx | 100.98.33.13, fd7a:115c:a1e0:1364:5d:63fb:6e57:afb0 | false | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online | no | | | 6 | localhost | localhost | [qR0EX] | [/aHEX] | xxxxx | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false | 2024-12-17 08:08:03 | 2025-06-15 08:07:29 | offline | no | | | ``` Nothing is listed in the `ValidTags`, `InvalidTags`, `ForcedTags` columns. ### Expected Behavior When running the command. ``` headscale node ls --tags ``` I expect the nodes tags to be displayed in the `ValidTags`, `InvalidTags`, `ForcedTags` columns. ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired | ForcedTags | InvalidTags | ValidTags 2 | app-srv-01 | syd-ts-rou | [0DU0p] | [hp9cC] | xxxxx | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a | false | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online | no | | | tag:prod-aus-rou 3 | server1 | ncl-ts-rou | [q8PuG] | [qZxRu] | xxxxx | 100.98.33.13, fd7a:115c:a1e0:1364:5d:63fb:6e57:afb0 | false | 2024-12-17 08:26:21 | 0001-01-01 00:00:00 | online | no | | | tag:prod-lan-sec-nvr 6 | localhost | localhost | [qR0EX] | [/aHEX] | xxxxx | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false | 2024-12-17 08:08:03 | 2025-06-15 08:07:29 | offline | no | | | ``` ### Steps To Reproduce 1. Upgrade to v0.24.0-beta. 2. migrate all oidc users 3. restart tailscaled on nodes 4. run `headscale nodes ls --tags` ### Environment ```markdown - OS: FreeBSD 14.1-RELEASE-p5 - Headscale version: v0.24.0-beta.1 - Tailscale version: 1.78.1 go version: go1.23.3 - Reverse Proxy: caddy ``` ### Runtime environment - [X] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Anything else? Even tho nothing is displayed in the output, ACLs seems to be working as if the tags were set correctly. When running the command `tailscale debug netmap` I can see `RequestTags` being returned. ``` "RequestTags": [ "tag:prod-aus-rou" ], ``` Running the following for a machine which I installed recently also does not return any tags. ``` curl -X 'GET' 'https://tailscale.sigaint.au/api/v1/node/20' -H 'accept: application/json' -H 'Authorization: Bearer xxxxx' ``` Output: ``` { "node": { "id": "20", "machineKey": "mkey:xxx", "nodeKey": "nodekey:xxx", "discoKey": "discokey:xxx", "ipAddresses": [ "100.87.117.196", "fd7a:115c:a1e0:ff4c:5f4f:a0a5:394d:b483" ], "name": "lan-mon-01", "user": { "id": "1", "name": "mhahl", "createdAt": "2024-11-04T21:33:47.127833898Z", "displayName": "Mark Hahl", "email": "mark.hahl@sigaint.au", "providerId": "https://auth.sigaint.au/realms/SIGAINT/4e754011-75e3-4c65-8c71-0265ff4c6e4c", "provider": "oidc", "profilePicUrl": "" }, "lastSeen": "2024-12-17T08:26:21.434232205Z", "expiry": "2025-06-09T22:43:17.888225301Z", "preAuthKey": null, "createdAt": "2024-12-11T22:43:17.895004534Z", "registerMethod": "REGISTER_METHOD_OIDC", "forcedTags": [], "invalidTags": [], "validTags": [], "givenName": "lan-mon-01", "online": true } } ```

adam added the bug label 2025-12-29 02:25:29 +01:00

adam closed this issue

adam commented

@kradalby commented on GitHub (Dec 17, 2024):

Thanks for the thorough writeup! are you able to test my fixes from here? https://github.com/juanfont/headscale/pull/2309

@kradalby commented on GitHub (Dec 17, 2024): Thanks for the thorough writeup! are you able to test my fixes from here? https://github.com/juanfont/headscale/pull/2309

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

Hey! Thanks for such a quick response. I have compiled your branch and tested the changes however it does not appear to have changed. The output is exactly the same.

I added some prints to the following lines which appear to only ever return an empty array.

diff --git a/hscontrol/grpcv1.go b/hscontrol/grpcv1.go
index 7daa8478..2f7360d3 100644
--- a/hscontrol/grpcv1.go
+++ b/hscontrol/grpcv1.go
@@ -489,6 +489,8 @@ func nodesToProto(polMan policy.PolicyManager, isLikelyConnected *xsync.MapOf[ty
                }
 
                tags := polMan.Tags(node)
+               log.Print("********************")
+               log.Print(tags)
                resp.ValidTags = lo.Uniq(append(tags, node.ForcedTags...))
                response[index] = resp

@mhahl commented on GitHub (Dec 18, 2024): Hey! Thanks for such a quick response. I have compiled your branch and tested the changes however it does not appear to have changed. The output is exactly the same. I added some prints to the following lines which appear to only ever return an empty array. ``` diff --git a/hscontrol/grpcv1.go b/hscontrol/grpcv1.go index 7daa8478..2f7360d3 100644 --- a/hscontrol/grpcv1.go +++ b/hscontrol/grpcv1.go @@ -489,6 +489,8 @@ func nodesToProto(polMan policy.PolicyManager, isLikelyConnected *xsync.MapOf[ty } tags := polMan.Tags(node) + log.Print("********************") + log.Print(tags) resp.ValidTags = lo.Uniq(append(tags, node.ForcedTags...)) response[index] = resp ``` ![image](https://github.com/user-attachments/assets/4dc904fd-9cb0-4fe2-bbbd-7a5aa259306e)

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

Just checking in the database db.sqlite looks like everything is there specifically the
RequestTags

sqlite> .mode ascii
sqlite> select host_info from nodes where id=20;

{
  "IPNVersion": "1.78.1-t8903926f7-gc4163954e",
  "BackendLogID": "c47d29ec30a00e447370a2ab0b42c1ff64a31244102894efddb6da50a70b69a4",
  "OS": "linux",
  "OSVersion": "6.11.4-301.fc41.x86_64",
  "Container": false,
  "Distro": "fedora",
  "DistroVersion": "41",
  "Desktop": false,
  "Hostname": "lan-mon-01",
  "NoLogsNoSupport": true,
  "Machine": "x86_64",
  "GoArch": "amd64",
  "GoArchVar": "v1",
  "GoVersion": "go1.23.3",
  "RequestTags": [
    "tag:prod-lan-dmz-mon"
  ],
  "NetInfo": {
    "MappingVariesByDestIP": false,
    "HairPinning": null,
    "WorkingIPv6": true,
    "OSHasIPv6": true,
    "WorkingUDP": true,
    "WorkingICMPv4": false,
    "UPnP": false,
    "PMP": false,
    "PCP": false,
    "PreferredDERP": 5,
    "DERPLatency": {
      "3-v4": 0.098853432,
      "3-v6": 0.099786139,
      "5-v4": 0.008345342,
      "5-v6": 0.008298199,
      "7-v4": 0.106848926
    },
    "FirewallMode": "ipt-default"
  },
  "Userspace": false,
  "UserspaceRouter": false,
  "AppConnector": false
}

@mhahl commented on GitHub (Dec 18, 2024): Just checking in the database `db.sqlite` looks like everything is there specifically the `RequestTags` ``` sqlite> .mode ascii sqlite> select host_info from nodes where id=20; { "IPNVersion": "1.78.1-t8903926f7-gc4163954e", "BackendLogID": "c47d29ec30a00e447370a2ab0b42c1ff64a31244102894efddb6da50a70b69a4", "OS": "linux", "OSVersion": "6.11.4-301.fc41.x86_64", "Container": false, "Distro": "fedora", "DistroVersion": "41", "Desktop": false, "Hostname": "lan-mon-01", "NoLogsNoSupport": true, "Machine": "x86_64", "GoArch": "amd64", "GoArchVar": "v1", "GoVersion": "go1.23.3", "RequestTags": [ "tag:prod-lan-dmz-mon" ], "NetInfo": { "MappingVariesByDestIP": false, "HairPinning": null, "WorkingIPv6": true, "OSHasIPv6": true, "WorkingUDP": true, "WorkingICMPv4": false, "UPnP": false, "PMP": false, "PCP": false, "PreferredDERP": 5, "DERPLatency": { "3-v4": 0.098853432, "3-v6": 0.099786139, "5-v4": 0.008345342, "5-v6": 0.008298199, "7-v4": 0.106848926 }, "FirewallMode": "ipt-default" }, "Userspace": false, "UserspaceRouter": false, "AppConnector": false } ```

adam commented

@kradalby commented on GitHub (Dec 18, 2024):

hmm interesting, the valid tags is a product of the RequestTags and the ACL, do you have your ACL? I need to see if I can write up a failing test because the test case we have is currently passing, and I'm not sure why.

@kradalby commented on GitHub (Dec 18, 2024): hmm interesting, the valid tags is a product of the `RequestTags` and the ACL, do you have your ACL? I need to see if I can write up a failing test because the test case we have is currently passing, and I'm not sure why.

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

I'll attach a copy of my ACLs in a moment. I'll try replicate from a fresh setup and perform the upgrade again.

@mhahl commented on GitHub (Dec 18, 2024): I'll attach a copy of my ACLs in a moment. I'll try replicate from a fresh setup and perform the upgrade again.

adam commented

@kradalby commented on GitHub (Dec 18, 2024):

Great, I just push another log line, so you can look for tags provided by policy in the logs and see if it finds them as authed or unauthed.

@kradalby commented on GitHub (Dec 18, 2024): Great, I just push another log line, so you can look for `tags provided by policy` in the logs and see if it finds them as authed or unauthed.

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

Here is the config.yaml and acls.json from headscale policy get with some of users removed.
config.zip

@mhahl commented on GitHub (Dec 18, 2024): Here is the config.yaml and acls.json from `headscale policy get` with some of users removed. [config.zip](https://github.com/user-attachments/files/18177178/config.zip)

adam commented

@kradalby commented on GitHub (Dec 18, 2024):

hmm, I added a small test for what I think should be roughly equivalent to your config here db1aa9ee4d, as I didnt test with the tag owner being a group, but it does look like it passes correctly.

I'll be driving and then getting back to this. Not sure what this could be or what has changed, could you look for the tags provided by policy log lines on a machine you can reproduce it on?

@kradalby commented on GitHub (Dec 18, 2024): hmm, I added a small test for what I think should be roughly equivalent to your config here https://github.com/juanfont/headscale/pull/2309/commits/db1aa9ee4d7cd07a25849ba4ec3f84a62e8deb2e, as I didnt test with the tag owner being a group, but it does look like it passes correctly. I'll be driving and then getting back to this. Not sure what this could be or what has changed, could you look for the `tags provided by policy` log lines on a machine you can reproduce it on?

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

Really appreciate the help, I will have to continue debugging tomorrow since its late now. I've attached some output that may or may not help and the logs you requested at the end.

From the server

$ headscale users list

ID | Name         | Username | Email                  | Created
1  | Mark Hahl    | mhahl    | mark.hahl@sigaint.au   | 2024-11-04 21:33:47
2  | xxxx xxxx    | xxxxxx   | xx.xxxxxxxx@sigaint.au | 2024-11-06 03:53:47
3  | admin1       | admin1   | admin.user@sigaint.au  | 2024-11-10 07:01:04

$ headscale nodes ls --tags
2024-12-18T20:03:55+11:00 DBG Setting timeout timeout=5000
2024-12-18T20:03:55+11:00 DBG HEADSCALE_CLI_ADDRESS environment is not set, connecting to unix socket. socket=/var/run/headscale/headscale.sock
ID | Hostname        | Name               | MachineKey | NodeKey | User   | IP addresses                                             | Ephemeral | Last seen           | Expiration          | Connected | Expired | ForcedTags | InvalidTags | ValidTags
1  | localhost       | mhahl-pixel-6      | [zHfG0]    | [NmQeb] | mhahl  | 100.119.178.179, fd7a:115c:a1e0:d4bf:d0e:e20a:45ec:e299  | false     | 2024-12-10 22:09:09 | 2025-05-03 21:33:47 | offline   | no      |            |             | 
2  | app-srv-01      | syd-ts-rou         | [0DU0p]    | [hp9cC] | mhahl  | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a   | false     | 2024-12-18 08:52:29 | 0001-01-01 00:00:00 | online    | no      |            |             | 
6  | localhost       | localhost          | [qR0EX]    | [/aHEX] | xxxxxx | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false     | 2024-12-17 08:57:33 | 2025-06-15 08:07:29 | offline   | no      |            |             | 
8  | localhost       | localhost-kjofzlld | [tiFu6]    | [t8Unp] | admin1 | 100.64.233.225, fd7a:115c:a1e0:23:7749:a35b:1732:3484    | false     | 2024-12-18 03:35:59 | 2025-05-09 07:01:04 | offline   | no      |            |             | 
9  | localhost       | localhost-hyyfvmil | [23vO6]    | [c3wTD] | admin1 | 100.108.140.205, fd7a:115c:a1e0:ba32:a4c:e2c8:da5:577c   | false     | 2024-12-18 08:52:29 | 2025-05-09 07:03:32 | online    | no      |            |             | 
10 | localhost       | localhost-pshydxog | [hbZNu]    | [r2dyY] | admin1 | 100.124.47.233, fd7a:115c:a1e0:62b1:29ed:e045:9d01:fcf3  | false     | 2024-12-18 09:02:32 | 2025-05-09 07:04:36 | online    | no      |            |             | 
11 | localhost       | localhost-lv43fl2o | [w7tsH]    | [H3Q19] | mhahl  | 100.64.194.75, fd7a:115c:a1e0:42c9:4630:b32a:1a2:50a8    | false     | 2024-12-16 20:03:39 | 2025-05-09 07:24:42 | offline   | no      |            |             | 
14 | frigate         | lan-nvr-01         | [JEAWt]    | [EgrIp] | mhahl  | 100.114.106.228, fd7a:115c:a1e0:53f4:a3dc:4f86:2e26:31cc | false     | 2024-12-18 08:52:29 | 2025-05-12 03:09:34 | online    | no      |            |             | 
15 | de-ts-rou       | de-ts-rou          | [U1gkP]    | [AfgUj] | mhahl  | 100.64.9.211, fd7a:115c:a1e0:3b59:5475:695e:45e1:2470    | false     | 2024-12-18 08:52:29 | 2025-05-20 03:06:28 | online    | no      |            |             | 
16 | localhost       | localhost-emf6qrjn | [xPTGC]    | [wGlr3] | mhahl  | 100.89.107.215, fd7a:115c:a1e0:5485:f295:896c:63ad:f471  | false     | 2024-12-10 10:46:33 | 2025-05-22 09:49:31 | offline   | no      |            |             | 
18 | localhost       | localhost-7400dglf | [YJfi6]    | [fPlNQ] | mhahl  | 100.67.92.100, fd7a:115c:a1e0:cde4:d677:d550:136c:1841   | false     | 2024-12-17 10:23:08 | 2024-12-17 10:23:08 | offline   | yes     |            |             | 
19 | lagoon          | lagoon             | [Ba8fF]    | [whA8W] | mhahl  | 100.96.206.185, fd7a:115c:a1e0:b35b:800b:db6e:f245:83c4  | false     | 2024-12-18 09:03:11 | 2025-06-15 07:16:11 | online    | no      |            |             | 
20 | lan-mon-01      | lan-mon-01         | [R5j6J]    | [Px6wg] | mhahl  | 100.87.117.196, fd7a:115c:a1e0:ff4c:5f4f:a0a5:394d:b483  | false     | 2024-12-18 08:52:29 | 2025-06-09 22:43:17 | online    | no      |            |             | 
21 | assistant       | assistant          | [YP4Lr]    | [XlmVV] | mhahl  | 100.117.128.183, fd7a:115c:a1e0:72f2:d4c9:fa12:6d05:a8d9 | false     | 2024-12-18 08:50:24 | 2025-06-10 07:41:33 | online    | no      |            |             | 
22 | huon            | huon               | [/ORbQ]    | [Mhc5z] | mhahl  | 100.102.163.89, fd7a:115c:a1e0:918c:6614:ed36:2b80:942e  | false     | 2024-12-16 20:58:49 | 2025-06-13 20:34:31 | offline   | no      |            |             | 
23 | lan-ts-rou      | lan-ts-rou         | [QNWX0]    | [b4RJu] | mhahl  | 100.81.130.81, fd7a:115c:a1e0:88b8:49f8:ba62:fbed:c213   | false     | 2024-12-18 08:52:29 | 2025-06-14 05:00:37 | online    | no      |            |             | 
24 | BLUE-IRIS-NSW02 | blue-iris-nsw02    | [xMM1l]    | [bd7D+] | admin1 | 100.82.191.229, fd7a:115c:a1e0:6eb4:5953:f773:6864:d42   | false     | 2024-12-18 08:52:29 | 2025-06-15 07:19:15 | online    | no      |            |             | 
25 | ncl-ts-rou      | ncl-ts-rou         | [q8PuG]    | [8M2wB] | admin1 | 100.123.168.215, fd7a:115c:a1e0:8aca:1c49:5bf7:b98:25da  | false     | 2024-12-18 09:03:38 | 2025-06-15 09:25:13 | online    | no      |            |             | 
26 | localhost       | localhost-pebrpc7k | [Hd+KZ]    | [WqO4I] | mhahl  | 100.80.149.158, fd7a:115c:a1e0:84e9:eeeb:63a3:8e30:cd9d  | false     | 2024-12-18 08:52:29 | 2025-06-15 10:17:20 | online    | no      |            |             | 
27 | localhost       | localhost-kupj2kko | [/yN7L]    | [8o6Z3] | mhahl  | 100.90.104.229, fd7a:115c:a1e0:e996:3093:7034:2629:753   | false     | 2024-12-18 08:52:29 | 2025-06-15 10:25:44 | online    | no      |            |             |

From my workstation:

(not sure why these are showing up as email and not the preferred_username from oidc?)

$ tailscale status

100.96.206.185  lagoon               mark.hahl@   freebsd -
100.117.128.183 assistant            mark.hahl@   linux   -
100.82.191.229  blue-iris-nsw02      admin.user@ windows -
100.64.9.211    de-ts-rou            mark.hahl@   freebsd idle; offers exit node
100.102.163.89  huon                 mark.hahl@   freebsd offline
100.87.117.196  lan-mon-01           mark.hahl@   linux   idle, tx 155248 rx 146484
100.114.106.228 lan-nvr-01           mark.hahl@   linux   active; direct 203.123.124.164:28703, tx 147917376 rx 3158742704
100.81.130.81   lan-ts-rou           mark.hahl@   freebsd idle; offers exit node, tx 373488684 rx 744300404
100.67.92.100   localhost-7400dglf   mark.hahl@   android offline
100.89.107.215  localhost-emf6qrjn   mark.hahl@   android offline
100.90.104.229  localhost-kupj2kko   mark.hahl@   android -
100.64.194.75   localhost-lv43fl2o   mark.hahl@   android offline
100.80.149.158  localhost-pebrpc7k   mark.hahl@   android -
100.119.178.179 mhahl-pixel-6        mark.hahl@   android offline
100.123.168.215 ncl-ts-rou           admin.user@  linux   idle; offers exit node
100.102.75.139  syd-ts-rou           mark.hahl@   freebsd idle; offers exit node

# Health check:
#     - exit status 1

Logs from the server.

daemon.log

adam commented

@kradalby commented on GitHub (Dec 18, 2024):

(not sure why these are showing up as email and not the preferred_username from oidc?)

Email takes precedence if available now, in the future, username will require an @, at least in the policy.

Actually, can you try to use your email in the group: definition in the Policy? it might be running into the same bug as #2300, I'll try to set up a test that should cover that.

If that work, you can use the email, right now this would be considered a regression as I didnt mean to push people towards emails yet.

@kradalby commented on GitHub (Dec 18, 2024): > (not sure why these are showing up as email and not the preferred_username from oidc?) Email takes precedence if available now, in the future, username will require an `@`, at least in the policy. Actually, can you try to use your email in the `group:` definition in the Policy? it might be running into the same bug as #2300, I'll try to set up a test that should cover that. If that work, you can use the email, right now this would be considered a regression as I didnt mean to push people towards emails yet.

adam commented

@mhahl commented on GitHub (Dec 18, 2024):

Changing the acls to use the email works correctly!

grep authorised_tags /var/log/daemon.log | tail

<30>1 2024-12-18T20:17:37.309514+11:00 app-srv-01.syd.sigaint.au headscale 89707 - - 2024-12-18T20:17:37+11:00 DBG tags provided by policy authorised_tags=["tag:prod-aus-rou"] node.id=2 unauthorised_tags=[]

$ headscale nodes list --tags

2024-12-18T20:17:44+11:00 DBG HEADSCALE_CLI_ADDRESS environment is not set, connecting to unix socket. socket=/var/run/headscale/headscale.sock
ID | Hostname        | Name               | MachineKey | NodeKey | User   | IP addresses                                             | Ephemeral | Last seen           | Expiration          | Connected | Expired | ForcedTags | InvalidTags | ValidTags
1  | localhost       | mhahl-pixel-6      | [zHfG0]    | [NmQeb] | mhahl  | 100.119.178.179, fd7a:115c:a1e0:d4bf:d0e:e20a:45ec:e299  | false     | 2024-12-10 22:09:09 | 2025-05-03 21:33:47 | offline   | no      |            |             | 
2  | app-srv-01      | syd-ts-rou         | [0DU0p]    | [hp9cC] | mhahl  | 100.102.75.139, fd7a:115c:a1e0:623d:eb1a:ee76:1a5:2d2a   | false     | 2024-12-18 09:17:26 | 0001-01-01 00:00:00 | online    | no      |            |             | tag:prod-aus-rou
6  | localhost       | localhost          | [qR0EX]    | [/aHEX] | admin1 | 100.101.179.189, fd7a:115c:a1e0:fd45:1c94:4bee:9b90:496d | false     | 2024-12-17 08:57:33 | 2025-06-15 08:07:29 | offline   | no      |            |             | 
8  | localhost       | localhost-kjofzlld | [tiFu6]    | [t8Unp] | admin1 | 100.64.233.225, fd7a:115c:a1e0:23:7749:a35b:1732:3484    | false     | 2024-12-18 03:35:59 | 2025-05-09 07:01:04 | offline   | no      |            |             | 
9  | localhost       | localhost-hyyfvmil | [23vO6]    | [c3wTD] | admin1 | 100.108.140.205, fd7a:115c:a1e0:ba32:a4c:e2c8:da5:577c   | false     | 2024-12-18 09:17:26 | 2025-05-09 07:03:32 | online    | no      |            |             | 
10 | localhost       | localhost-pshydxog | [hbZNu]    | [r2dyY] | admin1 | 100.124.47.233, fd7a:115c:a1e0:62b1:29ed:e045:9d01:fcf3  | false     | 2024-12-18 09:17:26 | 2025-05-09 07:04:36 | offline   | no      |            |             | 
11 | localhost       | localhost-lv43fl2o | [w7tsH]    | [H3Q19] | mhahl  | 100.64.194.75, fd7a:115c:a1e0:42c9:4630:b32a:1a2:50a8    | false     | 2024-12-16 20:03:39 | 2025-05-09 07:24:42 | offline   | no      |            |             | 
14 | frigate         | lan-nvr-01         | [JEAWt]    | [EgrIp] | mhahl  | 100.114.106.228, fd7a:115c:a1e0:53f4:a3dc:4f86:2e26:31cc | false     | 2024-12-18 09:17:26 | 2025-05-12 03:09:34 | online    | no      |            |             | tag:prod-lan-sec-nvr
15 | de-ts-rou       | de-ts-rou          | [U1gkP]    | [AfgUj] | mhahl  | 100.64.9.211, fd7a:115c:a1e0:3b59:5475:695e:45e1:2470    | false     | 2024-12-18 08:52:29 | 2025-05-20 03:06:28 | online    | no      |            |             | tag:prod-deu-rou
16 | localhost       | localhost-emf6qrjn | [xPTGC]    | [wGlr3] | mhahl  | 100.89.107.215, fd7a:115c:a1e0:5485:f295:896c:63ad:f471  | false     | 2024-12-10 10:46:33 | 2025-05-22 09:49:31 | offline   | no      |            |             | 
18 | localhost       | localhost-7400dglf | [YJfi6]    | [fPlNQ] | mhahl  | 100.67.92.100, fd7a:115c:a1e0:cde4:d677:d550:136c:1841   | false     | 2024-12-17 10:23:08 | 2024-12-17 10:23:08 | offline   | yes     |            |             | 
19 | lagoon          | lagoon             | [Ba8fF]    | [whA8W] | mhahl  | 100.96.206.185, fd7a:115c:a1e0:b35b:800b:db6e:f245:83c4  | false     | 2024-12-18 09:17:26 | 2025-06-15 07:16:11 | online    | no      |            |             | 
20 | lan-mon-01      | lan-mon-01         | [R5j6J]    | [Px6wg] | mhahl  | 100.87.117.196, fd7a:115c:a1e0:ff4c:5f4f:a0a5:394d:b483  | false     | 2024-12-18 09:17:26 | 2025-06-09 22:43:17 | online    | no      |            |             | tag:prod-lan-dmz-mon
21 | assistant       | assistant          | [YP4Lr]    | [XlmVV] | mhahl  | 100.117.128.183, fd7a:115c:a1e0:72f2:d4c9:fa12:6d05:a8d9 | false     | 2024-12-18 09:17:26 | 2025-06-10 07:41:33 | online    | no      |            |             | tag:prod-lan-shared
22 | huon            | huon               | [/ORbQ]    | [Mhc5z] | mhahl  | 100.102.163.89, fd7a:115c:a1e0:918c:6614:ed36:2b80:942e  | false     | 2024-12-16 20:58:49 | 2025-06-13 20:34:31 | offline   | no      |            |             | 
23 | lan-ts-rou      | lan-ts-rou         | [QNWX0]    | [b4RJu] | mhahl  | 100.81.130.81, fd7a:115c:a1e0:88b8:49f8:ba62:fbed:c213   | false     | 2024-12-18 09:17:26 | 2025-06-14 05:00:37 | online    | no      |            |             | tag:prod-aus-rou
24 | BLUE-IRIS-NSW02 | blue-iris-nsw02    | [xMM1l]    | [bd7D+] | admin1 | 100.82.191.229, fd7a:115c:a1e0:6eb4:5953:f773:6864:d42   | false     | 2024-12-18 09:17:26 | 2025-06-15 07:19:15 | online    | no      |            |             | 
25 | ncl-ts-rou      | ncl-ts-rou         | [q8PuG]    | [8M2wB] | admin1 | 100.123.168.215, fd7a:115c:a1e0:8aca:1c49:5bf7:b98:25da  | false     | 2024-12-18 09:13:40 | 2025-06-15 09:25:13 | online    | no      |            |             | 
26 | localhost       | localhost-pebrpc7k | [Hd+KZ]    | [WqO4I] | mhahl  | 100.80.149.158, fd7a:115c:a1e0:84e9:eeeb:63a3:8e30:cd9d  | false     | 2024-12-18 08:52:29 | 2025-06-15 10:17:20 | offline   | no      |            |             | 
27 | localhost       | localhost-kupj2kko | [/yN7L]    | [8o6Z3] | mhahl  | 100.90.104.229, fd7a:115c:a1e0:e996:3093:7034:2629:753   | false     | 2024-12-18 09:17:26 | 2025-06-15 10:25:44 | offline   | no      |            |             |

adam commented

@kradalby commented on GitHub (Dec 18, 2024):

ah awesome, that narrows down where I need to look, thank you, I will write a test so I can start figuring out what needs to be fixed.

@kradalby commented on GitHub (Dec 18, 2024): ah awesome, that narrows down where I need to look, thank you, I will write a test so I can start figuring out what needs to be fixed.

adam commented