[Bug] Group-based ACL not working for local users #1043

New Issue

adam · 2025-12-29T02:27:53+01:00

adam commented

2025-12-29 02:27:53 +01:00

Originally created by @maprambo on GitHub (Jun 4, 2025).

Is this a support request?

This is not a support request

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

In my acl, I have mutliple groups. All of the usernames end in an @, like these:

"group:/Benutzer": ["maprambo@"],
"group:server": ["prod@"],

The first entry and all its rules are respected and they work, the second group is used in the same rules and does not work. There is a difference between these users:
maprambo is from OIDC with the preferred_username="maprambo"
prod is a local user with the username="prod" (both without @)

Both look the same in the users table:

ID	Name	Username	Email	Created
3		prod		2023-09-14 21:32:34
11	Full Name	maprambo		2025-05-14 18:21:25

The hosts from the user prod accept the rules when they are assigned a tag that is also listed as a source (see below) -- but also only after a logout and login.

Expected Behavior

Both groups and their rules should be working, no matter if the user is from OIDC or local

Steps To Reproduce

Create one local user
Create one OIDC user
Add both users to a group
Use the group in ACL

Environment

- OS: docker image
- Headscale version: 0.26.0
- Tailscale version: 1.84.0

Runtime environment

Headscale is behind a (reverse) proxy
Headscale runs in a container

Debug information

ACL:

{
	"groups": {
            "group:/Benutzer": ["maprambo@"],
            "group:server": ["prod@"],
        },
	"tagOwners": {
		"tag:benutzer-router": ["group:/Administrator/Netzwerk"],
	},
	"Hosts": {
		"revproxy.internal": "10.7.2.7/32",
	},
	"acls": [
		{
		  "action": "accept",
		  "src": [
				"group:server",
				"tag:benutzer-router", // <-- with this tag assigned, the nodes from the group server get this rule and its route
				"group:/Benutzer"
			],
		  "dst": [
				"revproxy.internal:80,443,3023",
				"tag:int-revproxy:80,443,3023"
		  ],
		},
	],
}

Originally created by @maprambo on GitHub (Jun 4, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior In my acl, I have mutliple groups. All of the usernames end in an @, like these: ``` "group:/Benutzer": ["maprambo@"], "group:server": ["prod@"], ``` The first entry and all its rules are respected and they work, the second group is used in the same rules and does not work. There is a difference between these users: maprambo is from OIDC with the preferred_username="maprambo" prod is a local user with the username="prod" (both without @) Both look the same in the users table: ID | Name | Username | Email | Created -- | ----------------- | ------------------- | -------- | ------------ 3 | | prod | | 2023-09-14 21:32:34 11 | Full Name | maprambo | | 2025-05-14 18:21:25 The hosts from the user prod accept the rules when they are assigned a tag that is also listed as a source (see below) -- but also only after a logout and login. ### Expected Behavior Both groups and their rules should be working, no matter if the user is from OIDC or local ### Steps To Reproduce 1. Create one local user 2. Create one OIDC user 3. Add both users to a group 4. Use the group in ACL ### Environment ```markdown - OS: docker image - Headscale version: 0.26.0 - Tailscale version: 1.84.0 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information ACL: ``` { "groups": { "group:/Benutzer": ["maprambo@"], "group:server": ["prod@"], }, "tagOwners": { "tag:benutzer-router": ["group:/Administrator/Netzwerk"], }, "Hosts": { "revproxy.internal": "10.7.2.7/32", }, "acls": [ { "action": "accept", "src": [ "group:server", "tag:benutzer-router", // <-- with this tag assigned, the nodes from the group server get this rule and its route "group:/Benutzer" ], "dst": [ "revproxy.internal:80,443,3023", "tag:int-revproxy:80,443,3023" ], }, ], } ```

adam added the bug no-stale-bot policy 📝 labels 2025-12-29 02:27:53 +01:00

adam closed this issue

2025-12-29 02:27:54 +01:00

adam commented

2025-12-29 02:27:54 +01:00

@zingmars commented on GitHub (Jun 10, 2025):

Ran into the same issue, and I'm using local users exclusively (so it probably has nothing to do with OICD users being present). Using tags for the time being works, but any acl rule that involves groups containing local users in the source doesn't seem to be functional. Unfortunately, since the upgrade seems to have nuked some tables from the database it's impossible to downgrade to 0.25.

@zingmars commented on GitHub (Jun 10, 2025): Ran into the same issue, and I'm using local users exclusively (so it probably has nothing to do with OICD users being present). Using tags for the time being works, but any acl rule that involves groups containing local users in the source doesn't seem to be functional. Unfortunately, since the upgrade seems to have nuked some tables from the database it's impossible to downgrade to 0.25.

adam commented

2025-12-29 02:27:54 +01:00

@andreyrd commented on GitHub (Jun 10, 2025):

I think we might be running into this as well, but it's very hard to debug because everything sometimes works, sometimes doesn't. A ton of errors in tailscaled like open-conn-track: timeout opening (TCP xx => xx) to node [xx]; online=yes, lastRecv=2h19m53s.

@andreyrd commented on GitHub (Jun 10, 2025): I think we might be running into this as well, but it's very hard to debug because everything sometimes works, sometimes doesn't. A ton of errors in tailscaled like `open-conn-track: timeout opening (TCP xx => xx) to node [xx]; online=yes, lastRecv=2h19m53s`.

adam commented

2025-12-29 02:27:54 +01:00

@rainbend commented on GitHub (Jun 13, 2025):

I encountered the same issue with headscale 0.26.1. After updating the machine's tag, the new rules only take effect on the client after restarting headscale.

@rainbend commented on GitHub (Jun 13, 2025): I encountered the same issue with headscale 0.26.1. After updating the machine's tag, the new rules only take effect on the client after restarting headscale.

adam commented

2025-12-29 02:27:55 +01:00

@aritas1 commented on GitHub (Jun 28, 2025):

maybe this is also a related problem here, running v0.26.1:

we now (need to) use the group ACL feature after the authentication-flow rework in v0.25.0

only OIDC users are in our groups, there are no local users anywhere in the ACL

we sometimes see similar issues with ACLs not applying to users:
it’s somewhat random whether a group membership, and any connected ACL, gets populated on node connection into the compiled filter / matchers

checked via the policy-manager debug endpoint:
curl -s "http://172.16.0.53:9090/debug/policy-manager?debugkey=xxxxxxx"

we’ve been debugging this for 2 days now, but no clear pattern emerges with node last-connected, user logins, or node-key validity, noting seems to correlate atm.

@aritas1 commented on GitHub (Jun 28, 2025): maybe this is also a related problem here, running v0.26.1: we now (need to) use the group ACL feature after the authentication-flow rework in [v0.25.0](https://github.com/juanfont/headscale/releases/tag/v0.25.0) only OIDC users are in our groups, there are no local users anywhere in the ACL we sometimes see similar issues with ACLs not applying to users: it’s somewhat random whether a group membership, and any connected ACL, gets populated on node connection into the `compiled filter` / `matchers` checked via the policy-manager debug endpoint: `curl -s "http://172.16.0.53:9090/debug/policy-manager?debugkey=xxxxxxx"` we’ve been debugging this for 2 days now, but no clear pattern emerges with node last-connected, user logins, or node-key validity, noting seems to correlate atm.

adam commented

2025-12-29 02:27:55 +01:00

@aritas1 commented on GitHub (Jul 4, 2025):

Found a correlation for my problem:
If any (forced?) tag is present on a node, the group policies from the node’s user will not be populated on this node.

=> Don’t mix groups and tags, for now?

@aritas1 commented on GitHub (Jul 4, 2025): Found a correlation for my problem: If any (forced?) tag is present on a node, the group policies from the node’s user will not be populated on this node. => Don’t mix groups and tags, for now?

adam commented

2025-12-29 02:27:55 +01:00

@nonanonymousanon commented on GitHub (Jul 23, 2025):

Can confirm what @aritas1 noted. That behavior is in line with tailscale docs, see here for tagging (https://tailscale.com/kb/1068/tags#use-cases). Applying a tag to a device removes any user-based authentication so if you tag the device you are connecting from then the ACLs won't see the user tied to the device and any group policy which includes the user will be rendered effectively useless for that user on that device. As @rainbend noted once you remove the tags you will need to restart your headscale instance. I was able to get user/group based ACLs working properly after removing tags from the device I was connecting from. It doesn't exactly work how you would expect with tagging but thats a tailscale issue not headscale.

@nonanonymousanon commented on GitHub (Jul 23, 2025): Can confirm what @aritas1 noted. That behavior is in line with tailscale docs, see here for tagging (https://tailscale.com/kb/1068/tags#use-cases). _Applying a tag to a device removes any user-based authentication_ so if you tag the device you are connecting **from** then the ACLs won't see the user tied to the device and any group policy which includes the user will be rendered effectively useless for that user on that device. As @rainbend noted once you remove the tags you will need to restart your headscale instance. I was able to get user/group based ACLs working properly after removing tags from the device I was connecting from. It doesn't exactly work how you would expect with tagging but thats a tailscale issue not headscale.

adam commented

2025-12-29 02:27:55 +01:00

@Geofferey commented on GitHub (Aug 8, 2025):

So I am experiencing issues seemingly related to this for anything after v0.23.0, the question being, why did it work in v0.23.0 without issues? What was broken about tags that got "fixed"?

The reason why I liked tags:

I could create tags and set them as the source in my ACLS, assign the tags to various clients nodes and dictate their remote access all without modify ACLs moving forward through Headscale Admin. This was before the UI had an ACL editor....

For example: if I wanted to give internet access, just assign the tag internet to a node, etc, etc.

I realize the way in which I was using them was likely against Tailscale best practices and now I am paying the price for it BUT for my use case it JUST MADE SENSE to me. Sadly this is deterring me from moving on, while sacrificing potential security improvements.

I just want to add one final note on behavior...

v0.23.0 adding a tag to a node through the headscale admin UI did not result in the node also advertising said tag... When applying a tag on v0.23.0 I would usually have to restart the client for it to pick up access but removing the tag would immediately revoke access.

Moving forward to v0.24.0 I am noticing adding a tag also causes the node to advertise that tag in Headscale Admin UI, the node does not immediately pick up access when restarting client and removing a tag does not immediately revoke access..

As noted the tags either lag behind or do not take effect until the control server is restarted. I'm doing this through the web UI just in case that matters, which it very well may. The behavior is being displayed on OIDC and preauth key registered nodes.

This STILL seems like a bit of a regression. I'm screaming at the screen, it ain't a bug it's a feature! 😂 If it's impossible to function in this manner when it comes to staying in line with the official Tailscale implementation I completely understand but I AM HOPING for this to be fixed. Has anyone played with official implementation of Tailscale in this manner, what is the behavior when adding and removing tags from nodes when ACLs are written with TAGs defined as the source?

HERE IS A SAMPLE OF MY ACLs on v0.23.0:

    "groups": {
        "group:netlabwork": [
            ""
        ],
        "group:admin": [
            ""
        ],
        "group:exit": [
            ""
        ],
        "group:unrestricted": [
            ""
        ],
        "group:atakcot": [],
        "group:dns": []
    },
    "tagOwners": {
        "tag:internet": [
            "group:exit"
        ],
        "tag:unrestricted": [
            "tag:unrestricted"
        ],
        "tag:nlw": [
            "group:netlabwork"
        ],
        "tag:atakcot": [
            "group:atakcot"
        ],
        "tag:dns": [
            "group:dns"
        ]
    },
    "hosts": {
        "netlabwork-lan": "10.0.0.0/24",
        "atak": "10.0.0.108/32",
        "win25-service": "10.0.0.16/32",
        "router01": "10.0.0.1/32"
    },
    "acls": [
        {
            "action": "accept",
            "src": [
                "tag:unrestricted"
            ],
            "dst": [
                "*:*"
            ]
        },
        {
            "action": "accept",
            "src": [
                "tag:nlw"
            ],
            "dst": [
                "netlabwork-lan:*"
            ]
        },
        {
            "action": "accept",
            "src": [
                "tag:nlw"
            ],
            "dst": [
                "tag:nlw:*"
            ]
        },
        {
            "action": "accept",
            "src": [
                "tag:internet"
            ],
            "dst": [
                "autogroup:internet:*"
            ]
        },
        {
            "action": "accept",
            "proto": "tcp",
            "src": [
                "tag:atakcot"
            ],
            "dst": [
                "atak:8087"
            ]
        },
        {
            "action": "accept",
            "src": [
                "tag:dns"
            ],
            "dst": [
                "win25-service:53"
            ],
            "proto": "udp"
        },
        {
            "action": "accept",
            "proto": "udp",
            "src": [
                "tag:dns"
            ],
            "dst": [
                "router01:53"
            ]
        }
    ],
    "ssh": []
}````

@Geofferey commented on GitHub (Aug 8, 2025): So I am experiencing issues seemingly related to this for anything after v0.23.0, the question being, why did it work in v0.23.0 without issues? What was broken about tags that got "fixed"? The reason why I liked tags: I could create tags and set them as the source in my ACLS, assign the tags to various clients nodes and dictate their remote access all without modify ACLs moving forward through Headscale Admin. This was before the UI had an ACL editor.... For example: if I wanted to give internet access, just assign the tag internet to a node, etc, etc. I realize the way in which I was using them was likely against Tailscale best practices and now I am paying the price for it BUT for my use case it JUST MADE SENSE to me. Sadly this is deterring me from moving on, while sacrificing potential security improvements. I just want to add one final note on behavior... v0.23.0 adding a tag to a node through the headscale admin UI did not result in the node also advertising said tag... When applying a tag on v0.23.0 I would usually have to restart the client for it to pick up access but removing the tag would immediately revoke access. Moving forward to v0.24.0 I am noticing adding a tag also causes the node to advertise that tag in Headscale Admin UI, the node does not immediately pick up access when restarting client and removing a tag does not immediately revoke access.. As noted the tags either lag behind or do not take effect until the control server is restarted. I'm doing this through the web UI just in case that matters, which it very well may. The behavior is being displayed on OIDC and preauth key registered nodes. This STILL seems like a bit of a regression. I'm screaming at the screen, it ain't a bug it's a feature! 😂 If it's impossible to function in this manner when it comes to staying in line with the official Tailscale implementation I completely understand but I AM HOPING for this to be fixed. Has anyone played with official implementation of Tailscale in this manner, what is the behavior when adding and removing tags from nodes when ACLs are written with TAGs defined as the source? HERE IS A SAMPLE OF MY ACLs on v0.23.0: ````{ "groups": { "group:netlabwork": [ "" ], "group:admin": [ "" ], "group:exit": [ "" ], "group:unrestricted": [ "" ], "group:atakcot": [], "group:dns": [] }, "tagOwners": { "tag:internet": [ "group:exit" ], "tag:unrestricted": [ "tag:unrestricted" ], "tag:nlw": [ "group:netlabwork" ], "tag:atakcot": [ "group:atakcot" ], "tag:dns": [ "group:dns" ] }, "hosts": { "netlabwork-lan": "10.0.0.0/24", "atak": "10.0.0.108/32", "win25-service": "10.0.0.16/32", "router01": "10.0.0.1/32" }, "acls": [ { "action": "accept", "src": [ "tag:unrestricted" ], "dst": [ "*:*" ] }, { "action": "accept", "src": [ "tag:nlw" ], "dst": [ "netlabwork-lan:*" ] }, { "action": "accept", "src": [ "tag:nlw" ], "dst": [ "tag:nlw:*" ] }, { "action": "accept", "src": [ "tag:internet" ], "dst": [ "autogroup:internet:*" ] }, { "action": "accept", "proto": "tcp", "src": [ "tag:atakcot" ], "dst": [ "atak:8087" ] }, { "action": "accept", "src": [ "tag:dns" ], "dst": [ "win25-service:53" ], "proto": "udp" }, { "action": "accept", "proto": "udp", "src": [ "tag:dns" ], "dst": [ "router01:53" ] } ], "ssh": [] }````

adam commented

2025-12-29 02:27:55 +01:00

@djholt commented on GitHub (Aug 23, 2025):

Found a correlation for my problem: If any (forced?) tag is present on a node, the group policies from the node’s user will not be populated on this node.

=> Don’t mix groups and tags, for now?

Banged my head for a few days until I discovered this. Thanks!

@djholt commented on GitHub (Aug 23, 2025): > Found a correlation for my problem: If any (forced?) tag is present on a node, the group policies from the node’s user will not be populated on this node. > > => Don’t mix groups and tags, for now? Banged my head for a few days until I discovered this. Thanks!

adam commented

2025-12-29 02:27:55 +01:00

@almereyda commented on GitHub (Aug 23, 2025):

This helped, thanks. Replacing group:admin entries with the list of individual username@s e.g. within a src statement in the ssh list of the policy (file) restored access to the nodes.

Here all our users come from OIDC, thus this behaviour does not seem to be restricted to what is called local users in this issue.

@almereyda commented on GitHub (Aug 23, 2025): This helped, thanks. Replacing `group:admin` entries with the list of individual `username@`s e.g. within a `src` statement in the `ssh` list of the policy (file) restored access to the nodes. Here all our users come from OIDC, thus this behaviour does not seem to be restricted to what is called local users in this issue.

adam commented

2025-12-29 02:27:55 +01:00

@almereyda commented on GitHub (Sep 23, 2025):

There are multiple other side-effects involved here. Listing related issues to make the reference explicit:

@rainbend After updating the machine's tag, the new rules only take effect on the client after restarting headscale.

@aritas1 If any (forced?) tag is present on a node, the group policies from the node’s user will not be populated on this node.

[PR #301] [MERGED] Set goreleaser to only care about Go 1.17 (#1369)