[Bug] v0.27.0 policy apply errors #1118

Closed
opened 2025-12-29 02:28:21 +01:00 by adam · 14 comments
Owner

Originally created by @YouSysAdmin on GitHub (Oct 27, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Multiple errors during policy apply

2025-10-27T13:10:04Z TRC ../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"user@\" not found\n\tuser with token \"user1@\" ... "

As far as I understand, this error is displayed separately for each group and with list of all users in group.

Users:

root@ip-10-255-1-11:/home/u# headscale users list
ID | Name     | Username.   | Email                     | Created
1  | user     | user        | user@example.com          | 2025-07-29 10:46:19
2  | user1    | user1       | user1@example.com         | 2025-07-29 10:54:18
...

Policy:

  "groups": {
    "group:example": [
      "user@",
      "user1@"
]
...

Expected Behavior

headscale policy set -f policy.json

Steps To Reproduce

  1. apply policy

Environment

- OS: CLI MacOS, Server Linux
- Headscale version: 0.27.0
- Tailscale version:

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

2025-10-27T13:10:04Z TRC ../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"user@\" not found\n\tuser with token \"user1@\" ... "

2025-10-27T15:31:31+02:00 DBG Policy filter hash changed filter.hash.new=73d84082 filter.hash.old=00000000 filter.rules=0 filter.rules.new=1
2025-10-27T15:31:31+02:00 DBG Tag owner hash changed tagOwner.hash.new=35dcc9c4 tagOwner.hash.old=00000000 tagOwners.new=7 tagOwners.old=0
2025-10-27T15:31:31+02:00 DBG Auto-approvers hash changed autoApprove.hash.new=ac2bfb0f autoApprove.hash.old=00000000 autoApprovers.new=1 autoApprovers.old=0
2025-10-27T15:31:31+02:00 DBG Exit node set hash changed exitSet.hash.new=43a78ec8 exitSet.hash.old=00000000
2025-10-27T15:31:31+02:00 DBG Policy changes require node updates autoApprovers.changed=true exitNodes.changed=true filter.changed=true tagOwners.changed=true
Policy updated.
Originally created by @YouSysAdmin on GitHub (Oct 27, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Multiple errors during policy apply ``` 2025-10-27T13:10:04Z TRC ../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"user@\" not found\n\tuser with token \"user1@\" ... " ``` As far as I understand, this error is displayed separately for each group and with list of all users in group. Users: ``` root@ip-10-255-1-11:/home/u# headscale users list ID | Name | Username. | Email | Created 1 | user | user | user@example.com | 2025-07-29 10:46:19 2 | user1 | user1 | user1@example.com | 2025-07-29 10:54:18 ... ``` Policy: ``` "groups": { "group:example": [ "user@", "user1@" ] ... ``` ### Expected Behavior `headscale policy set -f policy.json` ### Steps To Reproduce 1. apply policy ### Environment ```markdown - OS: CLI MacOS, Server Linux - Headscale version: 0.27.0 - Tailscale version: ``` ### Runtime environment - [ ] Headscale is behind a (reverse) proxy - [ ] Headscale runs in a container ### Debug information ``` 2025-10-27T13:10:04Z TRC ../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"user@\" not found\n\tuser with token \"user1@\" ... " 2025-10-27T15:31:31+02:00 DBG Policy filter hash changed filter.hash.new=73d84082 filter.hash.old=00000000 filter.rules=0 filter.rules.new=1 2025-10-27T15:31:31+02:00 DBG Tag owner hash changed tagOwner.hash.new=35dcc9c4 tagOwner.hash.old=00000000 tagOwners.new=7 tagOwners.old=0 2025-10-27T15:31:31+02:00 DBG Auto-approvers hash changed autoApprove.hash.new=ac2bfb0f autoApprove.hash.old=00000000 autoApprovers.new=1 autoApprovers.old=0 2025-10-27T15:31:31+02:00 DBG Exit node set hash changed exitSet.hash.new=43a78ec8 exitSet.hash.old=00000000 2025-10-27T15:31:31+02:00 DBG Policy changes require node updates autoApprovers.changed=true exitNodes.changed=true filter.changed=true tagOwners.changed=true Policy updated. ```
adam added the bug label 2025-12-29 02:28:21 +01:00
adam closed this issue 2025-12-29 02:28:21 +01:00
Author
Owner

@kradalby commented on GitHub (Oct 27, 2025):

Could you share more of your policy?

apply policy

The useful thing would be a minimally reproducible example as this is not sufficient to reproduce, most policies and our tests pass, so there must be a case we do not know about.

@kradalby commented on GitHub (Oct 27, 2025): Could you share more of your policy? > apply policy The useful thing would be a minimally reproducible example as this is not sufficient to reproduce, most policies and our tests pass, so there must be a case we do not know about.
Author
Owner

@YouSysAdmin commented on GitHub (Oct 27, 2025):

Could you share more of your policy?

apply policy

The useful thing would be a minimally reproducible example as this is not sufficient to reproduce, most policies and our tests pass, so there must be a case we do not know about.

Yes, I'll try to reproduce the problem in a test environment and provide a ready-made database. There's nothing special in the policy itself, just a bunch of groups and ACLs, but unfortunately I can't publish it.

@YouSysAdmin commented on GitHub (Oct 27, 2025): > Could you share more of your policy? > > > apply policy > > The useful thing would be a minimally reproducible example as this is not sufficient to reproduce, most policies and our tests pass, so there must be a case we do not know about. Yes, I'll try to reproduce the problem in a test environment and provide a ready-made database. There's nothing special in the policy itself, just a bunch of groups and ACLs, but unfortunately I can't publish it.
Author
Owner

@mhahl commented on GitHub (Oct 28, 2025):

I started seeing the same error after upgrading 0.26.1 -> 0.27.0 with Keycloak.

I had to change my acls to use the email.

  "groups": {
    "group:grp-prod-admins": [
      "mark.hahl@sigiant.au", # Previously was mark.hahl@
       ...
]

However I can't replicate on a fresh install.

@mhahl commented on GitHub (Oct 28, 2025): I started seeing the same error after upgrading 0.26.1 -> 0.27.0 with Keycloak. I had to change my acls to use the email. ``` "groups": { "group:grp-prod-admins": [ "mark.hahl@sigiant.au", # Previously was mark.hahl@ ... ] ``` However I can't replicate on a fresh install.
Author
Owner

@schwankner commented on GitHub (Oct 28, 2025):

I've been getting this error cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well.

home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"
@schwankner commented on GitHub (Oct 28, 2025): I've been getting this error `cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"` since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well. ``` home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" ```
Author
Owner

@kradalby commented on GitHub (Oct 28, 2025):

Hi!,

First, I really wish you guys would have helped testing the betas with a copy of your databases and policies, it would have helps us tremendously and it would have been great to get all of these things sorted before we release.

We have an issue templates that asks for information, including version of the policy file that reproduce the issue, and so on. We ask this for us to be able to fix it quickly and efficiently.

We have quite an extensive test set for the policy, and clearly, we have missing cases, those are cases we already do not know about, and reversing them from the errors are not trivial. When we know how to test it, we aim to both fix, improve or change the error message to be more helpful. But we need information.

@kradalby commented on GitHub (Oct 28, 2025): Hi!, First, I really wish you guys would have helped testing the betas with a copy of your databases and policies, it would have helps us tremendously and it would have been great to get all of these things sorted before we release. We have an [issue templates](https://github.com/juanfont/headscale/blob/main/.github/ISSUE_TEMPLATE/bug_report.yaml) that asks for information, including version of the policy file that reproduce the issue, and so on. We ask this for us to be able to fix it quickly and efficiently. We have quite an extensive test set for the policy, and clearly, we have missing cases, those are cases we already do not know about, and reversing them from the errors are not trivial. When we know how to test it, we aim to both fix, improve or change the error message to be more helpful. But we need information.
Author
Owner

@mhahl commented on GitHub (Oct 28, 2025):

I've been getting this error cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well.

home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"

I think this one is mentioned in the release notes.

Policy: Zero or empty destination port is no longer allowed

Have a look through: https://github.com/juanfont/headscale/pull/2606

@mhahl commented on GitHub (Oct 28, 2025): > I've been getting this error `cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"` since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well. > > ``` > home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" > ``` I think this one is mentioned in the release notes. > Policy: Zero or empty destination port is no longer allowed Have a look through: https://github.com/juanfont/headscale/pull/2606
Author
Owner

@YouSysAdmin commented on GitHub (Oct 28, 2025):

@kradalby
I started testing the beta, but unfortunately I'm from Ukraine and reality is making its own adjustments in the life. :(
I will try to reproduce the problem in a test environment and share the db and policy.

@YouSysAdmin commented on GitHub (Oct 28, 2025): @kradalby I started testing the beta, but unfortunately I'm from Ukraine and reality is making its own adjustments in the life. :( I will try to reproduce the problem in a test environment and share the db and policy.
Author
Owner

@YouSysAdmin commented on GitHub (Oct 28, 2025):

@kradalby

DB+Policy: https://storage.sysadmin.host/public/headscale-test.tar.gz

The mode for policy database

policy:
  mode: "database"
❯ ./headscale_0.27.0_darwin_arm64 -c data/config.yaml policy set -f data/policy.json
2025-10-28T13:19:45+02:00 TRC ../../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"ops@\" not found\n\tuser with token \"user1@\" not found\n\tuser with token \"user2@\" not found\n\tuser with token \"unregistered_user@\" not found"
2025-10-28T13:19:45+02:00 DBG Policy filter hash changed filter.hash.new=bc11fee0 filter.hash.old=00000000 filter.rules=0 filter.rules.new=0
2025-10-28T13:19:45+02:00 DBG Tag owner hash changed tagOwner.hash.new=a691bfa8 tagOwner.hash.old=00000000 tagOwners.new=2 tagOwners.old=0
2025-10-28T13:19:45+02:00 DBG Auto-approvers hash changed autoApprove.hash.new=36dd032e autoApprove.hash.old=00000000 autoApprovers.new=1 autoApprovers.old=0
2025-10-28T13:19:45+02:00 DBG Exit node set hash changed exitSet.hash.new=8465bda0 exitSet.hash.old=00000000
2025-10-28T13:19:45+02:00 DBG Policy changes require node updates autoApprovers.changed=true exitNodes.changed=true filter.changed=true tagOwners.changed=true
Policy updated.

It appears the problem is occurring on the CLI side during policy application.
During the policy verification process, the CLI attempts to retrieve user addresses, but it does not have this data on its side.
If the mode is switched to file, Headscale does not show any errors on startup. Similarly, if the policy already exists in the database and the mode is set to database, there will also be no errors on startup – the problem only occurs on the CLI side for the policy set command.

P.S. No errors occur if the ACL list is empty.

@YouSysAdmin commented on GitHub (Oct 28, 2025): @kradalby DB+Policy: https://storage.sysadmin.host/public/headscale-test.tar.gz The mode for policy `database` ``` policy: mode: "database" ``` ``` ❯ ./headscale_0.27.0_darwin_arm64 -c data/config.yaml policy set -f data/policy.json 2025-10-28T13:19:45+02:00 TRC ../../home/runner/work/headscale/headscale/hscontrol/policy/v2/filter.go:37 > resolving source ips error="multiple errors:\n\tuser with token \"ops@\" not found\n\tuser with token \"user1@\" not found\n\tuser with token \"user2@\" not found\n\tuser with token \"unregistered_user@\" not found" 2025-10-28T13:19:45+02:00 DBG Policy filter hash changed filter.hash.new=bc11fee0 filter.hash.old=00000000 filter.rules=0 filter.rules.new=0 2025-10-28T13:19:45+02:00 DBG Tag owner hash changed tagOwner.hash.new=a691bfa8 tagOwner.hash.old=00000000 tagOwners.new=2 tagOwners.old=0 2025-10-28T13:19:45+02:00 DBG Auto-approvers hash changed autoApprove.hash.new=36dd032e autoApprove.hash.old=00000000 autoApprovers.new=1 autoApprovers.old=0 2025-10-28T13:19:45+02:00 DBG Exit node set hash changed exitSet.hash.new=8465bda0 exitSet.hash.old=00000000 2025-10-28T13:19:45+02:00 DBG Policy changes require node updates autoApprovers.changed=true exitNodes.changed=true filter.changed=true tagOwners.changed=true Policy updated. ``` It appears the problem is occurring on the CLI side during policy application. During the policy verification process, the CLI attempts to retrieve user addresses, but it does not have this data on its side. If the mode is switched to `file`, Headscale does not show any errors on startup. Similarly, if the policy already exists in the database and the mode is set to `database`, there will also be no errors on startup – the problem only occurs on the CLI side for the `policy set` command. P.S. No errors occur if the ACL list is empty.
Author
Owner

@schwankner commented on GitHub (Oct 28, 2025):

I've been getting this error cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well.

home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"

I think this one is mentioned in the release notes.

Policy: Zero or empty destination port is no longer allowed

Have a look through: #2606

Yes, this solved the problem, thanks!

@schwankner commented on GitHub (Oct 28, 2025): > > I've been getting this error `cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"` since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well. > > ``` > > home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" > > ``` > > I think this one is mentioned in the release notes. > > > Policy: Zero or empty destination port is no longer allowed > > Have a look through: [#2606](https://github.com/juanfont/headscale/pull/2606) Yes, this solved the problem, thanks!
Author
Owner

@mhahl commented on GitHub (Oct 29, 2025):

@YouSysAdmin Just quickly testing with your database and policy:

Using the following compose.yaml

version: "3.0"

services:
  headscale:
    image: quay.sigaint.au/sigaint/headscale:v0.27.0
    container_name: headscale
    network_mode: host
    command: "/go/bin/headscale serve"
    environment:
      - "TZ=Australia/Sydney"
    ports:
      - 8070:8070
    volumes:
      - /data/containers/headscale/data:/var/lib/headscale:z
      - /data/containers/headscale/config:/etc/headscale:z
    restart: unless-stopped

Dropped your database into /data/containers/headscale/data and your policy.json in to /data/containers/headscale/data with a basic config.yaml.

Running the container (built with the v0.27.0 tag):

podman exec -it headscale headscale policy set -f /etc/headscale/policy.json 

I can indeed replicate your issue!

2025-10-29T11:24:21+11:00 TRC hscontrol/policy/v2/filter.go:40 > resolving source ips error="multiple errors:\n\tuser with token \"ops@\" not found\n\tuser with token \"user1@\" not found\n\tuser with token \"user2@\" not found\n\tuser with token \"unregistered_user@\" not found"

Just quickly looking though the code. (For the record I'm not a Go dev)

450a7b15ec/cmd/headscale/cli/policy.go (L130-L134)

Seems that setting the second argument in NewPolicyManager to nil causes the len() check below for potentialUsers to always be 0,

450a7b15ec/hscontrol/policy/v2/types.go (L163-L192)

Always returning the error:

450a7b15ec/hscontrol/policy/v2/types.go (L184)

@mhahl commented on GitHub (Oct 29, 2025): @YouSysAdmin Just quickly testing with your database and policy: Using the following `compose.yaml` ``` version: "3.0" services: headscale: image: quay.sigaint.au/sigaint/headscale:v0.27.0 container_name: headscale network_mode: host command: "/go/bin/headscale serve" environment: - "TZ=Australia/Sydney" ports: - 8070:8070 volumes: - /data/containers/headscale/data:/var/lib/headscale:z - /data/containers/headscale/config:/etc/headscale:z restart: unless-stopped ``` Dropped your database into `/data/containers/headscale/data` and your `policy.json` in to `/data/containers/headscale/data` with a basic config.yaml. Running the container (built with the v0.27.0 tag): ``` podman exec -it headscale headscale policy set -f /etc/headscale/policy.json ``` I can indeed replicate your issue! ``` 2025-10-29T11:24:21+11:00 TRC hscontrol/policy/v2/filter.go:40 > resolving source ips error="multiple errors:\n\tuser with token \"ops@\" not found\n\tuser with token \"user1@\" not found\n\tuser with token \"user2@\" not found\n\tuser with token \"unregistered_user@\" not found" ``` Just quickly looking though the code. (For the record I'm not a Go dev) https://github.com/juanfont/headscale/blob/450a7b15ec7b08926738e308bd11ec17753d06ab/cmd/headscale/cli/policy.go#L130-L134 Seems that setting the second argument in `NewPolicyManager` to `nil` causes the `len()` check below for `potentialUsers` to always be 0, https://github.com/juanfont/headscale/blob/450a7b15ec7b08926738e308bd11ec17753d06ab/hscontrol/policy/v2/types.go#L163-L192 Always returning the error: https://github.com/juanfont/headscale/blob/450a7b15ec7b08926738e308bd11ec17753d06ab/hscontrol/policy/v2/types.go#L184
Author
Owner

@StarkZarn commented on GitHub (Nov 1, 2025):

I've been getting this error cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well.

home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"

I think this one is mentioned in the release notes.

Policy: Zero or empty destination port is no longer allowed

Have a look through: #2606

I had include port "0" in my ACLs to permit the use of a node for routability to on-network endpoints without allowing explicit access to that node itself. Is this no longer required? I don't see mention of this use-case in the release notes or the PR.

@StarkZarn commented on GitHub (Nov 1, 2025): > > I've been getting this error `cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")"` since updating from 0.26.1 to 0.27.0 — I think it's related to this issue as well. > > ``` > > home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: parsing policy from bytes: json: cannot unmarshal JSON string into Go v2.AliasWithPorts within \"/acls/1/dst/0\": hostport must contain a colon (\":\")" > > ``` > > I think this one is mentioned in the release notes. > > > Policy: Zero or empty destination port is no longer allowed > > Have a look through: [#2606](https://github.com/juanfont/headscale/pull/2606) I had include port "0" in my ACLs to permit the use of a node for routability to on-network endpoints without allowing explicit access to that node itself. Is this no longer required? I don't see mention of this use-case in the release notes or the PR.
Author
Owner

@nblock commented on GitHub (Nov 2, 2025):

I had include port "0" in my ACLs to permit the use of a node for routability to on-network endpoints without allowing explicit access to that node itself. Is this no longer required? I don't see mention of this use-case in the release notes or the PR.

You could use a icmp rule instead:

{
    "action": "accept",
    "src": ["..."],
    "dst": ["...:*"],
    "proto": "icmp"
}

But I believe this is no longer required.

@nblock commented on GitHub (Nov 2, 2025): > I had include port "0" in my ACLs to permit the use of a node for routability to on-network endpoints without allowing explicit access to that node itself. Is this no longer required? I don't see mention of this use-case in the release notes or the PR. You could use a icmp rule instead: ```json { "action": "accept", "src": ["..."], "dst": ["...:*"], "proto": "icmp" } ``` [But I believe this is no longer required](https://headscale.net/0.27.0/ref/routes/#restrict-the-use-of-a-subnet-router-with-acl).
Author
Owner

@YouSysAdmin commented on GitHub (Nov 2, 2025):

Hi @kradalby @nblock
Thnx for the fix.
I will test this on a real infrastructure and let you know the results.

@YouSysAdmin commented on GitHub (Nov 2, 2025): Hi @kradalby @nblock Thnx for the fix. I will test this on a real infrastructure and let you know the results.
Author
Owner

@YouSysAdmin commented on GitHub (Nov 10, 2025):

Hi @kradalby @nblock
Everything seems fine.
I hope that 0.27.1 is just around the corner :)

It would be great to include this fix, without it, viewing routes is very inconvenient :)
https://github.com/juanfont/headscale/pull/2760

@YouSysAdmin commented on GitHub (Nov 10, 2025): Hi @kradalby @nblock Everything seems fine. I hope that 0.27.1 is just around the corner :) It would be great to include this fix, without it, viewing routes is very inconvenient :) https://github.com/juanfont/headscale/pull/2760
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1118