Subnet router ACL&#39;s broken on 0.23.0-alpha1 #581

@kradalby commented on GitHub (Dec 10, 2023):

0.23.0-alpha2 addresses a series of issues with node synchronisation, online status and subnet routers, please test this release and report back if the issue still persist.

@kradalby commented on GitHub (Dec 10, 2023): [0.23.0-alpha2](https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha2) addresses a series of issues with node synchronisation, online status and subnet routers, please test this release and report back if the issue still persist.

adam commented

@winterheart commented on GitHub (Dec 13, 2023):

Hello, seems 0.23.0-alpha2 still has this issue. I've just updated and reverted back to 0.22.3 mine deployment due inability to use subnet routers. On 0.22.3 all works as expected.

@winterheart commented on GitHub (Dec 13, 2023): Hello, seems 0.23.0-alpha2 still has this issue. I've just updated and reverted back to 0.22.3 mine deployment due inability to use subnet routers. On 0.22.3 all works as expected.

adam commented

@jwischka commented on GitHub (Dec 19, 2023):

I think this may be related to the ACLs - I have an alpha2 where subnet routes are working properly (albeit I have not tested with nodes that are exit nodes).

@jwischka commented on GitHub (Dec 19, 2023): I think this may be related to the ACLs - I have an alpha2 where subnet routes are working properly (albeit I have not tested with nodes that are exit nodes).

adam commented

@kfkawalec commented on GitHub (Dec 19, 2023):

@jwischka Can you share your ACL? I have the same problem, only ACL with "*:0" helps but its not what I want.

@kfkawalec commented on GitHub (Dec 19, 2023): @jwischka Can you share your ACL? I have the same problem, only ACL with "*:0" helps but its not what I want.

adam commented

@jwischka commented on GitHub (Dec 20, 2023):

Mine are basically a series of:

{
  "acls": [
    {
      "action": "accept",
      "src": ["user1"],
      "dst": ["*:*"],
    },
    {
      "action": "accept",
      "src": ["user2"],
      "dst": ["user2:*"],
    },
    {
      "action": "accept",
      "src": ["user3"],
      "dst": ["user2:*"],
    },
  ],
  "disableIPv4": false,
  "randomizeClientPort": true,
}

@jwischka commented on GitHub (Dec 20, 2023): Mine are basically a series of: ``` { "acls": [ { "action": "accept", "src": ["user1"], "dst": ["*:*"], }, { "action": "accept", "src": ["user2"], "dst": ["user2:*"], }, { "action": "accept", "src": ["user3"], "dst": ["user2:*"], }, ], "disableIPv4": false, "randomizeClientPort": true, } ```

adam commented

@kfkawalec commented on GitHub (Dec 20, 2023):

In my case:

if user1 is the router, then user2 and user3 can connect through the router
if user2 is the router, user1 can connet (:) but user3 cannot

@kfkawalec commented on GitHub (Dec 20, 2023): In my case: - if user1 is the router, then user2 and user3 can connect through the router - if user2 is the router, user1 can connet (*:*) but user3 cannot

adam commented

@kradalby commented on GitHub (Jan 3, 2024):

@Sh4d @kfkawalec

Could you please get me a copy of the netmap from the nodes from both version 0.23.0-alpha2 and 0.22.3 so I can compare them?

You can do that with tailscale debug netmap > netmap.json for each node. I recon the most interesting node is the one that can no longer ping.

@kradalby commented on GitHub (Jan 3, 2024): @Sh4d @kfkawalec Could you please get me a copy of the netmap from the nodes from both version 0.23.0-alpha2 and 0.22.3 so I can compare them? You can do that with `tailscale debug netmap > netmap.json` for each node. I recon the most interesting node is the one that can no longer ping.

adam commented

@kradalby commented on GitHub (Jan 3, 2024):

I think I have fixed this in https://github.com/juanfont/headscale/pull/1673, could you try that PR?

@kradalby commented on GitHub (Jan 3, 2024): I think I have fixed this in https://github.com/juanfont/headscale/pull/1673, could you try that PR?

adam commented

@kfkawalec commented on GitHub (Jan 5, 2024):

At the moment I do not have a 0.23.0-alpha2 installation.

@kfkawalec commented on GitHub (Jan 5, 2024): At the moment I do not have a 0.23.0-alpha2 installation.

adam commented

@sniff122 commented on GitHub (Jan 8, 2024):

Not sure if its entirely related, however I am seeing broken subnet routes when setting an ACL.

I have the following ACL:

"acls": [
            {
                "action": "accept",
                "src": [
                        "group:developers"
                ],
                "dst": [
                        "test-server"
                ]
            },
            {
                "action": "accept",
                "src": [
                        "group:developers"
                ],
                "dst": [
                        "group:developers:*"
                ]
            },
            {
                "action": "accept",
                "src": [
                    "*",
                    "group:internal-exitnode"
                ],
                "dst": [
                    "group:internal-exitnode:*"
                ]
            }
    ]

the test-server is defined as a host and is on the same subnet as a the "exit node" advertising the route for the subnet, it is enabled.

With this ACL config, i am not seeing the subnet route in the PacketFilterRules under DstPorts (still seeing the other 2 rules though).

When setting a wildcard dst for the developer group, i am able to ping everything on the subnet and can see the wildcard in DstPorts in the PacketFilterRules.

I have also tried @kradalby's #1673 with the same result.

Tried on MacOS, Windows and Linux clients with the exact same result, running a packet capture on the client for the tailscale interface shows the ICMP with no response found, running a wireshark sshdump on tailscale0 of the subnet route server, i see no ICMP packets

@sniff122 commented on GitHub (Jan 8, 2024): Not sure if its entirely related, however I am seeing broken subnet routes when setting an ACL. I have the following ACL: ``` "acls": [ { "action": "accept", "src": [ "group:developers" ], "dst": [ "test-server" ] }, { "action": "accept", "src": [ "group:developers" ], "dst": [ "group:developers:*" ] }, { "action": "accept", "src": [ "*", "group:internal-exitnode" ], "dst": [ "group:internal-exitnode:*" ] } ] ``` the test-server is defined as a host and is on the same subnet as a the "exit node" advertising the route for the subnet, it is enabled. With this ACL config, i am not seeing the subnet route in the PacketFilterRules under DstPorts (still seeing the other 2 rules though). When setting a wildcard dst for the developer group, i am able to ping everything on the subnet and can see the wildcard in DstPorts in the PacketFilterRules. I have also tried @kradalby's #1673 with the same result. Tried on MacOS, Windows and Linux clients with the exact same result, running a packet capture on the client for the tailscale interface shows the ICMP with no response found, running a wireshark sshdump on tailscale0 of the subnet route server, i see no ICMP packets

adam commented

@kradalby commented on GitHub (Jan 8, 2024):

@sniff122 does it work with 0.22.3? I just want to understand if it is already broken or a regression.

@kradalby commented on GitHub (Jan 8, 2024): @sniff122 does it work with 0.22.3? I just want to understand if it is already broken or a regression.

adam commented

@sniff122 commented on GitHub (Jan 8, 2024):

@kradalby I believe I was having the issue that was fixed by #1564 which is in 0.23.0-alpha2, i shall try 0.22.3 again

@sniff122 commented on GitHub (Jan 8, 2024): @kradalby I believe I was having the issue that was fixed by #1564 which is in 0.23.0-alpha2, i shall try 0.22.3 again

adam commented

@sniff122 commented on GitHub (Jan 8, 2024):

It doesnt look like i have a recent snapshot of my VM with 0.22.3 however with 0.23.0-alpha1 the issues are still present

@sniff122 commented on GitHub (Jan 8, 2024): It doesnt look like i have a recent snapshot of my VM with 0.22.3 however with 0.23.0-alpha1 the issues are still present

adam commented

@kradalby commented on GitHub (Jan 9, 2024):

ok, I it would be ideal to get that tested as it will indicate if it is a new issue or an already existing one.

If it is a new issue (since 0.22.3), it will block the new release, but if it is an existing issue, then we should create a separate issue and solve it after 0.23.0 goes out.

@kradalby commented on GitHub (Jan 9, 2024): ok, I it would be ideal to get that tested as it will indicate if it is a new issue or an already existing one. If it is a new issue (since 0.22.3), it will block the new release, but if it is an existing issue, then we should create a separate issue and solve it after 0.23.0 goes out.

adam commented

@Sh4d commented on GitHub (Jan 9, 2024):

I just tested and confirmed this issue still exists on headscale_0.23.0-alpha2_linux_amd64.deb

@Sh4d commented on GitHub (Jan 9, 2024): I just tested and confirmed this issue still exists on headscale_0.23.0-alpha2_linux_amd64.deb

adam commented

@Sh4d commented on GitHub (Jan 9, 2024):

Here's a (slightly scrubbed) netmap from the client while broken. I was using the exact ACL at the top of this issue with just the allow 0.0.0.0 removed.

netmap.txt

@Sh4d commented on GitHub (Jan 9, 2024): Here's a (slightly scrubbed) netmap from the client while broken. I was using the exact ACL at the top of this issue with just the allow 0.0.0.0 removed. [netmap.txt](https://github.com/juanfont/headscale/files/13877293/netmap.txt)

adam commented

@sniff122 commented on GitHub (Jan 15, 2024):

ok, I it would be ideal to get that tested as it will indicate if it is a new issue or an already existing one.

If it is a new issue (since 0.22.3), it will block the new release, but if it is an existing issue, then we should create a separate issue and solve it after 0.23.0 goes out.

@kradalby

Just tested with 0.22.3, found an older backup and it ACLs appear to be working fine

@sniff122 commented on GitHub (Jan 15, 2024): > ok, I it would be ideal to get that tested as it will indicate if it is a new issue or an already existing one. > > If it is a new issue (since 0.22.3), it will block the new release, but if it is an existing issue, then we should create a separate issue and solve it after 0.23.0 goes out. @kradalby Just tested with 0.22.3, found an older backup and it ACLs appear to be working fine

adam commented

@kradalby commented on GitHub (Jan 18, 2024):

Could you give 0.23.0-alpha3 a go and report back?

@kradalby commented on GitHub (Jan 18, 2024): Could you give 0.23.0-alpha3 a go and report back?

adam commented