Bugs related to state change in #1492 #559

New Issue

@almereyda commented on GitHub (Feb 15, 2024):

To note, the tag of the container image moved from 0.23.0-alpha3 to v0.23.0-alpha4 by introducing the v in the beginning.

We also had to change the command that is run in the container from headscale serve to serve.

Also the syntax from database configuration in config.yaml changed slightly in 94b30abf56 #1700.

This also comes with a new parameter automatically_add_embedded_derp_region: true one wants to copy. I'm regularily running diff config/config.yaml config-example.yaml on a fresh download of the example to find eventual, sic, differences.

Although the image possibly misses a directory at /var/run/headscale/, since the daemon cannot bind the gRPC socket to the expected location, due to no such file or directory. We are solving this by mapping an empty directory at the location.

All of these are probably worth noting as breaking changes, as they will possibly prohibit a seamless upgrade path.

@almereyda commented on GitHub (Feb 15, 2024): To note, the tag of the container image moved from `0.23.0-alpha3` to `v0.23.0-alpha4` by introducing the `v` in the beginning. We also had to change the command that is run in the container from `headscale serve` to `serve`. Also the syntax from database configuration in `config.yaml` changed slightly in 94b30abf56ae09d82a1541bbc3d19557914f9b27 #1700. This also comes with a new parameter `automatically_add_embedded_derp_region: true` one wants to copy. I'm regularily running `diff config/config.yaml config-example.yaml` on a fresh download of the example to find eventual, sic, differences. Although the image possibly misses a directory at `/var/run/headscale/`, since the daemon cannot bind the gRPC socket to the expected location, due to `no such file or directory`. We are solving this by mapping an empty directory at the location. All of these are probably worth noting as breaking changes, as they will possibly prohibit a seamless upgrade path.

adam commented

@TotoTheDragon commented on GitHub (Feb 15, 2024):

To note, the tag of the container image moved from 0.23.0-alpha3 to v0.23.0-alpha4 by introducing the v in the beginning.

We also had to change the command that is run in the container from headscale serve to serve.

Also the syntax from database configuration in config.yaml changed slightly in 94b30ab #1700.

This also comes with a new parameter automatically_add_embedded_derp_region: true one wants to copy. I'm regularily running diff config/config.yaml config-example.yaml on a fresh download of the example to find eventual, sic, differences.

Although the image possibly misses a directory at /var/run/headscale/, since the daemon cannot bind the gRPC socket to the expected location, due to no such file or directory. We are solving this by mapping an empty directory at the location.

All of these are probably worth noting as breaking changes, as they will possibly prohibit a seamless upgrade path.

Made an issue regarding config migrations for you https://github.com/juanfont/headscale/issues/1758

Regarding the docker part, that is of course not officially supported and not well documented. If you feel it is a big issue as of now, feel free to create an issue (and possibly PR).

@TotoTheDragon commented on GitHub (Feb 15, 2024): > To note, the tag of the container image moved from `0.23.0-alpha3` to `v0.23.0-alpha4` by introducing the `v` in the beginning. > > We also had to change the command that is run in the container from `headscale serve` to `serve`. > > Also the syntax from database configuration in `config.yaml` changed slightly in [94b30ab](https://github.com/juanfont/headscale/commit/94b30abf56ae09d82a1541bbc3d19557914f9b27) #1700. > > This also comes with a new parameter `automatically_add_embedded_derp_region: true` one wants to copy. I'm regularily running `diff config/config.yaml config-example.yaml` on a fresh download of the example to find eventual, sic, differences. > > Although the image possibly misses a directory at `/var/run/headscale/`, since the daemon cannot bind the gRPC socket to the expected location, due to `no such file or directory`. We are solving this by mapping an empty directory at the location. > > All of these are probably worth noting as breaking changes, as they will possibly prohibit a seamless upgrade path. Made an issue regarding config migrations for you https://github.com/juanfont/headscale/issues/1758 Regarding the docker part, that is of course not officially supported and not well documented. If you feel it is a big issue as of now, feel free to create an issue (and possibly PR).

adam commented

@kradalby commented on GitHub (Feb 19, 2024):

Could you please test if this is still the case with https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha5 ?

@kradalby commented on GitHub (Feb 19, 2024): Could you please test if this is still the case with https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha5 ?

adam commented

@almereyda commented on GitHub (Feb 19, 2024):

Thank you, very well.

Upgrading alpha4 to alpha5 (in a container) has Headscale end up in a startup crash loop:

2024-02-19T16:29:12Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/root.go:54 > Failed to get headscale configuration error="netip.ParsePrefix(\"\"): no '/'"

This is alleviated with changing the name of the configuration key ip_prefixes to prefixes and changing it into a dictionary with separate keys for v6 and v4, as the diff tells.

< ip_prefixes:                                                                                                                                     
<   - fd7a:115c:a1e0::/48                                                                                                                          
<   - 100.64.0.0/10                                                                                                                                
---
> prefixes:
>   v6: fd7a:115c:a1e0::/48
>   v4: 100.64.0.0/10

It will be good to include this information in the migration guide as well.

If there is interest, I can take look at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-container.md again and prepare a PR including the current changes.

And because we fixed the broken migration manually, we cannot run the new migration, as the column to be deleted does not exist right now #1748.

2024-02-19T16:39:43Z FTL Migration failed: ERROR: column "last_successful_update" of relation "nodes" does not exist (SQLSTATE 42703) error="ERROR: column \"last_successful_update\" of relation \"nodes\" does not exist (SQLSTATE 42703)"

Running alter table nodes add column last_successful_update timestamp; allowed to apply the migration in our special case and alpha5 is now running as expected.

@almereyda commented on GitHub (Feb 19, 2024): Thank you, very well. Upgrading `alpha4` to `alpha5` (in a container) has Headscale end up in a startup crash loop: 2024-02-19T16:29:12Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/root.go:54 > Failed to get headscale configuration error="netip.ParsePrefix(\"\"): no '/'" This is alleviated with changing the name of the configuration key `ip_prefixes` to `prefixes` and changing it into a dictionary with separate keys for `v6` and `v4`, as the diff tells. ```diff < ip_prefixes: < - fd7a:115c:a1e0::/48 < - 100.64.0.0/10 --- > prefixes: > v6: fd7a:115c:a1e0::/48 > v4: 100.64.0.0/10 ``` It will be good to include this information in the migration guide as well. If there is interest, I can take look at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-container.md again and prepare a PR including the current changes. And because we fixed the broken migration manually, we cannot run the new migration, as the column to be deleted does not exist right now #1748. ``` 2024-02-19T16:39:43Z FTL Migration failed: ERROR: column "last_successful_update" of relation "nodes" does not exist (SQLSTATE 42703) error="ERROR: column \"last_successful_update\" of relation \"nodes\" does not exist (SQLSTATE 42703)" ``` Running `alter table nodes add column last_successful_update timestamp;` allowed to apply the migration in our special case and `alpha5` is now running as expected.

adam commented

@kradalby commented on GitHub (Apr 17, 2024):

Could you please try the newest alpha (https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha6) and report back?

@kradalby commented on GitHub (Apr 17, 2024): Could you please try the newest alpha (https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha6) and report back?

adam commented

@vsychov commented on GitHub (Apr 18, 2024):

Thanks @kradalby , I'll make tests today or tomorrow

@vsychov commented on GitHub (Apr 18, 2024): Thanks @kradalby , I'll make tests today or tomorrow

adam commented

@vsychov commented on GitHub (Apr 20, 2024):

Hello @kradalby , I tried testing it on the alpha-8 revision but couldn't manage it due to an issue with ACL. Here's the ACL file:

"acls":
- "action": "accept"
  "dst":
  - "*:*"
  "src":
  - "group:full-access"
"groups":
  "group:full-access":
  - "user.example.com"
"ssh":
- "action": "accept"
  "dst":
  - "*"
  "src":
  - "group:full-access"
  "users":
  - "root"

I created two nodes:

tmp-tailscale-1-ams3-do
tmp-tailscale-2-ams3-do

Then, I connected them to headscale using a pre-auth key for the user user.example.com:

tailscale up --auth-key XXXX --advertise-exit-node --login-server=https://headscale-test.example.com --advertise-routes=10.110.0.0/16 --shields-up=false

Node details:

ID | Hostname                | Name                    | MachineKey | NodeKey | User                 | IP addresses | Ephemeral | Last seen           | Expiration          | Connected | Expired
1  | tmp-tailscale-1-ams3-do | tmp-tailscale-1-ams3-do | [uvObE]    | [JSAmz] | user.example.com     | ,            | false     | 2024-04-20 09:06:14 | 0001-01-01 00:00:00 | online    | no
2  | tmp-tailscale-2-ams3-do | tmp-tailscale-2-ams3-do | [MvLkM]    | [nIo1T] | user.example.com     | ,            | false     | 2024-04-20 09:10:54 | 0001-01-01 00:00:00 | online    | no

However, both nodes are invisible to each other:

root@tmp-tailscale-2-ams3-do:~# tailscale status
tmp-tailscale-2-ams3-do user.example.com linux   idle; offers exit node

root@tmp-tailscale-1-ams3-do:~# tailscale status
tmp-tailscale-1-ams3-do user.example.com linux   idle; offers exit node

Additionally, I noticed that there are no IP addresses in headscale node ls, which might also be incorrect.

@vsychov commented on GitHub (Apr 20, 2024): Hello @kradalby , I tried testing it on the `alpha-8` revision but couldn't manage it due to an issue with ACL. Here's the ACL file: ``` "acls": - "action": "accept" "dst": - "*:*" "src": - "group:full-access" "groups": "group:full-access": - "user.example.com" "ssh": - "action": "accept" "dst": - "*" "src": - "group:full-access" "users": - "root" ``` I created two nodes: 1. `tmp-tailscale-1-ams3-do` 2. `tmp-tailscale-2-ams3-do` Then, I connected them to headscale using a pre-auth key for the user `user.example.com`: ```bash tailscale up --auth-key XXXX --advertise-exit-node --login-server=https://headscale-test.example.com --advertise-routes=10.110.0.0/16 --shields-up=false ``` Node details: ``` ID | Hostname | Name | MachineKey | NodeKey | User | IP addresses | Ephemeral | Last seen | Expiration | Connected | Expired 1 | tmp-tailscale-1-ams3-do | tmp-tailscale-1-ams3-do | [uvObE] | [JSAmz] | user.example.com | , | false | 2024-04-20 09:06:14 | 0001-01-01 00:00:00 | online | no 2 | tmp-tailscale-2-ams3-do | tmp-tailscale-2-ams3-do | [MvLkM] | [nIo1T] | user.example.com | , | false | 2024-04-20 09:10:54 | 0001-01-01 00:00:00 | online | no ``` However, both nodes are invisible to each other: ```bash root@tmp-tailscale-2-ams3-do:~# tailscale status tmp-tailscale-2-ams3-do user.example.com linux idle; offers exit node ``` ```bash root@tmp-tailscale-1-ams3-do:~# tailscale status tmp-tailscale-1-ams3-do user.example.com linux idle; offers exit node ``` Additionally, I noticed that there are no IP addresses in `headscale node ls`, which might also be incorrect.

adam commented

@kradalby commented on GitHub (Apr 29, 2024):

Yes the IPs is strange I suspect they might be the case. Is this sqlite or postgres? do you have the config?

Edit: the config might not have the new IP prefix syntax?

@kradalby commented on GitHub (Apr 29, 2024): Yes the IPs is strange I suspect they might be the case. Is this sqlite or postgres? do you have the config? Edit: the config might not have the new IP prefix syntax?

adam commented

@vsychov commented on GitHub (Apr 29, 2024):

@kradalby, you are right! There were missing prefixes in configuration, probably it's also a validation bug. I'll try testing with the new config.

@vsychov commented on GitHub (Apr 29, 2024): @kradalby, you are right! There were missing `prefixes` in configuration, probably it's also a validation bug. I'll try testing with the new config.

adam commented

@kradalby commented on GitHub (Apr 29, 2024):

Yea, we need to throw an error if there are no prefixes!

@kradalby commented on GitHub (Apr 29, 2024): Yea, we need to throw an error if there are no prefixes!

adam commented

@kradalby commented on GitHub (Apr 30, 2024):

https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha10 was also released to address a couple of regressions in the ACL.

@kradalby commented on GitHub (Apr 30, 2024): https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha10 was also released to address a couple of regressions in the ACL.

adam commented

@kradalby commented on GitHub (Apr 30, 2024):

and addressed no prefix issue in https://github.com/juanfont/headscale/pull/1918

@kradalby commented on GitHub (Apr 30, 2024): and addressed no prefix issue in https://github.com/juanfont/headscale/pull/1918

adam commented

@kradalby commented on GitHub (May 1, 2024):

@vsychov let me know when you have had a time to give this a go, if these issues are resolved, I will tag a beta release after resolving one other issue.

@kradalby commented on GitHub (May 1, 2024): @vsychov let me know when you have had a time to give this a go, if these issues are resolved, I will tag a beta release after resolving one other issue.

adam commented

@vsychov commented on GitHub (May 2, 2024):

@kradalby , I was just about to write that everything was fine, but it seems something went wrong. I deployed a test environment consisting of 3 machines and one headscale control server.

I left it running for about 3 days, and for the first 24 hours, it worked super stably. Subnet routers failed over very quickly, there were no problems with nodes going offline, and so on. Everything seemed just fine. But literally just before writing here, I decided to do a retest and found that both subnet routers became Primary == false.

ID | Node                       | Prefix     | Advertised | Enabled | Primary
3  | tmp-tailscale-test-02-ams3 | 0.0.0.0/0  | true       | true    | -
1  | tmp-tailscale-test-02-ams3 | ::/0       | true       | true    | -
2  | tmp-tailscale-test-02-ams3 | 10.0.0.0/8 | true       | true    | false
5  | tmp-tailscale-test-ams3    | 0.0.0.0/0  | true       | true    | -
6  | tmp-tailscale-test-ams3    | ::/0       | true       | true    | -
4  | tmp-tailscale-test-ams3    | 10.0.0.0/8 | true       | true    | false

Consequently, from the clients, the subnet 10.0.0.0/8 became inaccessible.

➜  ~ /Applications/Tailscale.app/Contents/MacOS/Tailscale ping 10.110.0.9
2024/05/02 11:16:35 no matching peer

Testing was performed on version 0.23.0-alpha8.

If I can provide any additional information that would help identify the cause of this behavior, let me know, and I'll try to get it (perhaps logs would be useful or something else).

adam commented

@kradalby commented on GitHub (May 2, 2024):

@vsychov do you have the log of the machine? that would be helpful.

If you cannot share it, it would be useful to see if you see a lot log lines with rejected in them.

@kradalby commented on GitHub (May 2, 2024): @vsychov do you have the log of the machine? that would be helpful. If you cannot share it, it would be useful to see if you see a lot log lines with `rejected` in them.

adam commented

@vsychov commented on GitHub (May 2, 2024):

@kradalby, which machine's log? Where was headscale run? Or machine with tailscale?

@vsychov commented on GitHub (May 2, 2024): @kradalby, which machine's log? Where was headscale run? Or machine with tailscale?

adam commented

@kradalby commented on GitHub (May 2, 2024):

Headscale please

@kradalby commented on GitHub (May 2, 2024): Headscale please

adam commented

@vsychov commented on GitHub (May 2, 2024):

I noticed that it was at the 'INFO' level, so there are no lines with rejected. I'll switch it to trace level and rerun the tests. Hopefully, I can reproduce it again. However, there's a full log available; perhaps it contains something useful. Additionally, would it make sense to downgrade logs like 'Sending Changed MapResponse' or 'received stream update' to debug level from info?
headscale.log

@vsychov commented on GitHub (May 2, 2024): I noticed that it was at the 'INFO' level, so there are no lines with `rejected`. I'll switch it to `trace` level and rerun the tests. Hopefully, I can reproduce it again. However, there's a full log available; perhaps it contains something useful. Additionally, would it make sense to downgrade logs like 'Sending Changed MapResponse' or 'received stream update' to debug level from info? [headscale.log](https://github.com/juanfont/headscale/files/15189164/headscale.log)

adam commented

@vsychov commented on GitHub (May 2, 2024):

I've switched the logs to trace mode, and I suggest waiting a few more days in an attempt to reproduce the issue. Meanwhile, as far as I can see, headscale doesn't seem to perform route reselection upon restart, or it seems better to do it once in a while, for example, every 10 seconds (the time could be configurable), by going through all routes and finding those without any 'Primary' nodes, and forcibly performing reselections. This way, it seems possible to protect against some failures when routes are chosen based on events (as I understand it, that's how it's currently done).

@vsychov commented on GitHub (May 2, 2024): I've switched the logs to trace mode, and I suggest waiting a few more days in an attempt to reproduce the issue. Meanwhile, as far as I can see, headscale doesn't seem to perform route reselection upon restart, or it seems better to do it once in a while, for example, every 10 seconds (the time could be configurable), by going through all routes and finding those without any 'Primary' nodes, and forcibly performing reselections. This way, it seems possible to protect against some failures when routes are chosen based on events (as I understand it, that's how it's currently done).

adam commented

@vsychov commented on GitHub (May 8, 2024):

I'm writing an update on the testing results. Since I started running headscale with logs in trace mode, I still haven't been able to reproduce this issue. It seems to be a very rare case, and everything else looks very stable at the moment.

I'll continue to keep the headscale test instance running in the hope that I can reproduce this situation and gather more logs. Perhaps I'll increase the size of the test infrastructure, add more clients, and try to reproduce the problem in that scenario. However, I still believe that a better solution would be to write a background goroutine that monitors all routes and, in case a route doesn't have an Primary node, selects one. I can try to submit a PR with this approach if you approve @kradalby .

@vsychov commented on GitHub (May 8, 2024): I'm writing an update on the testing results. Since I started running `headscale` with logs in `trace` mode, I still haven't been able to reproduce this issue. It seems to be a very rare case, and everything else looks very stable at the moment. I'll continue to keep the `headscale` test instance running in the hope that I can reproduce this situation and gather more logs. Perhaps I'll increase the size of the test infrastructure, add more clients, and try to reproduce the problem in that scenario. However, I still believe that a better solution would be to write a background goroutine that monitors all routes and, in case a route doesn't have an `Primary` node, selects one. I can try to submit a PR with this approach if you approve @kradalby .

adam commented

@kradalby commented on GitHub (May 24, 2024):

I've fixed up some of the things that could cause a deadlock in https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha12, I will leave this open for you to test, and hopefully we can close this.

@kradalby commented on GitHub (May 24, 2024): I've fixed up some of the things that could cause a deadlock in https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha12, I will leave this open for you to test, and hopefully we can close this.

adam commented

@almereyda commented on GitHub (May 24, 2024):

Extremely thorough progress, plus attentive and reactive intervention. The way this came out gives a lot of confidence with regards to #1072.

@almereyda commented on GitHub (May 24, 2024): Extremely thorough progress, plus attentive and reactive intervention. The way this came out gives a lot of confidence with regards to #1072.

adam commented