MachineKey not displayed for some nodes when listing nodes from CLI #415

Closed
opened 2025-12-29 01:28:36 +01:00 by adam · 5 comments
Owner

Originally created by @magkopian on GitHub (Jan 22, 2023).

Bug description

Recently I noticed that when I run headscale nodes list -t on our server, the MachineKey field of the nodes after a certain point is missing. In case I request a JSON output the machine_key is going to be 0000000000000000000000000000000000000000000000000000000000000000 for the affected devices.

To Reproduce

Do a listing of all nodes from CLI by running headscale nodes list.

Context info

In our particular case, we current have a total of 157 nodes (141 online). The first 133 nodes correctly display the MachineKey, while everything bellow it is empty. Whether a node appears to be offline or online does not appear to play any role.

  • Version of headscale used: v0.18.0
  • Version of tailscale client: N/A
  • OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version: Debian 11.6 (Bullseye)
  • Kernel version: 5.10.0-20-amd64

Update 25/1/2023:

I have recently rebooted the server due to a kernel update and I noticed that the MachineKey field is now populated for some of the ones that were previously missing, but not all. Maybe it's some kind of caching issue?

Originally created by @magkopian on GitHub (Jan 22, 2023). **Bug description** Recently I noticed that when I run `headscale nodes list -t` on our server, the MachineKey field of the nodes after a certain point is missing. In case I request a JSON output the `machine_key` is going to be `0000000000000000000000000000000000000000000000000000000000000000` for the affected devices. **To Reproduce** Do a listing of all nodes from CLI by running `headscale nodes list`. **Context info** In our particular case, we current have a total of 157 nodes (141 online). The first 133 nodes correctly display the MachineKey, while everything bellow it is empty. Whether a node appears to be offline or online does not appear to play any role. - Version of headscale used: v0.18.0 - Version of tailscale client: N/A - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version: Debian 11.6 (Bullseye) - Kernel version: 5.10.0-20-amd64 Update 25/1/2023: I have recently rebooted the server due to a kernel update and I noticed that the MachineKey field is now populated for some of the ones that were previously missing, but not all. Maybe it's some kind of caching issue?
adam added the bug label 2025-12-29 01:28:36 +01:00
adam closed this issue 2025-12-29 01:28:37 +01:00
Author
Owner

@juanfont commented on GitHub (Jan 31, 2023):

Those that are missing, are actually connected to headscale?

@juanfont commented on GitHub (Jan 31, 2023): Those that are missing, are actually connected to headscale?
Author
Owner

@magkopian commented on GitHub (Jan 31, 2023):

When I first reported this I was able to SSH to these devices without a problem even though the MachineKey was missing from the headscale nodes list response. I'd like to note though that I SSHed to the server again just now, and most of the devices that were missing the MachineKey before now they don't.

We currently have a total of 157 devices on Headscale, from which 18 are currently offline. And from these, at least right now, only 4 miss the MachineKey. Also, it just so happens that these 4 are currently offline, but when I first reported the issue I had both online and offline devices that did miss the MachineKey.

If you need to me to provide any kind of logs or additional information please let me know.

@magkopian commented on GitHub (Jan 31, 2023): When I first reported this I was able to SSH to these devices without a problem even though the MachineKey was missing from the `headscale nodes list` response. I'd like to note though that I SSHed to the server again just now, and most of the devices that were missing the MachineKey before now they don't. We currently have a total of 157 devices on Headscale, from which 18 are currently offline. And from these, at least right now, only 4 miss the MachineKey. Also, it just so happens that these 4 are currently offline, but when I first reported the issue I had both online and offline devices that did miss the MachineKey. If you need to me to provide any kind of logs or additional information please let me know.
Author
Owner

@juanfont commented on GitHub (Jan 31, 2023):

Ok. I think this is expected.

For a bit we had a bug that prevented headscale from storing the MachineKey of a client. As soon as they become online again, they should get their MachineKey filled.

Can you try to make them log in again?

@juanfont commented on GitHub (Jan 31, 2023): Ok. I think this is expected. For a bit we had a bug that prevented headscale from storing the MachineKey of a client. As soon as they become online again, they should get their MachineKey filled. Can you try to make them log in again?
Author
Owner

@magkopian commented on GitHub (Feb 1, 2023):

Unfortunately, these specific 4 devices are not currently in our possession, so I can't just boot them up and check. I can ask a colleague though to notify me when any of them come online so I can check.

Regarding the bug you mentioned, in which version was it marked as fixed? Because I am using Headscale since v0.16.4.

@magkopian commented on GitHub (Feb 1, 2023): Unfortunately, these specific 4 devices are not currently in our possession, so I can't just boot them up and check. I can ask a colleague though to notify me when any of them come online so I can check. Regarding the bug you mentioned, in which version was it marked as fixed? Because I am using Headscale since [v0.16.4](https://github.com/juanfont/headscale/releases/tag/v0.16.4).
Author
Owner

@magkopian commented on GitHub (Mar 19, 2023):

Just wanted to make a quick update. Since the time I opened this issue we've added quite a few devices to the Tailnet and I haven't seen any sign of the issue reemerging. One of the 4 devices also came online at some point and now it has a MachineKey. So, I think it should be safe to close this issue.

@magkopian commented on GitHub (Mar 19, 2023): Just wanted to make a quick update. Since the time I opened this issue we've added quite a few devices to the Tailnet and I haven't seen any sign of the issue reemerging. One of the 4 devices also came online at some point and now it has a MachineKey. So, I think it should be safe to close this issue.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#415