Running headscale commands as non-root produces generic "deadline exceeded" error message #245

Closed
opened 2025-12-29 01:24:51 +01:00 by adam · 9 comments
Owner

Originally created by @victorhooi on GitHub (Mar 23, 2022).

Bug description

If you attempt to run "headscale" commands as a non-root user, it prints out a generic and somewhat unhelpful "context deadline exceeded" error message:

victorhooi@headscale-test:~$ headscale nodes list
2022-03-23T08:56:46Z FTL ../runner/work/headscale/headscale/cmd/headscale/cli/utils.go:482 > Could not connect: context deadline exceeded error="context deadline exceeded"

This is rather unhelpful, and doesn't really tell the user where the problem is.

I upped the logging level to "trace", and noticed that it was trying to access /var/run/headscale.sock - I re-ran with sudo, and the command worked:

victorhooi@headscale-test:~$ sudo headscale nodes list
ID | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired

Ideally:

  1. It should be documented somewhere (e.g. README) that all headscale commands, including things like listing nodes need to run as the root user
  2. headscale should detect when run as a non-root user, and print out a friendlier error message. (e.g. `headscale requires permissions to run. Please run as root")
  3. Potentially if there's some way of running headscale as non-root - e.g. by adding instructions to README on setting up a *nix group with the right permissions needed etc. - or maybe changing permissions on the headscale.sock file - this should be documented.

To Reproduce

Run a headscale command as a normal (non-root) user.

Context info

Originally created by @victorhooi on GitHub (Mar 23, 2022). **Bug description** If you attempt to run "headscale" commands as a non-root user, it prints out a generic and somewhat unhelpful "context deadline exceeded" error message: ``` victorhooi@headscale-test:~$ headscale nodes list 2022-03-23T08:56:46Z FTL ../runner/work/headscale/headscale/cmd/headscale/cli/utils.go:482 > Could not connect: context deadline exceeded error="context deadline exceeded" ``` This is rather unhelpful, and doesn't really tell the user where the problem is. I upped the logging level to "trace", and noticed that it was trying to access `/var/run/headscale.sock` - I re-ran with `sudo`, and the command worked: ``` victorhooi@headscale-test:~$ sudo headscale nodes list ID | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired ``` Ideally: 1. It should be documented somewhere (e.g. README) that all headscale commands, including things like listing nodes need to run as the `root` user 2. `headscale` should detect when run as a non-root user, and print out a friendlier error message. (e.g. `headscale requires <x> permissions to run. Please run as root") 3. Potentially if there's some way of running headscale as non-root - e.g. by adding instructions to README on setting up a *nix group with the right permissions needed etc. - or maybe changing permissions on the `headscale.sock` file - this should be documented. **To Reproduce** Run a `headscale` command as a normal (non-root) user. **Context info** <!-- Please add relevant information about your system. For example: - Version of headscale used: 0.15.0 - Version of tailscale client: N/A - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version: Debian 11 (Bullseye) - Kernel version: 5.10.0-12-cloud-amd64 - The relevant config parameters you used - Log output -->
adam added the bug label 2025-12-29 01:24:51 +01:00
adam closed this issue 2025-12-29 01:24:51 +01:00
Author
Owner

@e-zk commented on GitHub (Mar 23, 2022):

Hello! headscale runs as a non-root user and seems to work fine in my setup.

I think what's wrong here could be your file permissions on the headscale related stuff in /var/*. Could you run a quick ls -l on those files to make sure that isn't the case?

ls -l /var/run/headscale.sock

Edit: I see you've actually mentioned this in your post - adding docs about how to change these permissions seems like a good idea. I'll see what I can do.

@e-zk commented on GitHub (Mar 23, 2022): Hello! headscale runs as a non-root user and seems to work fine in my setup. I think what's wrong here could be your file permissions on the headscale related stuff in `/var/*`. Could you run a quick `ls -l` on those files to make sure that isn't the case? ls -l /var/run/headscale.sock Edit: I see you've actually mentioned this in your post - adding docs about how to change these permissions seems like a good idea. I'll see what I can do.
Author
Owner

@victorhooi commented on GitHub (Mar 24, 2022):

Here are the permissions for the socket file on my system:

$ ls -l /var/run/headscale.sock
srwxrwx--- 1 root root 0 Mar 23 16:46 /var/run/headscale.sock

This is on a Debian 11 machine.

From memory, when I was following along the docs at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - I had to use sudo to run the mkdir commands:

sudo mkdir -p /var/lib/headscale

(I assume that's expected behaviour, right?)

Or is there another way we should create those directories, with user access?

@victorhooi commented on GitHub (Mar 24, 2022): Here are the permissions for the socket file on my system: ``` $ ls -l /var/run/headscale.sock srwxrwx--- 1 root root 0 Mar 23 16:46 /var/run/headscale.sock ``` This is on a Debian 11 machine. From memory, when I was following along the docs at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - I had to use `sudo` to run the `mkdir` commands: `sudo mkdir -p /var/lib/headscale` (I assume that's expected behaviour, right?) Or is there another way we should create those directories, with user access?
Author
Owner

@e-zk commented on GitHub (Mar 24, 2022):

Yeah so I think that's the root (heh) of the problem.

When you've created that directory with sudo it's given ownership rights to root and root only.

If you're running headscale as a non-root user you should change the ownership of this dir to that non-root user.

Try running:

# chown -R hscale:hscale /var/lib/headscale

Where hscale is the username of your non-root user headscale is running as.

@e-zk commented on GitHub (Mar 24, 2022): Yeah so I think that's the root (heh) of the problem. When you've created that directory with sudo it's given ownership rights to root and root only. If you're running headscale as a non-root user you should change the ownership of this dir to that non-root user. Try running: ``` # chown -R hscale:hscale /var/lib/headscale ``` Where `hscale` is the username of your non-root user headscale is running as.
Author
Owner

@victorhooi commented on GitHub (Mar 25, 2022):

Hmm - ok, so you're saying we should create a dedicated headscale (or hscale) user and group, and set ownership of the headscale directories (/etc/headscale, /var/lib/headscale to that.). Should the /usr/local/bin/headscale binary also be set to belong to headscale:headscale?

If this is the recommended approach - we should document it in https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md.

There is mention of a headscale user, but it's a passing reference at the bottom of the page in the systemd section. I suspect many people are like me, and just start at the "Configure and run headscale" section at the top 😛, and follow the instructions.

@victorhooi commented on GitHub (Mar 25, 2022): Hmm - ok, so you're saying we should create a dedicated `headscale` (or `hscale`) user and group, and set ownership of the headscale directories (`/etc/headscale`, `/var/lib/headscale` to that.). Should the `/usr/local/bin/headscale` binary also be set to belong to `headscale:headscale`? If this is the recommended approach - we should document it in https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md. There is mention of a headscale user, but it's a passing reference at the bottom of the page in the systemd section. I suspect many people are like me, and just start at the "Configure and run headscale" section at the top 😛, and follow the instructions.
Author
Owner

@e-zk commented on GitHub (Mar 26, 2022):

I'm not saying you should create a dedicated headscale user (unless you want to - I personally do).

You said you are running headscale as a non-root user, correct? What I am saying, is to make sure that non-root user has the right permissions to access the config files and stuff in /var.

@e-zk commented on GitHub (Mar 26, 2022): I'm not saying you _should_ create a dedicated headscale user (unless you want to - I personally do). You said you are running headscale as a non-root user, correct? What I am saying, is to make sure that non-root user has the right permissions to access the config files and stuff in `/var`.
Author
Owner

@e-zk commented on GitHub (Mar 26, 2022):

Because that non-root user currently can't access anything in /var, because it's all owned by root. You understand that, right?

@e-zk commented on GitHub (Mar 26, 2022): Because that non-root user currently can't access anything in /var, because it's all owned by `root`. You understand that, right?
Author
Owner

@victorhooi commented on GitHub (Mar 29, 2022):

Yup, I do understand it =) - thanks for clarifying.

My question is more - if somebody is just blindly following the instructions at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - they're going to need to run headscale as root, which sounds like it's not necessarily a great idea.

Would it make sense to add some steps there - possibly just add a headscale user, and set appropriate ownerships on the folders/files?

@victorhooi commented on GitHub (Mar 29, 2022): Yup, I do understand it =) - thanks for clarifying. My question is more - if somebody is just blindly following the instructions at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - they're going to need to run `headscale` as root, which sounds like it's not necessarily a great idea. Would it make sense to add some steps there - possibly just add a headscale user, and set appropriate ownerships on the folders/files?
Author
Owner

@thenitai commented on GitHub (Apr 14, 2022):

hmmmm........ I came here because of that "context deadline exceeded" issue. However in my experience it's because I'm running headscale with systemd.

According to the docs you would need to change the path for the sock. However, when you do that the command "headscale namespaces list" throws that error.

As soon as you stop the deamon. Change the socks path and start with "headscale server" all commands work again.

@thenitai commented on GitHub (Apr 14, 2022): hmmmm........ I came here because of that "context deadline exceeded" issue. However in my experience it's because I'm running headscale with systemd. According to the docs you would need to change the path for the sock. However, when you do that the command "headscale namespaces list" throws that error. As soon as you stop the deamon. Change the socks path and start with "headscale server" all commands work again.
Author
Owner

@thenitai commented on GitHub (Apr 14, 2022):

Ok, I got it to work. All that is required is to create the directory for the sock:

mkdir -p /var/run/headscale

and then it works with the path to the socket as described in the docs, i.e., unix_socket: /var/run/headscale/headscale.sock

@thenitai commented on GitHub (Apr 14, 2022): Ok, I got it to work. All that is required is to create the directory for the sock: ``` mkdir -p /var/run/headscale ``` and then it works with the path to the socket as described in the docs, i.e., unix_socket: /var/run/headscale/headscale.sock
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#245