starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

MagicDNS issues with systemd resolver configuration and Docker containers (Ubuntu) #416

New Issue
Closed
opened 2025-12-29 01:28:36 +01:00 by adam · 3 comments
adam commented 2025-12-29 01:28:36 +01:00
Owner
Copy Link

Originally created by @ky4n on GitHub (Jan 20, 2023).

Bug description

Short version:

Headscale with MagicDNS enabled appends 100.100.100.100 entry to system's resolve.conf (looks like getMapResponseDNSConfig is the method but I'm not a Go programmer). This can cause issues if there are already 3 name servers listed in the file (e.g. from DHCP). There's even a helpful message in the file itself (Too many DNS servers configured, the following entries may be ignored.). The nameserver being appended as a 4th entry causes MagicDNS resolution problems for Docker containers that are ran on the host.

Details:

After initial discovery, this was also tested on a fresh instance (Linode VM) of Ubuntu 20.04 after I've observed it on other servers already plugged into Headscale/Tailscale network.

Docker (docker-ce) and Tailscale were installed following the official instructions from official apt mirrors added on the host.

There are actually 2 files set up by systemd resolver:

  • /run/systemd/resolve/stub-resolv.conf which is symlinked to /etc/resolv.conf and used by the host.
  • /run/systemd/resolve/resolv.conf which corresponds to what Docker puts inside the container as /etc/resolv.conf

After fresh instance provisioning the values were as follows (skipping comments in them):
/run/systemd/resolve/resolv.conf

nameserver 212.71.253.5
nameserver 176.58.116.5
nameserver 109.74.192.20
search members.linode.com

Linode provides 3 nameservers, no room to add another.

/run/systemd/resolve/stub-resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search members.linode.com

After connecting to Headscale network the first file was changed to the following:

nameserver 212.71.253.5
nameserver 176.58.116.5
nameserver 109.74.192.20
# Too many DNS servers configured, the following entries may be ignored.
nameserver 100.100.100.100
search members.linode.com my.fancy.domain

This file is used as /etc/resolv.conf by Docker containers ran on the host and with 4 nameserver entries. When file is edited inside the container to put nameserver 100.100.100.100 as the first name server instead, DNS resolution inside the container starts working.

Both files are auto-generated by Systemd/Docker so editing them by hand is obviously not a proper solution. I suspect that changing the order to insert istead of appending the nameserver to the list should do the trick.

Interestingly enough, I've tried a Tailscale-only setup on the same host and when using Tailscale directly, /run/systemd/resolve/resolv.conf was as follows (search domain was added, but not any nameserver):

nameserver 212.71.253.5
nameserver 176.58.116.5
nameserver 109.74.192.20
search members.linode.com testingtailnet.ts.net

DNS resolution inside the container did not work and adding nameserver 100.100.100.100 as the first entry by hand made it work.

Expected behavior
DNS resolution for MagicDNS domains works inside Docker containers the same way as on a host that is connected to Headscale network.

To Reproduce

Context info

OS: Ubuntu 20.04
Kernel: 5.4.0-125-generic (and 5.4.0-136-generic)
Headscale version: v0.18.0
Tailscale version: 1.34.2
Docker package: 5:20.10.233-0ubuntu-focal

Headscale config, DNS part (MagicDNS enabled):

dns_config:
  nameservers:
    - 1.1.1.1
  domains: []
  magic_dns: true
  base_domain: "my.fancy.domain"
  extra_records:
     - < a couple, details probably not relevant >

The rest of the config is generally stock (I can provide it if needed).

Originally created by @ky4n on GitHub (Jan 20, 2023). **Bug description** Short version: Headscale with MagicDNS enabled appends `100.100.100.100` entry to system's `resolve.conf` (looks like `getMapResponseDNSConfig` is the method but I'm not a Go programmer). This can cause issues if there are already 3 name servers listed in the file (e.g. from DHCP). There's even a helpful message in the file itself (`Too many DNS servers configured, the following entries may be ignored.`). The nameserver being appended as a 4th entry causes MagicDNS resolution problems for Docker containers that are ran on the host. Details: After initial discovery, this was also tested on a fresh instance (Linode VM) of Ubuntu 20.04 after I've observed it on other servers already plugged into Headscale/Tailscale network. Docker (`docker-ce`) and Tailscale were installed following the official instructions from official apt mirrors added on the host. There are actually 2 files set up by systemd resolver: - `/run/systemd/resolve/stub-resolv.conf` which is symlinked to `/etc/resolv.conf` and used by the host. - `/run/systemd/resolve/resolv.conf` which corresponds to what Docker puts inside the container as `/etc/resolv.conf` After fresh instance provisioning the values were as follows (skipping comments in them): `/run/systemd/resolve/resolv.conf` ``` nameserver 212.71.253.5 nameserver 176.58.116.5 nameserver 109.74.192.20 search members.linode.com ``` Linode provides 3 nameservers, no room to add another. `/run/systemd/resolve/stub-resolv.conf` ``` nameserver 127.0.0.53 options edns0 trust-ad search members.linode.com ``` After connecting to Headscale network the first file was changed to the following: ``` nameserver 212.71.253.5 nameserver 176.58.116.5 nameserver 109.74.192.20 # Too many DNS servers configured, the following entries may be ignored. nameserver 100.100.100.100 search members.linode.com my.fancy.domain ``` This file is used as `/etc/resolv.conf` by Docker containers ran on the host and with 4 `nameserver` entries. When file is edited inside the container to put `nameserver 100.100.100.100` as the first name server instead, DNS resolution inside the container starts working. Both files are auto-generated by Systemd/Docker so editing them by hand is obviously not a proper solution. I suspect that changing the order to insert istead of appending the nameserver to the list should do the trick. Interestingly enough, I've tried a Tailscale-only setup on the same host and when using Tailscale directly, `/run/systemd/resolve/resolv.conf` was as follows (search domain was added, but not any nameserver): ``` nameserver 212.71.253.5 nameserver 176.58.116.5 nameserver 109.74.192.20 search members.linode.com testingtailnet.ts.net ``` DNS resolution inside the container did not work and adding `nameserver 100.100.100.100` as the first entry by hand made it work. --- **Expected behavior** DNS resolution for MagicDNS domains works inside Docker containers the same way as on a host that is connected to Headscale network. **To Reproduce** - Start with fresh Ubuntu 20.04 install as `root`. - Install Docker (`docker-ce`) from official sources (<https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository>) - Install Tailscale from official sources (<https://tailscale.com/kb/1039/install-ubuntu-2004/>) - Set up Headscale for Linux with MagicDNS enabled so there are some records to resolve `@100.100.100.100` - Test MagicDNS resolution on host - should work - start a simple container, e.g. `docker run -ti --rm ubuntu:20.04 bash`) - Test MagicDNS resolution inside the container - should not work **Context info** OS: Ubuntu 20.04 Kernel: 5.4.0-125-generic (and 5.4.0-136-generic) Headscale version: v0.18.0 Tailscale version: 1.34.2 Docker package: 5:20.10.23~3-0~ubuntu-focal Headscale config, DNS part (MagicDNS enabled): ```yaml dns_config: nameservers: - 1.1.1.1 domains: [] magic_dns: true base_domain: "my.fancy.domain" extra_records: - < a couple, details probably not relevant > ``` The rest of the config is generally stock (I can provide it if needed).
adam added the bug label 2025-12-29 01:28:36 +01:00
adam closed this issue 2025-12-29 01:28:36 +01:00
adam commented 2025-12-29 01:28:37 +01:00
Author
Owner
Copy Link

@ky4n commented on GitHub (Jan 22, 2023):

My workaround for this was to set DNS (and search domain) for each container explicitly, basically through equivalent of --dns Docker option.

The drawback is that it requires adjusting for all necessary containers (or possibly centrally in Docker daemon configuration, haven't tried it). I'm also not sure what else might be using non-stub configuration from systemd resolver and DHCP giving more than 2 name servers is something that can happen so it's a valid case.

@ky4n commented on GitHub (Jan 22, 2023): My workaround for this was to set DNS (and search domain) for each container explicitly, basically through equivalent of `--dns` Docker option. The drawback is that it requires adjusting for all necessary containers (or possibly centrally in Docker daemon configuration, haven't tried it). I'm also not sure what else might be using non-stub configuration from systemd resolver and DHCP giving more than 2 name servers is something that can happen so it's a valid case.
adam commented 2025-12-29 01:28:37 +01:00
Author
Owner
Copy Link

@kilo-gi commented on GitHub (Jun 8, 2023):

What network is the container running on? Does DNS work inside containers on a non-default bridge network?

@kilo-gi commented on GitHub (Jun 8, 2023): What network is the container running on? Does DNS work inside containers on a non-default bridge network?
adam commented 2025-12-29 01:28:38 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Jun 19, 2023):

Hi @ky4n

This seem like a support request and it would be great if you can as in the Docker part of Discord (https://discord.gg/c84AZQhmpx).

We (devs of headscale) has limited bandwidth to deal with support issues and the surface of the project and have a great community there to help us, since we also do not know or use Docker, we dont support it officially and its easier for us to let the community answer those things.

I will close this here, so we can track issues which we can action.

@kradalby commented on GitHub (Jun 19, 2023): Hi @ky4n This seem like a support request and it would be great if you can as in the Docker part of Discord (https://discord.gg/c84AZQhmpx). We (devs of headscale) has limited bandwidth to deal with support issues and the surface of the project and have a great community there to help us, since we also do not know or use Docker, we dont support it officially and its easier for us to let the community answer those things. I will close this here, so we can track issues which we can action.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#416
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?