CA Certificates missing #690

Originally created by @sunny-logic on GitHub (Apr 12, 2024).

Bug description

Does the "v0.23.0-alpha5-debug" have the ca-certificates added because I see the below error which was reported in https://github.com/juanfont/headscale/issues/1462 and I believe was fixed in https://github.com/juanfont/headscale/pull/1463.

ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default

I can't use v0.22.3 because it's doesn't have an ARMv7 variant.

Environment

Version of headscale used - "v0.23.0-alpha5-debug"
OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version - Linux 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
Log output
ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default

Headscale is behind a (reverse) proxy
Headscale runs in a container

To Reproduce

Use "v0.23.0-alpha5-debug"

Logs and attachments

Originally created by @sunny-logic on GitHub (Apr 12, 2024).  ## Bug description  Does the "v0.23.0-alpha5-debug" have the ca-certificates added because I see the below error which was reported in https://github.com/juanfont/headscale/issues/1462 and I believe was fixed in https://github.com/juanfont/headscale/pull/1463. `ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default ` I can't use [v0.22.3](https://github.com/juanfont/headscale/releases/tag/v0.22.3) because it's doesn't have an ARMv7 variant. ## Environment  - Version of headscale used - "v0.23.0-alpha5-debug" - OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version - `Linux 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux` - Log output `ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default `  - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ## To Reproduce  Use "v0.23.0-alpha5-debug" ## Logs and attachments

adam added the bug label 2025-12-29 02:22:06 +01:00

adam closed this issue

adam commented

@ohdearaugustin commented on GitHub (Apr 21, 2024):

Does this problem appear with the let's encrypt certificate or your own certificate?

The container are not build with the Dockerfile anymore. We use the buildin from goreleaser

@ohdearaugustin commented on GitHub (Apr 21, 2024): Does this problem appear with the let's encrypt certificate or your own certificate? The container are not build with the Dockerfile anymore. We use the buildin from goreleaser

adam commented

@sunny-logic commented on GitHub (Apr 22, 2024):

@ohdearaugustin -Thanks for the reply.
Yes, I am using let's encrypt certificate with Headscale docker.

@sunny-logic commented on GitHub (Apr 22, 2024): @ohdearaugustin -Thanks for the reply. Yes, I am using let's encrypt certificate with Headscale docker.

adam commented

@ypurpl commented on GitHub (Apr 25, 2024):

I also have this error on a fresh install. I pulled from docker hub and have default config.yaml. I don't know what certificate im using but I changed the URL to 0.0.0.0:8080:8080 for external use

Logs:

2024-04-25 17:09:08 2024-04-25T21:09:08Z TRC DNS configuration loaded dns_config={"Nameservers":["1.1.1.1"],"Proxied":true,"Resolvers":[{"Addr":"1.1.1.1"}]}
2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite
2024-04-25 17:09:08 2024-04-25T21:09:08Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default
2024-04-25 17:09:08 2024-04-25T21:09:08Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region
2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Setting up a DERPMap update worker frequency=86400000
2024-04-25 17:09:08 2024-04-25T21:09:08Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="initial DERPMap is empty, Headscale requires at least one entry"

@ypurpl commented on GitHub (Apr 25, 2024): I also have this error on a fresh install. I pulled from docker hub and have default config.yaml. I don't know what certificate im using but I changed the URL to 0.0.0.0:8080:8080 for external use Logs: ```shell 2024-04-25 17:09:08 2024-04-25T21:09:08Z TRC DNS configuration loaded dns_config={"Nameservers":["1.1.1.1"],"Proxied":true,"Resolvers":[{"Addr":"1.1.1.1"}]} 2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite 2024-04-25 17:09:08 2024-04-25T21:09:08Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default 2024-04-25 17:09:08 2024-04-25T21:09:08Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region 2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Setting up a DERPMap update worker frequency=86400000 2024-04-25 17:09:08 2024-04-25T21:09:08Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="initial DERPMap is empty, Headscale requires at least one entry" ```

adam commented

@ypurpl commented on GitHub (Apr 25, 2024):

Should I manually import the derp map locally as a temporary fix?

@ypurpl commented on GitHub (Apr 25, 2024): Should I manually import the derp map locally as a temporary fix?

adam commented

@ypurpl commented on GitHub (Apr 25, 2024):

I also realized that I might be the extra backslash added at the end of the URL

@ypurpl commented on GitHub (Apr 25, 2024): I also realized that I might be the extra backslash added at the end of the URL

adam commented

@sunny-logic commented on GitHub (Apr 28, 2024):

@ohdearaugustin - Sorry, It's not clear from your previous comment if this issue would this be fixed ?

The container are not build with the Dockerfile anymore. We use the buildin from goreleaser

@sunny-logic commented on GitHub (Apr 28, 2024): @ohdearaugustin - Sorry, It's not clear from your previous comment if this issue would this be fixed ? `The container are not build with the Dockerfile anymore. We use the buildin from goreleaser`

adam commented

@ypurpl commented on GitHub (Apr 28, 2024):

do you mean https://github.com/juanfont/headscale/pkgs/container/headscale ?

@ypurpl commented on GitHub (Apr 28, 2024): do you mean https://github.com/juanfont/headscale/pkgs/container/headscale ?

adam commented

@ypurpl commented on GitHub (Apr 28, 2024):

its updated the same as the docker hub image so guess not

@ypurpl commented on GitHub (Apr 28, 2024): its updated the same as the docker hub image so guess not

adam commented

@sunny-logic commented on GitHub (Apr 29, 2024):

I can't use v0.22.3 because it's doesn't have an ARMv7 variant.
I cant use "v0.23.0-alpha5-debug" because it doesn't have the ca-certificates.

Are there any other options that would support ARMv7 ?

@sunny-logic commented on GitHub (Apr 29, 2024): I can't use [v0.22.3](https://github.com/juanfont/headscale/releases/tag/v0.22.3) because it's doesn't have an ARMv7 variant. I cant use "v0.23.0-alpha5-debug" because it doesn't have the ca-certificates. Are there any other options that would support ARMv7 ?

adam commented

@ypurpl commented on GitHub (Apr 29, 2024):

oh is it the debug versions that don't have them?

@ypurpl commented on GitHub (Apr 29, 2024): oh is it the debug versions that don't have them?

adam commented

@ypurpl commented on GitHub (Apr 29, 2024):

ok that solves my problem

@ypurpl commented on GitHub (Apr 29, 2024): ok that solves my problem

adam commented

@sunny-logic commented on GitHub (Apr 30, 2024):

I've tried today to build a an image for the verion v0.22.3 because it's doesn't have an ARMv7 variant. I've used the Source code zip but encounter a certificate issue.

`=> [internal] load build context 0.4s
=> => transferring context: 9.81kB 0.3s
=> ERROR [stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lis 6.7s

[stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean:
2.600 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
2.809 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
2.809 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
3.461 Err:1 http://deb.debian.org/debian bullseye InRelease
3.461 At least one invalid signature was encountered.
3.989 Err:2 http://deb.debian.org/debian-security bullseye-security InRelease
3.989 At least one invalid signature was encountered.
4.578 Err:3 http://deb.debian.org/debian bullseye-updates InRelease
4.578 At least one invalid signature was encountered.
4.609 Reading package lists...
4.738 W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered.
4.738 E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed.
4.738 W: GPG error: http://deb.debian.org/debian-security bullseye-security InRelease: At least one invalid signature was encountered.
4.738 E: The repository 'http://deb.debian.org/debian-security bullseye-security InRelease' is not signed.
4.738 W: GPG error: http://deb.debian.org/debian bullseye-updates InRelease: At least one invalid signature was encountered.
4.738 E: The repository 'http://deb.debian.org/debian bullseye-updates InRelease' is not signed.
Dockerfile:19
18 |
19 | >>> RUN apt-get update
20 | >>> && apt-get install -y ca-certificates
21 | >>> && rm -rf /var/lib/apt/lists/*
22 | >>> && apt-get clean
23 |
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean" did not complete successfully: exit code: 100
`

@sunny-logic commented on GitHub (Apr 30, 2024): I've tried today to build a an image for the verion [v0.22.3](https://github.com/juanfont/headscale/releases/tag/v0.22.3) because it's doesn't have an ARMv7 variant. I've used the [ Source code zip](https://github.com/juanfont/headscale/archive/refs/tags/v0.22.3.zip) but encounter a certificate issue. `=> [internal] load build context 0.4s => => transferring context: 9.81kB 0.3s => ERROR [stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lis 6.7s > [stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean: 2.600 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB] 2.809 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB] 2.809 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] 3.461 Err:1 http://deb.debian.org/debian bullseye InRelease 3.461 At least one invalid signature was encountered. 3.989 Err:2 http://deb.debian.org/debian-security bullseye-security InRelease 3.989 At least one invalid signature was encountered. 4.578 Err:3 http://deb.debian.org/debian bullseye-updates InRelease 4.578 At least one invalid signature was encountered. 4.609 Reading package lists... 4.738 W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed. 4.738 W: GPG error: http://deb.debian.org/debian-security bullseye-security InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian-security bullseye-security InRelease' is not signed. 4.738 W: GPG error: http://deb.debian.org/debian bullseye-updates InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian bullseye-updates InRelease' is not signed. Dockerfile:19 18 | 19 | >>> RUN apt-get update \ 20 | >>> && apt-get install -y ca-certificates \ 21 | >>> && rm -rf /var/lib/apt/lists/* \ 22 | >>> && apt-get clean 23 | ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean" did not complete successfully: exit code: 100 `

adam commented

@tuxpeople commented on GitHub (May 15, 2024):

I can confirm the issue. When using docker.io/headscale/headscale:0.23.0-alpha10-debug with default config, Headscale can not download the derp map from Tailscale:

headscale     | 2024-05-15T09:50:31Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default
headscale     | 2024-05-15T09:50:31Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region

When building my own version like the following, it works:

FROM docker.io/headscale/headscale:0.23.0-alpha10-debug
RUN apt-get update \
  && apt-get install --no-install-recommends --yes ca-certificates \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean

@tuxpeople commented on GitHub (May 15, 2024): I can confirm the issue. When using `docker.io/headscale/headscale:0.23.0-alpha10-debug` with default config, Headscale can not download the derp map from Tailscale: ``` headscale | 2024-05-15T09:50:31Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default headscale | 2024-05-15T09:50:31Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region ``` When building my own version like the following, it works: ```Dockerfile FROM docker.io/headscale/headscale:0.23.0-alpha10-debug RUN apt-get update \ && apt-get install --no-install-recommends --yes ca-certificates \ && rm -rf /var/lib/apt/lists/* \ && apt-get clean ```

adam commented

@ohdearaugustin commented on GitHub (May 18, 2024):

So just to summarize only the debug image has the problem with the CA.

The production image is working?

@ohdearaugustin commented on GitHub (May 18, 2024): So just to summarize only the debug image has the problem with the CA. The production image is working?

adam commented