[Bug] ERROR: Failed to extract ServerMetadata from context #795

Closed
opened 2025-12-29 02:24:06 +01:00 by adam · 16 comments
Owner

Originally created by @madhathacker on GitHub (Sep 13, 2024).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I am running Headscale in a docker container. For some reason after editing the config.yaml while Headscale was running, I start getting spammed with the error ERROR: Failed to extract ServerMetadata from context in the Docker logs. Restarting the service doesn't seem to stop this. I am also running headscale-ui and headscale-admin but I don't think they are the problem here.

Expected Behavior

Edit the config while the server is running.

Steps To Reproduce

  1. Start Headscale container.
  2. From docker host, open the config.yaml
  3. Make a change and write it to the config file

Environment

- OS: Ubuntu 22.04.4 LTS
- Docker: Version 25.0.3, build 4debf41
- Headscale version: headscale/headscale:v0.23.0-alpha12

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

I was simply trying to add OIDC to headscale by editing the config.

Originally created by @madhathacker on GitHub (Sep 13, 2024). ### Is this a support request? - [X] This is not a support request ### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior I am running Headscale in a docker container. For some reason after editing the `config.yaml` while Headscale was running, I start getting spammed with the error `ERROR: Failed to extract ServerMetadata from context` in the Docker logs. Restarting the service doesn't seem to stop this. I am also running `headscale-ui` and `headscale-admin` but I don't think they are the problem here. ### Expected Behavior Edit the config while the server is running. ### Steps To Reproduce 1. Start Headscale container. 2. From docker host, open the config.yaml 3. Make a change and write it to the config file ### Environment ```markdown - OS: Ubuntu 22.04.4 LTS - Docker: Version 25.0.3, build 4debf41 - Headscale version: headscale/headscale:v0.23.0-alpha12 ``` ### Runtime environment - [X] Headscale is behind a (reverse) proxy - [X] Headscale runs in a container ### Anything else? I was simply trying to add OIDC to headscale by editing the config.
adam added the stalebugfaq labels 2025-12-29 02:24:06 +01:00
adam closed this issue 2025-12-29 02:24:06 +01:00
Author
Owner

@kradalby commented on GitHub (Sep 13, 2024):

Can you please try the latest beta, and ensure that the CLI binary and the server binary is the same version?

@kradalby commented on GitHub (Sep 13, 2024): Can you please try the latest beta, and ensure that the CLI binary and the server binary is the same version?
Author
Owner

@agarzon-orangerine commented on GitHub (Sep 17, 2024):

I see the same error, Im using docker image: headscale/headscale:0.23.0-rc.1 (latest at this moment)

My compose is quite straight forward:

services:
  headscale:
    image: headscale/headscale:0.23.0-rc.1
    restart: unless-stopped
    container_name: headscale
    environment:
      - TZ=America/Toronto
    volumes:
      - ./conf:/etc/headscale
      - headscale_data:/var/lib/headscale
    command: serve
    labels:
      - traefik.enable=true
      - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.rule=Host(`${DOMAIN}`)
      - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.tls=true
      - traefik.http.services.${COMPOSE_PROJECT_NAME}_headscale.loadbalancer.server.port=8080
      
volumes:
  headscale_data:

networks:
  default:
    external: true
    name: proxy

@agarzon-orangerine commented on GitHub (Sep 17, 2024): I see the same error, Im using docker image: headscale/headscale:0.23.0-rc.1 (latest at this moment) My compose is quite straight forward: ```yml services: headscale: image: headscale/headscale:0.23.0-rc.1 restart: unless-stopped container_name: headscale environment: - TZ=America/Toronto volumes: - ./conf:/etc/headscale - headscale_data:/var/lib/headscale command: serve labels: - traefik.enable=true - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.rule=Host(`${DOMAIN}`) - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.tls=true - traefik.http.services.${COMPOSE_PROJECT_NAME}_headscale.loadbalancer.server.port=8080 volumes: headscale_data: networks: default: external: true name: proxy ```
Author
Owner

@kradalby commented on GitHub (Sep 18, 2024):

This sounds proxy related, I found this from the upstream gRPC project we use: https://github.com/grpc-ecosystem/grpc-gateway/issues/2445

I've tested with my own server, putting it behind a nginx proxy I was unable to reproduce.

Please be mindful that we do not have the capacity to test headscale behind the X x Y x Z combinations of proxies and configuration, which is why it is not supported. You can manage, but your miles might vary.

We did a dependency update to that library, and that might be related, but the update of dependencies is more important than novel proxy configurations.

@kradalby commented on GitHub (Sep 18, 2024): This sounds proxy related, I found this from the upstream gRPC project we use: https://github.com/grpc-ecosystem/grpc-gateway/issues/2445 I've tested with my own server, putting it behind a nginx proxy I was unable to reproduce. Please be mindful that we do not have the capacity to test headscale behind the X x Y x Z combinations of proxies and configuration, which is _why it is not supported_. You can manage, but your miles might vary. We did a dependency update to that library, and that might be related, but the update of dependencies is more important than novel proxy configurations.
Author
Owner

@Typhon0 commented on GitHub (Sep 21, 2024):

Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it.

- Headscale v0.23.0
- Traefik 3.4.1
@Typhon0 commented on GitHub (Sep 21, 2024): Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it. ``` - Headscale v0.23.0 - Traefik 3.4.1 ```
Author
Owner

@kradalby commented on GitHub (Sep 22, 2024):

Sounds like this is out of our hands, but should be mentioned in an faq or common proxy issues.

@kradalby commented on GitHub (Sep 22, 2024): Sounds like this is out of our hands, but should be mentioned in an faq or common proxy issues.
Author
Owner

@davidhan888-code commented on GitHub (Sep 26, 2024):

Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it.

- Headscale v0.23.0
- Traefik 3.4.1

Thanks for the hint.
I still get the same error. Besides it is 3.1.4 rather than 3.4.1

@davidhan888-code commented on GitHub (Sep 26, 2024): > Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it. > > ``` > - Headscale v0.23.0 > - Traefik 3.4.1 > ``` Thanks for the hint. I still get the same error. Besides it is 3.1.4 rather than 3.4.1
Author
Owner

@appleimperio commented on GitHub (Sep 27, 2024):

I'm having the same issue with Nginx Proxy Manager and headscale 0.23.0

@appleimperio commented on GitHub (Sep 27, 2024): I'm having the same issue with Nginx Proxy Manager and headscale 0.23.0
Author
Owner

@skarpinis commented on GitHub (Sep 30, 2024):

yes same issue with NPM

@skarpinis commented on GitHub (Sep 30, 2024): yes same issue with NPM
Author
Owner

@appleimperio commented on GitHub (Sep 30, 2024):

In my case the problem was with the integration with a webui. I changed to headscale-admin and the error disappear.

@appleimperio commented on GitHub (Sep 30, 2024): In my case the problem was with the integration with a webui. I changed to headscale-admin and the error disappear.
Author
Owner

@kradalby commented on GitHub (Sep 30, 2024):

I would like to emphasis that we will not be taking any action to resolve this, the error seems to be as a result of us updating a dependency, in this case gRPC related. Since it is not an alternative to change it, nor to not update them, there is little we can or will attempt doing.

We often underline that we do not support proxies, one of the reasons is this, it seems particularly tricky to get gRPC working correctly. In addition, Tailscale's protocol does not always play nice with proxies either (as per https://github.com/juanfont/headscale/issues/1468).

I will probably close it a bit, but need to think about how to resurface it so we dont get new issues created about the same error.

@kradalby commented on GitHub (Sep 30, 2024): I would like to emphasis that we will not be taking any action to resolve this, the error seems to be as a result of us updating a dependency, in this case gRPC related. Since it is not an alternative to change it, nor to not update them, there is little we can or will attempt doing. We often underline that we do not support proxies, one of the reasons is this, it seems particularly tricky to get gRPC working correctly. In addition, Tailscale's protocol does not always play nice with proxies either (as per https://github.com/juanfont/headscale/issues/1468). I will probably close it a bit, but need to think about how to resurface it so we dont get new issues created about the same error.
Author
Owner

@skarpinis commented on GitHub (Sep 30, 2024):

I will ignore no another option

@skarpinis commented on GitHub (Sep 30, 2024): I will ignore no another option
Author
Owner

@reey commented on GitHub (Sep 30, 2024):

FYI: had the same issue with https://github.com/ifargle/headscale-webui and traefik, but no problems using: https://github.com/GoodiesHQ/headscale-admin together with traefik

@reey commented on GitHub (Sep 30, 2024): FYI: had the same issue with https://github.com/ifargle/headscale-webui and traefik, but no problems using: https://github.com/GoodiesHQ/headscale-admin together with traefik
Author
Owner

@dreem2001 commented on GitHub (Oct 10, 2024):

It can be caused enabled legacy API <23 in settings of Headscale-admin.

@dreem2001 commented on GitHub (Oct 10, 2024): It can be caused enabled legacy API <23 in settings of Headscale-admin.
Author
Owner

@appleimperio commented on GitHub (Oct 11, 2024):

@dreem2001 That was exactly my issue. I disable it and everything start working

@appleimperio commented on GitHub (Oct 11, 2024): @dreem2001 That was exactly my issue. I disable it and everything start working
Author
Owner

@github-actions[bot] commented on GitHub (Jan 10, 2025):

This issue is stale because it has been open for 90 days with no activity.

@github-actions[bot] commented on GitHub (Jan 10, 2025): This issue is stale because it has been open for 90 days with no activity.
Author
Owner

@github-actions[bot] commented on GitHub (Jan 18, 2025):

This issue was closed because it has been inactive for 14 days since being marked as stale.

@github-actions[bot] commented on GitHub (Jan 18, 2025): This issue was closed because it has been inactive for 14 days since being marked as stale.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#795