Suspected bug with ip_prefixes #390

Closed
opened 2025-12-29 01:28:10 +01:00 by adam · 4 comments
Owner

Originally created by @mhameed on GitHub (Dec 6, 2022).

Hi,

Thanks for your work on headscale.

I would be grateful if someone could reproduce the the following, which seems like a bug to me:

headscale v0.17.0
tailscale v1.34.0
Ubuntu 22.04

Following headscale config file:

    ---
    noise:
      private_key_path: /var/lib/headscale/noise.key
    
    # The url clients will connect to.
    # Typically this will be a domain like:
    #
    # https://myheadscale.example.com:443
    #
    server_url: https://test1.ephemeral.host
    
    # Address to listen to / bind to on the server
    #
    listen_addr: 0.0.0.0:443
    
    # Address to listen to /metrics, you may want
    # to keep this endpoint private to your internal
    # network
    #
    metrics_listen_addr: 0.0.0.0:9090
    
    # Address to listen for gRPC.
    # gRPC is used for controlling a headscale server
    # remotely with the CLI
    # Note: Remote access _only_ works if you have
    # valid certificates.
    grpc_listen_addr: 0.0.0.0:50443
    
    # Allow the gRPC admin interface to run in INSECURE
    # mode. This is not recommended as the traffic will
    # be unencrypted. Only enable if you know what you
    # are doing.
    grpc_allow_insecure: false
    
    # Private key used encrypt the traffic between headscale
    # and Tailscale clients.
    # The private key file which will be
    # autogenerated if it's missing
    private_key_path: /var/lib/headscale/private.key
    
    # List of IP prefixes to allocate tailaddresses from.
    # Each prefix consists of either an IPv4 or IPv6 address,
    # and the associated prefix length, delimited by a slash.
    ip_prefixes:
      #- fd7a:115c:a1e0::/48
      - 172.19.0.0/23
      #- 100.64.0.0/16
    
    # DERP is a relay system that Tailscale uses when a direct
    # connection cannot be established.
    # https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp
    #
    # headscale needs a list of DERP servers that can be presented
    # to the clients.
    derp:
      server:
        # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config
        # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place
        enabled: true
    
        # Region ID to use for the embedded DERP server.
        # The local DERP prevails if the region ID collides with other region ID coming from
        # the regular DERP config.
        region_id: 999
    
        # Region code and name are displayed in the Tailscale UI to identify a DERP region
        region_code: "headscale"
        region_name: "Headscale Embedded DERP"
    
        # Listens in UDP at the configured address for STUN connections to help on NAT traversal.
        # When the embedded DERP server is enabled stun_listen_addr MUST be defined.
        #
        # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/
        stun_listen_addr: "0.0.0.0:3478"
    
      # List of externally available DERP maps encoded in JSON
      urls:
        - https://controlplane.tailscale.com/derpmap/default
    
      # Locally available DERP map files encoded in YAML
      #
      # This option is mostly interesting for people hosting
      # their own DERP servers:
      # https://tailscale.com/kb/1118/custom-derp-servers/
      #
      # paths:
      #   - /etc/headscale/derp-example.yaml
      paths: []
    
      # If enabled, a worker will be set up to periodically
      # refresh the given sources and update the derpmap
      # will be set up.
      auto_update_enabled: true
    
      # How often should we check for DERP updates?
      update_frequency: 24h
    
    # Disables the automatic check for headscale updates on startup
    disable_check_updates: true
    
    # Time before an inactive ephemeral node is deleted?
    ephemeral_node_inactivity_timeout: 30m
    
    # SQLite config
    db_type: sqlite3
    db_path: /var/lib/headscale/db.sqlite
    
    ### TLS configuration
    #
    # Client (Tailscale/Browser) authentication mode (mTLS)
    # Acceptable values:
    # - disabled: client authentication disabled
    # - relaxed: client certificate is required but not verified
    # - enforced: client certificate is required and verified
    tls_client_auth_mode: disabled
    
    ## Use already defined certificates:
    tls_cert_path: "/var/lib/headscale/certs/test1.ephemeral.host.crt"
    tls_key_path: "/var/lib/headscale/certs/test1.ephemeral.host.key"
    
    log:
      # Output formatting for logs: text or json
      format: text
      level: trace
    
    # Path to a file containg ACL policies.
    # ACLs can be defined as YAML or HUJSON.
    # https://tailscale.com/kb/1018/acls/
    acl_policy_path: ""
    
    ## DNS
    #
    # headscale supports Tailscale's DNS configuration and MagicDNS.
    # Please have a look to their KB to better understand the concepts:
    #
    # - https://tailscale.com/kb/1054/dns/
    # - https://tailscale.com/kb/1081/magicdns/
    # - https://tailscale.com/blog/2021-09-private-dns-with-magicdns/
    #
    dns_config:
      # List of DNS servers to expose to clients.
      nameservers:
        - 1.1.1.1
    
      # Split DNS (see https://tailscale.com/kb/1054/dns/),
      # list of search domains and the DNS to query for each one.
      #
      # restricted_nameservers:
      #   foo.bar.com:
      #     - 1.1.1.1
      #   darp.headscale.net:
      #     - 1.1.1.1
      #     - 8.8.8.8
    
      # Search domains to inject.
      domains: []
    
      # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/).
      # Only works if there is at least a nameserver defined.
      magic_dns: false
    
      # Defines the base domain to create the hostnames for MagicDNS.
      #  must be a FQDNs, without the trailing dot.
      # The FQDN of the hosts will be
      #  (e.g., _myhost.mynamespace.example.com_).
      base_domain: ephemeral.host
    
    # Unix socket used for the CLI to connect without authentication
    # Note: for local development, you probably want to change this to:
    # unix_socket: ./headscale.sock
    unix_socket: /var/run/headscale/headscale.sock
    unix_socket_permission: "0770"
    
    # Logtail configuration
    # Logtail is Tailscales logging and auditing infrastructure, it allows the control panel
    # to instruct tailscale nodes to log their activity to a remote server.
    logtail:
      # Enable logtail for this headscales clients.
      # As there is currently no support for overriding the log server in headscale, this is
      # disabled by default. Enabling this will make your clients send logs to Tailscale Inc.
      enabled: false
    
    randomize_client_port: true

registering nodes works fine, and headscale node list shows the expected ip addresses for the registered machines.
The problem is when you do tailscale status or tailscale ping:

    $ headscale node list
    2022-12-06T08:45:49Z DBG Setting timeout timeout=5000
    2022-12-06T08:45:49Z DBG HEADSCALE_CLI_ADDRESS environment is not set, connecting to unix socket. socket=/var/run/headscale/headscale.sock
    2022-12-06T08:45:49Z TRC ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:133 > Connecting via gRPC address=/var/run/headscale/headscale.sock
    ID | Hostname | Name  | NodeKey | Namespace | IP addresses | Ephemeral | Last seen           | Online | Expired
    1  | test1    | test1 | [j6yvQ] | test      | 172.19.0.1,  | false     | 2022-12-06 08:45:11 | online | no
    2  | test2    | test2 | [+dxgH] | test      | 172.19.0.2,  | false     | 2022-12-06 08:45:32 | online | no
    $ tailscale status
    172.19.0.1      test1                test         linux   -
                    test2                test         linux   -
    $ tailscale ping test2
    node found but lacks an IP
    $

Similarly from test2:

    $ tailscale status
    172.19.0.2      test2                test         linux   -
                    test1                test         linux   -
    $ tailscale ping test1
    node found but lacks an IP
    $

If the ip_prefixes is set to 100.64.0.0/16 it all works fine.

Thanks.

Originally created by @mhameed on GitHub (Dec 6, 2022). Hi, Thanks for your work on headscale. I would be grateful if someone could reproduce the the following, which seems like a bug to me: headscale v0.17.0 tailscale v1.34.0 Ubuntu 22.04 Following headscale config file: --- noise: private_key_path: /var/lib/headscale/noise.key # The url clients will connect to. # Typically this will be a domain like: # # https://myheadscale.example.com:443 # server_url: https://test1.ephemeral.host # Address to listen to / bind to on the server # listen_addr: 0.0.0.0:443 # Address to listen to /metrics, you may want # to keep this endpoint private to your internal # network # metrics_listen_addr: 0.0.0.0:9090 # Address to listen for gRPC. # gRPC is used for controlling a headscale server # remotely with the CLI # Note: Remote access _only_ works if you have # valid certificates. grpc_listen_addr: 0.0.0.0:50443 # Allow the gRPC admin interface to run in INSECURE # mode. This is not recommended as the traffic will # be unencrypted. Only enable if you know what you # are doing. grpc_allow_insecure: false # Private key used encrypt the traffic between headscale # and Tailscale clients. # The private key file which will be # autogenerated if it's missing private_key_path: /var/lib/headscale/private.key # List of IP prefixes to allocate tailaddresses from. # Each prefix consists of either an IPv4 or IPv6 address, # and the associated prefix length, delimited by a slash. ip_prefixes: #- fd7a:115c:a1e0::/48 - 172.19.0.0/23 #- 100.64.0.0/16 # DERP is a relay system that Tailscale uses when a direct # connection cannot be established. # https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp # # headscale needs a list of DERP servers that can be presented # to the clients. derp: server: # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place enabled: true # Region ID to use for the embedded DERP server. # The local DERP prevails if the region ID collides with other region ID coming from # the regular DERP config. region_id: 999 # Region code and name are displayed in the Tailscale UI to identify a DERP region region_code: "headscale" region_name: "Headscale Embedded DERP" # Listens in UDP at the configured address for STUN connections to help on NAT traversal. # When the embedded DERP server is enabled stun_listen_addr MUST be defined. # # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/ stun_listen_addr: "0.0.0.0:3478" # List of externally available DERP maps encoded in JSON urls: - https://controlplane.tailscale.com/derpmap/default # Locally available DERP map files encoded in YAML # # This option is mostly interesting for people hosting # their own DERP servers: # https://tailscale.com/kb/1118/custom-derp-servers/ # # paths: # - /etc/headscale/derp-example.yaml paths: [] # If enabled, a worker will be set up to periodically # refresh the given sources and update the derpmap # will be set up. auto_update_enabled: true # How often should we check for DERP updates? update_frequency: 24h # Disables the automatic check for headscale updates on startup disable_check_updates: true # Time before an inactive ephemeral node is deleted? ephemeral_node_inactivity_timeout: 30m # SQLite config db_type: sqlite3 db_path: /var/lib/headscale/db.sqlite ### TLS configuration # # Client (Tailscale/Browser) authentication mode (mTLS) # Acceptable values: # - disabled: client authentication disabled # - relaxed: client certificate is required but not verified # - enforced: client certificate is required and verified tls_client_auth_mode: disabled ## Use already defined certificates: tls_cert_path: "/var/lib/headscale/certs/test1.ephemeral.host.crt" tls_key_path: "/var/lib/headscale/certs/test1.ephemeral.host.key" log: # Output formatting for logs: text or json format: text level: trace # Path to a file containg ACL policies. # ACLs can be defined as YAML or HUJSON. # https://tailscale.com/kb/1018/acls/ acl_policy_path: "" ## DNS # # headscale supports Tailscale's DNS configuration and MagicDNS. # Please have a look to their KB to better understand the concepts: # # - https://tailscale.com/kb/1054/dns/ # - https://tailscale.com/kb/1081/magicdns/ # - https://tailscale.com/blog/2021-09-private-dns-with-magicdns/ # dns_config: # List of DNS servers to expose to clients. nameservers: - 1.1.1.1 # Split DNS (see https://tailscale.com/kb/1054/dns/), # list of search domains and the DNS to query for each one. # # restricted_nameservers: # foo.bar.com: # - 1.1.1.1 # darp.headscale.net: # - 1.1.1.1 # - 8.8.8.8 # Search domains to inject. domains: [] # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/). # Only works if there is at least a nameserver defined. magic_dns: false # Defines the base domain to create the hostnames for MagicDNS. # must be a FQDNs, without the trailing dot. # The FQDN of the hosts will be # (e.g., _myhost.mynamespace.example.com_). base_domain: ephemeral.host # Unix socket used for the CLI to connect without authentication # Note: for local development, you probably want to change this to: # unix_socket: ./headscale.sock unix_socket: /var/run/headscale/headscale.sock unix_socket_permission: "0770" # Logtail configuration # Logtail is Tailscales logging and auditing infrastructure, it allows the control panel # to instruct tailscale nodes to log their activity to a remote server. logtail: # Enable logtail for this headscales clients. # As there is currently no support for overriding the log server in headscale, this is # disabled by default. Enabling this will make your clients send logs to Tailscale Inc. enabled: false randomize_client_port: true registering nodes works fine, and headscale node list shows the expected ip addresses for the registered machines. The problem is when you do tailscale status or tailscale ping: $ headscale node list 2022-12-06T08:45:49Z DBG Setting timeout timeout=5000 2022-12-06T08:45:49Z DBG HEADSCALE_CLI_ADDRESS environment is not set, connecting to unix socket. socket=/var/run/headscale/headscale.sock 2022-12-06T08:45:49Z TRC ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:133 > Connecting via gRPC address=/var/run/headscale/headscale.sock ID | Hostname | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired 1 | test1 | test1 | [j6yvQ] | test | 172.19.0.1, | false | 2022-12-06 08:45:11 | online | no 2 | test2 | test2 | [+dxgH] | test | 172.19.0.2, | false | 2022-12-06 08:45:32 | online | no $ tailscale status 172.19.0.1 test1 test linux - test2 test linux - $ tailscale ping test2 node found but lacks an IP $ Similarly from test2: $ tailscale status 172.19.0.2 test2 test linux - test1 test linux - $ tailscale ping test1 node found but lacks an IP $ If the ip_prefixes is set to 100.64.0.0/16 it all works fine. Thanks.
adam added the bug label 2025-12-29 01:28:10 +01:00
adam closed this issue 2025-12-29 01:28:10 +01:00
Author
Owner

@kradalby commented on GitHub (Jan 11, 2023):

I think we might have actually implemented this without testing or having a purpose for this. It actually looks like it is not supported in the client https://github.com/search?q=repo%3Atailscale%2Ftailscale%20IsTailscaleIP&type=code.

I'll make a PR to indicate this in the docs.

@kradalby commented on GitHub (Jan 11, 2023): I think we might have actually implemented this without testing or having a purpose for this. It actually looks like it is not supported in the client https://github.com/search?q=repo%3Atailscale%2Ftailscale%20IsTailscaleIP&type=code. I'll make a PR to indicate this in the docs.
Author
Owner

@juanfont commented on GitHub (May 10, 2023):

Please use supported ranges.

@juanfont commented on GitHub (May 10, 2023): Please use supported ranges.
Author
Owner

@xrd commented on GitHub (Jul 2, 2023):

hi @juanfont Does this mean we cannot use two headscale networks? Is there a way to create two headscale networks and have a user connect to both and avoid conflicting IP addresses with other users on either network? I don't see a way to safely do that without having separate prefixes?

@xrd commented on GitHub (Jul 2, 2023): hi @juanfont Does this mean we cannot use two headscale networks? Is there a way to create two headscale networks and have a user connect to both and avoid conflicting IP addresses with other users on either network? I don't see a way to safely do that without having separate prefixes?
Author
Owner

@xrd commented on GitHub (Jul 2, 2023):

@juanfont I suppose this is the right way for now: https://tailscale.com/blog/fast-user-switching/

This works perfectly and is great for what I need!

@xrd commented on GitHub (Jul 2, 2023): @juanfont I suppose this is the right way for now: https://tailscale.com/blog/fast-user-switching/ This works perfectly and is great for what I need!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#390