Update documentation to reflect the new concurrent test execution
capabilities and add guidance on run ID isolation.
AGENTS.md:
- Add examples for running multiple tests concurrently
- Document run ID format and container naming conventions
- Update "Critical Notes" to explain isolation mechanisms
.claude/agents/headscale-integration-tester.md:
- Add "Concurrent Execution and Run ID Isolation" section
- Document forbidden and safe operations for cleanup
- Add "Agent Session Isolation Rules" for multi-agent environments
- Add 6th core responsibility about concurrent execution awareness
- Add ISOLATION PRINCIPLE to critical principles
- Update pre-test cleanup documentation
Remove the concurrent test prevention logic and update cleanup to use
run ID-based isolation, allowing multiple tests to run simultaneously.
Changes:
- cleanup: Add killTestContainersByRunID() to clean only containers
belonging to a specific run, add cleanupStaleTestContainers() to
remove only stopped/exited containers without affecting running tests
- docker: Remove RunningTestInfo, checkForRunningTests(), and related
error types, update cleanupAfterTest() to use run ID-based cleanup
- run: Remove Force flag and concurrent test prevention check
The test runner now:
- Allows multiple concurrent test runs on the same Docker daemon
- Cleans only stale containers before tests (not running ones)
- Cleans only containers with matching run ID after tests
- Prints run ID and monitoring info for operator visibility
Add run ID-based isolation to container naming and network setup to
enable multiple integration tests to run concurrently on the same
Docker daemon without conflicts.
Changes:
- hsic: Add run ID prefix to headscale container names and use dynamic
port allocation for metrics endpoint (port 0 lets kernel assign)
- tsic: Add run ID prefix to tailscale container names
- dsic: Add run ID prefix to DERP container names
- scenario: Use run ID-aware test suite container name for network setup
Container naming now follows: {type}-{runIDShort}-{identifier}-{hash}
Example: ts-mdjtzx-1-74-fgdyls, hs-mdjtzx-pingallbyip-abc123
The run ID is obtained from HEADSCALE_INTEGRATION_RUN_ID environment
variable via dockertestutil.GetIntegrationRunID().
This commit replaces the ChangeSet with a simpler bool based
change model that can be directly used in the map builder to
build the appropriate map response based on the change that
has occured. Previously, we fell back to sending full maps
for a lot of changes as that was consider "the safe" thing to
do to ensure no updates were missed.
This was slightly problematic as a node that already has a list
of peers will only do full replacement of the peers if the list
is non-empty, meaning that it was not possible to remove all
nodes (if for example policy changed).
Now we will keep track of last seen nodes, so we can send remove
ids, but also we are much smarter on how we send smaller, partial
maps when needed.
Fixes#2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit adds tests to validate that there are
issues with how we propagate tag changes in the system.
This replicates #2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit changes so that node changes to the policy is
calculated if any of the nodes has changed in a way that might
affect the policy.
Previously we just checked if the number of nodes had changed,
which meant that if a node was added and removed, we would be
in a bad state.
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This PR restructures the integration tests and prebuilds all common assets used in all tests:
Headscale and Tailscale HEAD image
hi binary that is used to run tests
go cache is warmed up for compilation of the test
This essentially means we spend 6-10 minutes building assets before any tests starts, when that is done, all tests can just sprint through.
It looks like we are saving 3-9 minutes per test, and since we are limited to running max 20 concurrent tests across the repo, that means we had a lot of double work.
There is currently 113 checks, so we have to do five runs of 20, and the saving should be quite noticeable! I think the "worst case" saving would be 20+min and "best case" probably towards an hour.
Fixes#2927
In v0.27.0, the list-routes command with -i flag and -o json output
was returning all nodes instead of just the specified node.
The issue was that JSON output was happening before the identifier
filtering logic. This change moves the JSON output to after both
the identifier filter and route existence filter are applied,
ensuring the correct filtered results are returned.
This restores the v0.26.1 behavior where:
headscale nodes list-routes -i 12 -o json
correctly returns only node 12's route information.
This PR investigates, adds tests and aims to correctly implement Tailscale's model for how Tags should be accepted, assigned and used to identify nodes in the Tailscale access and ownership model.
When evaluating in Headscale's policy, Tags are now only checked against a nodes "tags" list, which defines the source of truth for all tags for a given node. This simplifies the code for dealing with tags greatly, and should help us have less access bugs related to nodes belonging to tags or users.
A node can either be owned by a user, or a tag.
Next, to ensure the tags list on the node is correctly implemented, we first add tests for every registration scenario and combination of user, pre auth key and pre auth key with tags with the same registration expectation as observed by trying them all with the Tailscale control server. This should ensure that we implement the correct behaviour and that it does not change or break over time.
Lastly, the missing parts of the auth has been added, or changed in the cases where it was wrong. This has in large parts allowed us to delete and simplify a lot of code.
Now, tags can only be changed when a node authenticates or if set via the CLI/API. Tags can only be fully overwritten/replaced and any use of either auth or CLI will replace the current set if different.
A user owned device can be converted to a tagged device, but it cannot be changed back. A tagged device can never remove the last tag either, it has to have a minimum of one.
This PR changes tags to be something that exists on nodes in addition to users, to being its own thing. It is part of moving our tags support towards the correct tailscale compatible implementation.
There are probably rough edges in this PR, but the intention is to get it in, and then start fixing bugs from 0.28.0 milestone (long standing tags issue) to discover what works and what doesnt.
Updates #2417Closes#2619
This improves security and explicitly fails on startup when a user picks
the wrong directory to store its data.
- Run in read-only mode
- Make /var/run/headscale a read-write tmpfs
- Mount the config volume read-only
- Use the /health endpoint to check if Headscale is up
Previously we tested migrations on schemas and dumps
of old databases.
The problems with testing migrations against the schemas
is that the migration table is empty, so we try to run
migrations that are already ran on that schema, which might
blow up.
This commit removes the schema approach and just leaves all
the dumps, which include the migration table.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Document the API endpoint and the built-in swagger docs at /swagger. The
remote control docs are just a use case for gRPC - move it in the API
docs and update links to it.
The former is a no-op in the base images (45491f2c5c/scripts/debuerreotype-minimizing-config (L87-L109)), and `apt-get dist-clean` is a safer/better version of the `rm -rf /var/lib/apt/lists/*` that keeps the cryptographic bits that help prevent downgrade attacks.
Move favicon.png, style.css, and headscale.svg to hscontrol/assets/
and create a single assets.go file with all embed directives.
Update hscontrol/handlers.go and hscontrol/templates/general.go to
use the centralized assets package.
Add test to validate HTML template output consistency across all
templates (OIDC callback, registration, Windows, Apple).
Verifies all templates produce valid HTML5 with:
- Proper DOCTYPE declaration
- HTML5 lang attribute
- UTF-8 charset
- Viewport meta tag
- Semantic HTML structure
Ensures template refactoring maintains standards compliance.
Refactor template system to use go:embed for external assets and
CSS classes for styling instead of inline styles:
- general.go: Add go:embed directives for style.css and headscale.svg,
replace inline styles with CSS classes (H1, H2, H3, P, etc.),
add mdTypesetBody wrapper with Material for MkDocs styling
- apple.go, oidc_callback.go, register_web.go, windows.go:
Update to use new CSS-based helper functions (H1, H2, P, etc.)
and mdTypesetBody for consistent layout
This separates content from presentation, making templates easier
to maintain and update. All styling is now centralized in style.css
with Material for MkDocs design system.
Add design system assets for HTML templates:
- headscale.svg: Logo with optimized viewBox for proper alignment
- style.css: Material for MkDocs CSS variables and typography
- design.go: Design system constants for consistent styling
The logo viewBox is adjusted to 32.92 0 1247.08 640 to eliminate
whitespace from the original export and ensure left alignment with
text content.
Replace html/template with type-safe elem-go templating for OIDC
callback page. Improves consistency with other templates and provides
compile-time safety. All UI elements and styling preserved.
When tailscaled restarts, it sends RegisterRequest with Auth=nil and
Expiry=zero. Previously this was treated as a logout because
time.Time{}.Before(time.Now()) returns true.
Add early return in handleRegister() to detect this case and preserve
the existing node state without modification.
Fixes#2862
Changed UpdateUser and re-registration flows to use Updates() which only
writes modified fields, preventing unintended overwrites of unchanged fields.
Also updated UsePreAuthKey to use Model().Update() for single field updates
and removed unused NodeSave wrapper.
Fixes a regression introduced in v0.27.0 where node expiry times were
being reset to zero when tailscaled restarts and sends a MapRequest.
The issue was caused by using GORM's Save() method in persistNodeToDB(),
which overwrites ALL fields including zero values. When a MapRequest
updates a node (without including expiry information), Save() would
overwrite the database expiry field with a zero value.
Changed to use Updates() which only updates non-zero values, preserving
existing database values when struct pointer fields are nil.
In BackfillNodeIPs, we need to explicitly update IPv4/IPv6 fields even
when nil (to remove IPs), so we use Select() to specify those fields.
Added regression test that validates expiry is preserved after MapRequest.
Fixes#2862
Skip auth key validation for existing nodes re-registering with the same
NodeKey. Pre-auth keys are only required for initial authentication.
NodeKey rotation still requires a valid auth key as it is a security-sensitive
operation that changes the node's cryptographic identity.
Fixes#2830
When we encounter a source we cannot resolve, we skipped the whole rule,
even if some of the srcs could be resolved. In this case, if we had one user
that exists and one that does not.
In the regular policy, we log this, and still let a rule be created from what
does exist, while in the SSH policy we did not.
This commit fixes it so the behaviour is the same.
Fixes#2863
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
On Windows, if the user clicks the Tailscale icon in the system tray,
it opens a login URL in the browser.
When the login URL is opened, `state/nonce` cookies are set for that particular URL.
If the user clicks the icon again, a new login URL is opened in the browser,
and new cookies are set.
If the user proceeds with auth in the first tab,
the redirect results in a "state did not match" error.
This patch ensures that each opened login URL sets an individual cookie
that remains valid on the `/oidc/callback` page.
`TestOIDCMultipleOpenedLoginUrls` illustrates and tests this behavior.
When we fixed the issue of node visibility of nodes
that only had access to eachother because of a subnet
route, we gave all nodes access to all exit routes by
accident.
This commit splits exit nodes and subnet routes in the
access.
If a matcher indicates that the node should have access to
any part of the subnet routes, we do not remove it from the
node list.
If a matcher destination is equal to the internet, and the
target node is an exit node, we also do not remove the access.
Fixes#2784Fixes#2788
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
There are situations where the subnet routes and exit nodes
must be treated differently. This splits it so SubnetRoutes
only returns routes that are not exit routes.
It adds `IsExitRoutes` and `AllApprovedRoutes` for convenience.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Correctly identify Viper's ConfigFileNotFoundError in LoadConfig to log a warning and use defaults, unifying behavior with empty config files. Fixes fatal error when no config file is present for CLI commands relying on environment variables.
This PR addresses some consistency issues that was introduced or discovered with the nodestore.
nodestore:
Now returns the node that is being put or updated when it is finished. This closes a race condition where when we read it back, we do not necessarily get the node with the given change and it ensures we get all the other updates from that batch write.
auth:
Authentication paths have been unified and simplified. It removes a lot of bad branches and ensures we only do the minimal work.
A comprehensive auth test set has been created so we do not have to run integration tests to validate auth and it has allowed us to generate test cases for all the branches we currently know of.
integration:
added a lot more tooling and checks to validate that nodes reach the expected state when they come up and down. Standardised between the different auth models. A lot of this is to support or detect issues in the changes to nodestore (races) and auth (inconsistencies after login and reaching correct state)
This PR was assisted, particularly tests, by claude code.
- tailscale client gets a new AuthUrl and sets entry in the regcache
- regcache entry expires
- client doesn't know about that
- client always polls followup request а gets error
When user clicks "Login" in the app (after cache expiry), they visit
invalid URL and get "node not found in registration cache". Some clients
on Windows for e.g. can't get a new AuthUrl without restart the app.
To fix that we can issue a new reg id and return user a new valid
AuthUrl.
RegisterNode is refactored to be created with NewRegisterNode() to
autocreate channel and other stuff.
When the node notifier was replaced with batcher, we removed
its closing, but forgot to add the batchers so it was never
stopping node connections and waiting forever.
Fixes#2751
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
The UserInfo endpoint is always queried since 5d8a2c2.
This allows to use all OIDC related features without any extra
configuration on Authelia.
For Keycloak, its sufficient to add the groups mapper to the userinfo
endpoint.
the client will send a lot of fields as `nil` if they have
not changed. NetInfo, which is inside Hostinfo, is one of those
fields and we often would override the whole hostinfo meaning that
we would remove netinfo if it hadnt changed.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Initial work on a nodestore which stores all of the nodes
and their relations in memory with relationship for peers
precalculated.
It is a copy-on-write structure, replacing the "snapshot"
when a change to the structure occurs. It is optimised for reads,
and while batches are not fast, they are grouped together
to do less of the expensive peer calculation if there are many
changes rapidly.
Writes will block until commited, while reads are never
blocked.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Before this patch, we would send a message to each "node stream"
that there is an update that needs to be turned into a mapresponse
and sent to a node.
Producing the mapresponse is a "costly" afair which means that while
a node was producing one, it might start blocking and creating full
queues from the poller and all the way up to where updates where sent.
This could cause updates to time out and being dropped as a bad node
going away or spending too time processing would cause all the other
nodes to not get any updates.
In addition, it contributed to "uncontrolled parallel processing" by
potentially doing too many expensive operations at the same time:
Each node stream is essentially a channel, meaning that if you have 30
nodes, we will try to process 30 map requests at the same time. If you
have 8 cpu cores, that will saturate all the cores immediately and cause
a lot of wasted switching between the processing.
Now, all the maps are processed by workers in the mapper, and the number
of workers are controlable. These would now be recommended to be a bit
less than number of CPU cores, allowing us to process them as fast as we
can, and then send them to the poll.
When the poll recieved the map, it is only responsible for taking it and
sending it to the node.
This might not directly improve the performance of Headscale, but it will
likely make the performance a lot more consistent. And I would argue the
design is a lot easier to reason about.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
- Fix directory hierarchy flattening by using full paths instead of filepath.Base()
- Remove redundant container hostname prefixes from directory names
- Strip top-level directory from tar extraction to avoid nested structure
- Ensure parent directories exist before creating files
- Results in clean structure: control_logs/mapresponses/1-ts-client/file.json
Share/Contribute Headscale Zabbix Monitoring scripts and templates.
Thank you for the awesome application to everyone involved in Headscale's development!
Previously, nil regions were not properly handled. This change allows users to disable regions in DERPMaps.
Particularly useful to disable some official regions.
This patch includes some changes to the OIDC integration in particular:
- Make sure that userinfo claims are queried *before* comparing the
user with the configured allowed groups, email and email domain.
- Update user with group claim from the userinfo endpoint which is
required for allowed groups to work correctly. This is essentially a
continuation of #2545.
- Let userinfo claims take precedence over id token claims.
With these changes I have verified that Headscale works as expected
together with Authelia without the documented escape hatch [0], i.e.
everything works even if the id token only contain the iss and sub
claims.
[0]: https://www.authelia.com/integration/openid-connect/headscale/#configuration-escape-hatch
There was a bug in HA subnet router handover where we used stale node data
from the longpoll session that we handed to Connect. This meant that we got
some odd behaviour where routes would not be deactivated correctly.
This commit changes to the nodeview is used through out, and we load the
current node to be updated in the write path and then handle it all there
to be consistent.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit changes most of our (*)types.Node to
types.NodeView, which is a readonly version of the
underlying node ensuring that there is no mutations
happening in the read path.
Based on the migration, there didnt seem to be any, but the
idea here is to prevent it in the future and simplify other
new implementations.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
We are already being punished by github actions, there seem to be
little value in running all the tests for both databases, so only
run a few key tests to check postgres isnt broken.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Restructure and rewrite the OpenID Connect documentation. Start from the
most minimal configuration and describe what needs to be done both in
Headscale and the identity provider. Describe additional features such
as PKCE and authorization filters in a generic manner with examples.
Document how Headscale populates its user profile and how it relates to
OIDC claims. This is a revised version from the table in the changelog.
Document the validation rules for fields and extend known limitations.
Sort the provider specific section alphabetically and add a section for
Authelia, Authentik, Kanidm and Keycloak. Also simplify and rename Azure
to Entra ID.
Update the description for the oidc section in the example
configuration. Give a short explanation of each configuration setting.
All documentend features were tested with Headscale 0.26 (using a fresh
database each time) using the following identity providers:
* Authelia
* Authentik
* Kanidm
* Keycloak
Fixes: #2295
this commit moves all of the read and write logic, and all different parts
of headscale that manages some sort of persistent and in memory state into
a separate package.
The goal of this is to clearly define the boundry between parts of the app
which accesses and modifies data, and where it happens. Previously, different
state (routes, policy, db and so on) was used directly, and sometime passed to
functions as pointers.
Now all access has to go through state. In the initial implementation,
most of the same functions exists and have just been moved. In the future
centralising this will allow us to optimise bottle necks with the database
(in memory state) and make the different parts talking to eachother do so
in the same way across headscale components.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* cmd/hi: add integration test runner CLI tool
Add a new CLI tool 'hi' for running headscale integration tests
with Docker automation. The tool replaces manual Docker command
composition with an automated solution.
Features:
- Run integration tests in golang:1.24 containers
- Docker context detection (supports colima and other contexts)
- Test isolation with unique run IDs and isolated control_logs
- Automatic Docker image pulling and container management
- Comprehensive cleanup operations for containers, networks, images
- Docker volume caching for Go modules
- Verbose logging and detailed test artifact reporting
- Support for PostgreSQL/SQLite selection and various test flags
Usage: go run ./cmd/hi run TestPingAllByIP --verbose
The tool uses creachadair/command and flax for CLI parsing and
provides cleanup subcommands for Docker resource management.
Updates flake.nix vendorHash for new Go dependencies.
* ci: update integration tests to use hi CLI tool
Replace manual Docker command composition in GitHub Actions
workflow with the new hi CLI tool for running integration tests.
Changes:
- Replace complex docker run command with simple 'go run ./cmd/hi run'
- Remove manual environment variable setup (handled by hi tool)
- Update artifact paths for new timestamped log directory structure
- Simplify command from 15+ lines to 3 lines
- Maintain all existing functionality (postgres/sqlite, timeout, test patterns)
The hi tool automatically handles Docker context detection, container
management, volume mounting, and environment variable setup that was
previously done manually in the workflow.
* makefile: remove test integration
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* feat: add verify client config for embedded DERP
* refactor: embedded DERP no longer verify clients via HTTP
- register the `headscale://` protocol in `http.DefaultTransport` to intercept network requests
- update configuration to use a single boolean option `verify_clients`
* refactor: use `http.HandlerFunc` for type definition
* refactor: some renaming and restructuring
* chore: some renaming and fix lint
* test: fix TestDERPVerifyEndpoint
- `tailscale debug derp` use random node private key
* test: add verify clients integration test for embedded DERP server
* fix: apply code review suggestions
* chore: merge upstream changes
* fix: apply code review suggestions
---------
Co-authored-by: Kristoffer Dalby <kristoffer@dalby.cc>
* Improve map auth logic
* Bugfix
* Add comment, improve error message
* noise: make func, get by node
this commit splits the additional validation into a
separate function so it can be reused if we add more
endpoints in the future.
It swaps the check, so we still look up by NodeKey, but before
accepting the connection, we validate the known machinekey from
the db against the noise connection.
The reason for this is that when a node logs in or out, the node key
is replaced and it will no longer be possible to look it up, breaking
reauthentication.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* noise: add comment to remind future use of getAndVal
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* changelog: add entry
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
The systemd target "syslog.target" and not required because syslog is
socket activated.
The directory /var/run is usually a symlink to /run and its created by
systemd via the RuntimeDirectory=headscale option. System creates and
handles permissions, no need to manually mark it as a read-write path.
Move files for packaging outside the docs directory into its own
packaging directory. Replace the existing postinstall and postremove
scripts with Debian maintainerscripts to behave more like a typical
Debian package:
* Start and enable the headscale systemd service by default
* Does not print informational messages
* No longer stop and disable the service on updates
This package also performs migrations for all changes done in previous
package versions on upgrade:
* Set login shell to /usr/sbin/nologin
* Set home directory to /var/lib/headscale
* Migrate to system UID/GID
The package is lintian-clean with a few exceptions that are documented
as excludes and it passes puipars (both tested on Debian 12).
The following scenarious were tested on Ubuntu 22.04, Ubuntu 24.04,
Debian 11, Debian 12:
* Install
* Install same version again
* Install -> Remove -> Install
* Install -> Purge -> Install
* Purge
* Update from 0.22.0
* Update from 0.26.0
See: #2278
See: #2133Fixes: #2311
This change makes editing the generated command easier.
For example, after pasting into a terminal, the cursor position will be
near the username portion which requires editing.
* policy/v2: allow Username as ssh source
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy/v2: validate that no undefined group or tag is used
Fixes#2570
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy: fixup tests which violated tag constraing
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Link to stable and development docs in the README
* Add Tailscale SSH and autogroup:nonroot to features page
* Use @ when referencing users in policy
* Remove unmaintained headscale-webui
The project seems to be unmaintained (last commit: 2023-05-08) and it
only supports Headscale 0.22 or earlier.
* Use full image URL in container docs
This makes it easy to switch the container runtime from docker <->
podman.
* Remove version from docker-compose.yml example
This is now deprecated and yields a warning.
* Add documentation for routes
* Rename exit-node to routes and add redirects
* Add a new section on subnet routers
* Extend the existing exit-node documentation
* Describe auto approvers for subnet routers and exit nodes
* Provide ACL examples for subnet routers and exit nodes
* Describe HA and its current limitations
* Add a troubleshooting section with IP forwarding
* Update features page for 0.26
Add auto approvers and link to our documentation if available.
* Prefer the console lexer when commandline and output mixed
If we assume someone doesn't already have the required go package, they might also not have the required git package installed either, so pkg_add both of them.
* Make matchers part of the Policy interface
* Prevent race condition between rules and matchers
* Test also matchers in tests for Policy.Filter
* Compute `filterChanged` in v2 policy correctly
* Fix nil vs. empty list issue in v2 policy test
* policy/v2: always clear ssh map
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Aras Ergus <aras.ergus@tngtech.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
* integration: ensure route is set before node joins, reproduce
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* auth: ensure that routes are autoapproved when the node is stored
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Tailscale allows to override the local DNS settings of a node via
"Override local DNS" [1]. Restore this flag with the same config setting
name `dns.override_local_dns` but disable it by default to align it with
Tailscale's default behaviour.
Tested with Tailscale 1.80.2 and systemd-resolved on Debian 12.
With `dns.override_local_dns: false`:
```
Link 12 (tailscale0)
Current Scopes: DNS
Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 100.100.100.100
DNS Domain: tn.example.com ~0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa [snip]
```
With `dns.override_local_dns: true`:
```
Link 12 (tailscale0)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 100.100.100.100
DNS Domain: tn.example.com ~.
```
[1] https://tailscale.com/kb/1054/dns#override-local-dnsFixes: #2256
* ensure final dot on node name
This ensures that nodes which have a base domain set, will have a dot appended to their FQDN.
Resolves: https://github.com/juanfont/headscale/issues/2501
* improve OIDC TTL expire test
Waiting a bit more than the TTL of the OIDC token seems to remove some flakiness of this test. This furthermore makes use of a go func safe buffer which should avoid race conditions.
* Only read relevant nodes from database in PeerChangedResponse
* Rework to ensure transactional consistency in PeerChangedResponse again
* An empty nodeIDs list should return an empty nodes list
* Add test to ListNodesSubset
* Link PR in CHANGELOG.md
* combine ListNodes and ListNodesSubset into one function
* query for all nodes in ListNodes if no parameter is given
* also add optional filtering for relevant nodes to ListPeers
* fix issue auto approve route on register bug
This commit fixes an issue where routes where not approved
on a node during registration. This cause the auto approval
to require the node to readvertise the routes.
Fixes#2497Fixes#2485
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* hsic: only set db policy if exist
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy: calculate changed based on policy and filter
v1 is a bit simpler than v2, it does not pre calculate the auto approver map
and we cannot tell if it is changed.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* populate serving from primary routes
Depends on #2464Fixes#2480
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* also exit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix route update outside of connection
there was a bug where routes would not be updated if
they changed while a node was connected and it was not part of an
autoapprove.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update expected test output, cli only shows service node
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Some endpoints in /debug send JSON data as string. Set the Content-Type
header to "application/json" which renders nicely in Firefox.
Mention the /debug route in the example configuration.
* utility iterator for ipset
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* split policy -> policy and v1
This commit split out the common policy logic and policy implementation
into separate packages.
policy contains functions that are independent of the policy implementation,
this typically means logic that works on tailcfg types and generic formats.
In addition, it defines the PolicyManager interface which the v1 implements.
v1 is a subpackage which implements the PolicyManager using the "original"
policy implementation.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use polivyv1 definitions in integration tests
These can be marshalled back into JSON, which the
new format might not be able to.
Also, just dont change it all to JSON strings for now.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* formatter: breaks lines
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove compareprefix, use tsaddr version
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove getacl test, add back autoapprover
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use policy manager tag handling
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* rename display helper for user
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* introduce policy v2 package
policy v2 is built from the ground up to be stricter
and follow the same pattern for all types of resolvers.
TODO introduce
aliass
resolver
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* wire up policyv2 in integration testing
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* split policy v2 tests into seperate workflow to work around github limit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add policy manager output to /debug
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* handle register auth errors
This commit handles register auth errors as the
Tailscale clients expect. It returns the error as
part of a tailcfg.RegisterResponse and not as a
http error.
In addition it fixes a nil pointer panic triggered
by not handling the errors as part of this chain.
Fixes#2434
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This PR switches the homegrown debug endpoint to using tsweb.Debugger, a neat toolkit with batteries included for pprof and friends, and making it easy to add additional debug info:
I've started out by adding a bunch of "introspect" endpoints
image
So users can see the acl, filter, config, derpmap and connected nodes as headscale sees them.
This helps preventing messages being sent with the wrong update type
and payload combination, and it is shorter/neater.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Separate the term "tailnet" from user and be more explicit about
providing a single tailnet.
Also be more explicit about users. Refer to "headscale users" when
mentioning commandline invocations and use the term "local users" when
discussing unix accounts.
Fixes: #2335
* add dedicated http error to propagate to user
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* classify user errors in http handlers
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* move validation of pre auth key out of db
This move separates the logic a bit and allow us to
write specific errors for the caller, in this case the web
layer so we can present the user with the correct error
codes without bleeding web stuff into a generic validate.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* set state and nounce in oidc to prevent csrf
Fixes#2276
* try to fix new postgres issue
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure valid tags is populated on user gets too
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure forced tags are added
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove unused envvar in test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* debug log auth/unauth tags in policy man
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* defer shutdown in tags test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add tag test with groups
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add email, display name, picture to create user
Updates #2166
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add ability to set display and email to cli
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add email to test users in integration
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix issue where tags were only assigned to email, not username
Fixes#2300Fixes#2307
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* expand principles to correct login name
and if fix an issue where nodeip principles might not expand to all
relevant IPs instead of taking the first in a prefix.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix ssh unit test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update cli and oauth tests for users with email
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* index by test email
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix last test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Describe both ways to add extra DNS records
* Use "extra" instead of "custom" to align with the configuration file
* Include dns.extra_records_path in the configuration file
* Add -race flag to Makefile and integration tests; fix data race in CreateTailscaleNodesInUser
* Fix data race in ExecuteCommand by using local buffers and mutex
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
* lint
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
---------
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
* Fix excess error message during writes
Fixes#2290
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* retry filewatcher on removed files
This should handled if files are deleted and added again, and for rename
scenarios.
Fixes#2289
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* test more write and remove in filewatcher
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Upgrade the use of dns.use_username_in_magic_dns or
dns_config.use_username_in_magic_dns to a fatal error and remove the
option from the example configuration and integration tests.
Fixes: #2219
Docker releases a patch release which changed the required permissions to be able to do tun devices in containers, this caused all containers to fail in tests causing us to fail all tests. This fixes it, and adds some tools for debugging in the future.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Setup mike to provide versioned builds of the documentation.
The goal is to have versioned docs for stable releases (0.23.0, 0.24.0)
and development docs that can progress along with the code. This allows
us to tailor docs to the next upcoming version as we no longer need to
care about diversion between rendered docs and the latest release.
Versions:
* development (alias: unstable) on each push to the main branch
* MAJOR.MINOR.PATCH (alias: stable, latest for the newest version)
* for each "final" release tag
* for each push to doc maintenance branches: doc/MAJOR.MINOR.PATCH
The default version should the current stable version. The doc
maintenance branches may be used to update the version specific
documentation when issues arise after a release.
This commit fixes the constraint syntax so it is both valid for
sqlite and postgres.
To validate this, I've added a new postgres testing library and a
helper that will spin up local postgres, setup a db and use it in
the constraints tests. This should also help testing db stuff in
the future.
postgres has been added to the nix dev shell and is now required
for running the unit tests.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit hardens the migration part of the OIDC from
the old username based approach to the new sub based approach
and makes it possible for the operator to opt out entirely.
Fixes#1990
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Use a trailing slash
recommended by mkdocs-material
* Update doc requirements
Let mkdocs-material resolve its imaging dependencies (cairosvg and
pillow) and fix a dependabot warning along the way.
Reference compatible versions by major.minor.
* config: loosen up BaseDomain and ServerURL checks
Requirements [here][1]:
> OK:
> server_url: headscale.com, base: clients.headscale.com
> server_url: headscale.com, base: headscale.net
>
> Not OK:
> server_url: server.headscale.com, base: headscale.com
>
> Essentially we have to prevent the possibility where the headscale
> server has a URL which can also be assigned to a node.
>
> So for the Not OK scenario:
>
> if the server is: server.headscale.com, and a node joins with the name
> server, it will be assigned server.headscale.com and that will break
> the connection for nodes which will now try to connect to that node
> instead of the headscale server.
Fixes#2210
[1]: https://github.com/juanfont/headscale/issues/2210#issuecomment-2488165187
* server_url and base_domain: re-word error message, fix a one-off bug and add a test case for the bug.
* lint
* lint again
* integration testing: add and validate build-time options for tailscale head
* fixup! integration testing: add and validate build-time options for tailscale head
integration testing: comply with linter
* fixup! fixup! integration testing: add and validate build-time options for tailscale head
integration testing: tsic.New must never return nil
* fixup! fixup! fixup! integration testing: add and validate build-time options for tailscale head
* minor fixes
* Document to either use a minimal configuration file or environment
variables to connect with a remote headscale instance.
* Document a workaround specific for headscale 0.23.0.
* Remove reference to ancient headscale version.
* Use `cli.insecure: true` or `HEADSCALE_CLI_INSECURE=1` to skip
certificate verification.
* Style and typo fixes
Ref: #2193
* Remove status from web-ui docs
Rename the title to indicate that there multiple web interfaces
available. Do not track the status of each web interface here as their
status is subject to change over time.
* Add page for third-party tools and scripts
According to 15fc6cd966
the routes `/derp/probe` and `/derp/latency-check` are the same and
different versions of the tailscale client use one or the other
endpoint.
Also handle /derp/latency-check
Fixes: #2211
* #2140 Fixed updating of hostname and givenName when it is updated in HostInfo
* #2140 Added integration tests
* #2140 Fix unit tests
* Changed IsAutomaticNameMode to GivenNameHasBeenChanged. Fixed errors in files according to golangci-lint rules
* Setup mkdocs-redirects
* Restructure existing documentation
* Move client OS support into the documentation
* Move existing Client OS support table into its own documentation page
* Link from README.md to the rendered documentation
* Document minimum Tailscale client version
* Reuse CONTRIBUTING.md" in the documentation
* Include "CONTRIBUTING.md" from the repository root
* Update FAQ and index page and link to the contributing docs
* Add configuration reference
* Add a getting started page and explain the first steps with headscale
* Use the existing "Using headscale" sections and combine them into a
single getting started guide with a little bit more explanation.
* Explain how to get help from the command line client.
* Remove duplicated sections from existing installation guides
* Document requirements and assumptions
* Document packages provided by the community
* Move deb install guide to official releases
* Move manual install guide to official releases
* Move container documentation to setup section
* Move sealos documentation to cloud install page
* Move OpenBSD docs to build from source
* Simplify DNS documentation
* Add sponsor page
* Add releases page
* Add features page
* Add help page
* Add upgrading page
* Adjust mkdocs nav
* Update wording
Use the term headscale for the project, Headscale on the beginning of a
sentence and `headscale` when refering to the CLI.
* Welcome to headscale
* Link to existing documentation in the FAQ
* Remove the goal header and use the text as opener
* Indent code block in OIDC
* Make a few pages linter compatible
Also update ignored files for prettier
* Recommend HTTPS on port 443
Fixes: #2164
* Use hosts in acl documentation
thx @efficacy38 for noticing this
Ref: #1863
* Use mkdocs-macros to set headscale version once
* Changed all the HTML into go using go-elem
Created templates package in ./hscontrol/templates.
Moved the registerWebAPITemplate into the templates package as a function to be called.
Replaced the apple and windows html files with go-elem.
* update flake
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make reauth test compat with tailscale head
tailscale/tailscale@1eaad7d broke our reauth test as it makes the client
retry with https/443 if it reconnects within 2 minutes.
This commit fixes this by running the test as a two part,
- with https, to confirm instant reconnect works
- with http, and a 3 min wait, to check that it work without.
The change is not a general consern as headscale in prod is ran
with https.
Updates #2164
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* sort test for stable order
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
expand user, add claims to user
This commit expands the user table with additional fields that
can be retrieved from OIDC providers (and other places) and
uses this data in various tailscale response objects if it is
available.
This is the beginning of implementing
https://docs.google.com/document/d/1X85PMxIaVWDF6T_UPji3OeeUqVBcGj_uHRM5CI-AwlY/edit
trying to make OIDC more coherant and maintainable in addition
to giving the user a better experience and integration with a
provider.
remove usernames in magic dns, normalisation of emails
this commit removes the option to have usernames as part of MagicDNS
domains and headscale will now align with Tailscale, where there is a
root domain, and the machine name.
In addition, the various normalisation functions for dns names has been
made lighter not caring about username and special character that wont
occur.
Email are no longer normalised as part of the policy processing.
untagle oidc and regcache, use typed cache
This commits stops reusing the registration cache for oidc
purposes and switches the cache to be types and not use any
allowing the removal of a bunch of casting.
try to make reauth/register branches clearer in oidc
Currently there was a function that did a bunch of stuff,
finding the machine key, trying to find the node, reauthing
the node, returning some status, and it was called validate
which was very confusing.
This commit tries to split this into what to do if the node
exists, if it needs to register etc.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Update headscale user creation settings in .deb
Update the headscale user settings to:
- shell = /usr/sbin/nologin
- home-dir = /var/lib/headscale
This syncs the .deb installation behavior with the current Linux docs:
fe68f50328/docs/running-headscale-linux-manual.md (L39-L45)Fixesjuanfont/headscale#2133
* slight refactor to use existing variables.
* Fixup for HOME_DIR var
this commit denormalises the Tags related to a Pre auth key
back onto the preauthkey table and struct as a string list.
There was not really any real normalisation here as we just added
a bunch of duplicate tags with new IDs and preauthkeyIDs, lots of
GORM cermony but no actual advantage.
This work is the start to fixup tags which currently are not working
as they should.
Updates #1369
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Some commands such as `nodes delete` require user interaction and they
fail if `-it` is no supplied to `docker exec`. Use `docker exec -it` in
documentation examples to also make them work in interactive commands.
* handle control protocol through websocket
The necessary behaviour is already in place,
but the wasm build only issued GETs, and the handler was not invoked.
* get DERP-over-websocket working for wasm clients
* Prepare for testing builtin websocket-over-DERP
Still needs some way to assert that clients are connected through websockets,
rather than the TCP hijacking version of DERP.
* integration tests: properly differentiate between DERP transports
* do not touch unrelated code
* linter fixes
* integration testing: unexport common implementation of derp server scenario
* fixup! integration testing: unexport common implementation of derp server scenario
* dockertestutil/logs: remove unhelpful comment
* update changelog
---------
Co-authored-by: Csaba Sarkadi <sarkadicsa@tutanota.de>
* Rename docs/ios-client.md to docs/apple-client.md. Add instructions
for macOS; those are copied from the /apple endpoint and slightly
modified. Fix doc links in the README.
* Move infoboxes for /apple and /windows under the "Goal" section to the
top. Those should be seen by users first as they contain *their*
specific headscale URL.
* Swap order of macOS and iOS to move "Profiles" further down.
* Remove apple configuration profiles
* Remove Tailscale versions hints
* Mention /apple and /windows in the README along with their docs
See: #2096
* move logic for validating node names
this commits moves the generation of "given names" of nodes
into the registration function, and adds validation of renames
to RenameNode using the same logic.
Fixes#2121
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix double arg
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add shutdown that asserts if headscale had panics
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add test case producing 2118 panic
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make stream shutdown if self-node has been removed
Currently we will read the node from database, and since it is
deleted, the id might be set to nil. Keep the node around and
just shutdown, so it is cleanly removed from notifier.
Fixes#2118
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Simplify /windows to the bare minimum. Also remove the
/windows/tailscale.reg endpoint as its generated file is no longer
valid for current Tailscale versions.
* Update and simplify the windows documentation accordingly.
* Add a "Unattended mode" section to the troubleshooting section
explaining how to enable "Unattended mode" in the via the Tailscale
tray icon.
* Add infobox about /windows to the docs
Tested on Windows 10, 22H2 with Tailscale 1.72.0
Replaces: #1995
See: #2096
* replace deprecated golangci-lint output format
CI was producing this kind of messages:
> [config_reader] The output format `github-actions` is deprecated, please use `colored-line-number`
* Actually lint files on CI
* Add support for service reload and sync service file
* Copy the systemd.service file to the manual linux docs and adjust the
path to the headscale binary to match with the previous documentation
blocks. Unfortunately, there seems to be no easy way to include a
file in mkdocs.
* Remove a redundant "deprecation" block. The beginning of the
documentation already states that.
* Add `ExecReload` to the systemd.service file.
Fixes: #2016
* Its called systemd
* Fix link to systemd homepage
* docs/acl: fix path to policy file
* docs/exit-node: fixup for 0.23
* Add newlines between commands to improve readability
* Use nodes instead on name
* Remove query parameter from link to Tailscale docs
* docs/remote-cli: fix formatting
* Indent blocks below line numbers to restore numbering
* Fix minor typos
* docs/reverse-proxy: remove version information
* Websocket support is always required now
* s/see detail/see details
* docs/exit-node: add warning to manual documentation
* Replace the warning section with a warning admonition
* Fix TODO link back to the regular linux documentation
* docs/openbsd: fix typos
* the database is created on-the-fly
* docs/sealos: fix typos
* docs/container: various fixes
* Remove a stray sentence
* Remove "headscale" before serve
* Indent line continuation
* Replace hardcoded 0.22 with <VERSION>
* Fix path in debug image to /ko-app/headscale
Fixes: #1822
aa
* Fix KeyExpiration when a zero time value has a timezone
When a zero time value is loaded from JSON or a DB in a way that
assigns it the local timezone, it does not roudtrip in JSON as a
value for which IsZero returns true. This causes KeyExpiry to be
treated as a far past value instead of a nilish value.
See https://github.com/golang/go/issues/57040
* Fix whitespace
* Ensure that postgresql is used for all tests when env var is set
* Pass through value of HEADSCALE_INTEGRATION_POSTGRES env var
* Add option to set timezone on headscale container
* Add test for registration with auth key in alternate timezone
* validate policy against nodes, error if not valid
this commit aims to improve the feedback of "runtime" policy
errors which would only manifest when the rules are compiled to
filter rules with nodes.
this change will in;
file-based mode load the nodes from the db and try to compile the rules on
start up and return an error if they would not work as intended.
database-based mode prevent a new ACL being written to the database if
it does not compile with the current set of node.
Fixes#2073Fixes#2044
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure stderr can be used in err checks
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* test policy set validation
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add new integration test to ghaction
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add back defer for cli tst
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Requiring someone to write a design doc/contribute to the feature shouldn't be a requirement for raising a feature request as users may lack the skills required to do this.
Code Rabbit is one of these new fancy LLM code review tools. I am skeptical
but we can try it for free and it might provide us with some value to let
people get feedback while waiting for other people.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
this commit changes and streamlines the dns_config into a new
key, dns. It removes a combination of outdates and incompatible
configuration options that made it easy to confuse what headscale
could and could not do, or what to expect from ones configuration.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Fix data race issues in EphemeralGarbageCollector tests
* Add defer for mutex unlock in TestEphemeralGarbageCollectorOrder
* Fix mutex unlock order in closure by updating defer placement
* reformat code
This is mostly an automated change with `make lint`.
I had to manually please golangci-lint in routes_test because of a short
variable name.
* fix start -> strategy which was wrongly corrected by linter
* replace ephemeral deletion logic
this commit replaces the way we remove ephemeral nodes,
currently they are deleted in a loop and we look at last seen
time. This time is now only set when a node disconnects and
there was a bug (#2006) where nodes that had never disconnected
was deleted since they did not have a last seen.
The new logic will start an expiry timer when the node disconnects
and delete the node from the database when the timer is up.
If the node reconnects within the expiry, the timer is cancelled.
Fixes#2006
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use uint64 as authekyid and ptr helper in tests
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add test db helper
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add list ephemeral node func
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* schedule ephemeral nodes for removal on startup
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix gorm query for postgres
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add godoc
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Added a new function `RegisterMethodToV1Enum()` to Node, converting the internal register method string to the corresponding V1 Enum value. Included corresponding unit test in `node_test.go` to ensure correct conversion for various register methods.
* correctly enable WAL log for sqlite
this commit makes headscale correctly enable write-ahead-log for
sqlite and adds an option to turn it on and off.
WAL is enabled by default and should make sqlite perform a lot better,
even further eliminating the need to use postgres.
It also adds a couple of other useful defaults.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
most of the time we dont even check this error and checking
the string for particular errors is very flake as different
databases (sqlite and psql) use different error messages, and
some users might have it in other languages.
Fixes#1956
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This PR removes the complicated session management introduced in https://github.com/juanfont/headscale/pull/1791 which kept track of the sessions in a map, in addition to the channel already kept track of in the notifier.
Instead of trying to close the mapsession, it will now be replaced by the new one and closed after so all new updates goes to the right place.
The map session serve function is also split into a streaming and a non-streaming version for better readability.
RemoveNode in the notifier will not remove a node if the channel is not matching the one that has been passed (e.g. it has been replaced with a new one).
A new tuning parameter has been added to added to set timeout before the notifier gives up to send an update to a node.
Add a keep alive resetter so we wait with sending keep alives if a node has just received an update.
In addition it adds a bunch of env debug flags that can be set:
- `HEADSCALE_DEBUG_HIGH_CARDINALITY_METRICS`: make certain metrics include per node.id, not recommended to use in prod.
- `HEADSCALE_DEBUG_PROFILING_ENABLED`: activate tracing
- `HEADSCALE_DEBUG_PROFILING_PATH`: where to store traces
- `HEADSCALE_DEBUG_DUMP_CONFIG`: calls `spew.Dump` on the config object startup
- `HEADSCALE_DEBUG_DEADLOCK`: enable go-deadlock to dump goroutines if it looks like a deadlock has occured, enabled in integration tests.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
this directory is unmaintained and not verified, if it should be restored, it should end up
under the community docs effort.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Add test stage to docs
Add new file with docs tets
Run only in pulls
* set explicit python version
* Revert "set explicit python version"
This reverts commit 4dd7b81f26.
* docs/requirements: update mkdocs-material
---------
Co-authored-by: ohdearaugustin <ohdearaugustin@users.noreply.github.com>
jagottsicher's fork fixed a bug in Windows implementation. While Windows may be not intended as a target platform,
some contributors may prefer it for development.
Also ran go mod tidy, thus two more unnecessary packages are removed from go.sum
This commit restructures the map session in to a struct
holding the state of what is needed during its lifetime.
For streaming sessions, the event loop is structured a
bit differently not hammering the clients with updates
but rather batching them over a short, configurable time
which should significantly improve cpu usage, and potentially
flakyness.
The use of Patch updates has been dialed back a little as
it does not look like its a 100% ready for prime time. Nodes
are now updated with full changes, except for a few things
like online status.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Fixes the issue reported in #1712. In Tailscale SaaS, ephemeral keys can be single-user or reusable. Until now, our ephemerals were only reusable. This PR makes us adhere to the .com behaviour.
A lot of things are breaking in 0.23 so instead of having this
be a long process, just rip of the plaster.
Updates #1758
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* rework docker tags
This commit tries to align the new docker tags with the old schema
A prerelease will end up with the following tags:
- unstable
- v0.23.0-alpha3
- 0.23.0.alpha3
- sha-1234adsfg
A release will end up with:
- latest
- stable
- v0.23.0
- v0.23
- v0
- 0.23.0
- 0.23
- 0
- sha-1234adsfg
All of the builds will also have a `-debug` version.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* TLS documentation updates
Move "Bring your own certificates" to the top
since the letsencrypt section is now much longer, it seems wrong to
keep such a short section way down at the bottom.
Restructure "Challenge types" into separate sections
Add technical description of letsencrypt renewals
this aims to answer:
- what can be expected in terms of renewals
- what logs can be expected (none)
- how to validate that renewal happened successfully
- the reason for some of the 'acme/autocert' logs, or at least
some best-effort assumptions
* +prettier
* Add test because of issue 1604
* Add peer for routes
* Revert previous change to try different way to add peer
* Add traces
* Remove traces
* Make sure tests have IPPrefix comparator
* Get allowedIps before loop
* Remove comment
* Add composite literals :)
We currently do not have a way to clean up api keys. There may be cases
where users of headscale may generate a lot of api keys and these may
end up accumulating in the database. This commit adds the command to
delete an api key given a prefix.
When Postgres is used as the backing database for headscale,
it does not set a limit on maximum open and idle connections
which leads to hundreds of open connections to the Postgres
server.
This commit introduces the configuration variables to set those
values and also sets default while opening a new postgres connection.
This commits removes the locks used to guard data integrity for the
database and replaces them with Transactions, turns out that SQL had
a way to deal with this all along.
This reduces the complexity we had with multiple locks that might stack
or recurse (database, nofitifer, mapper). All notifications and state
updates are now triggered _after_ a database change.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix#1706 - failover should disregard disabled routes during failover
* fixe tests for failover; all current tests assume routes to be enabled
* add testcase for #1706 - failover to disabled route
* upgrade tailscale
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make Node object use actualy tailscale key types
This commit changes the Node struct to have both a field for strings
to store the keys in the database and a dedicated Key for each type
of key.
The keys are populated and stored with Gorm hooks to ensure the data
is stored in the db.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use key types throughout the code
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make sure machinekey is concistently used
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use machine key in auth url
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix web register
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use key type in notifier
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix relogin with webauth
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit rearranges the poll handler to immediatly accept
updates and notify its peers and return, not travel down the
function for a bit. This reduces the DB calls and other
holdups that isnt necessary to send a "lite response", a
map response without peers, or accepting an endpoint update.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This field is no longer used, it was used in our old state
"algorithm" to determine if we should send an update.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit changes the internals of the mapper to
track all the changes to peers over its lifetime.
This means that it no longer depends on the database
and this should hopefully help with locks and timing issues.
When the mapper is created, it needs the current list of peers,
the world view, when the polling session was started. Then as
update changes are called, it tracks the changes and generates
responses based on its internal list.
As a side, the types.Machines and types.MachinesP, as well as
types.Machine being passed as a full struct and pointer has been
changed to always be pointers, everywhere.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Previously we did not update the packet filter
when nodes changed, which would cause new nodes
to be missing from packet filters of old nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commits extends the mapper with functions for creating "delta"
MapResponses for different purposes (peer changed, peer removed, derp).
This wires up the new state management with a new StateUpdate struct
letting the poll worker know what kind of update to send to the
connected nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit replaces the timestamp based state system with a new
one that has update channels directly to the connected nodes. It
will send an update to all listening clients via the polling
mechanism.
It introduces a new package notifier, which has a concurrency safe
manager for all our channels to the connected nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
There was a lot of tests that actually threw a lot of errors and that did
not pass all the way because we didnt check everything. This commit should
fix all of these cases.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Also bumps tailscale version to trigger build and fixes a CLI test
that had the wrong capitalisation
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit makes a wrapper function round the normalisation requiring
"stripEmailDomain" which has to be passed in almost all functions of
headscale by loading it from Viper instead.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit allows SSH rules to be assigned to each relevant not and
by doing that allow SSH to be rejected, completing the initial SSH
support.
This commit enables SSH by default and removes the experimental flag.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit renames a bunch of files to try to make it a bit less confusing;
protocol_ is now auth as they contained registration, auth and login/out flow
protocol_.*_poll is now poll.go
api.go and other generic handlers are now handlers.go
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Prior to the code reorg, we would generate rules from the Policy and
store it on the global object. Now we generate it on the fly for each node
and this commit cleans up the old variables to make sure we have no
unexpected side effects.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
The mapper package contains functions related to creating and marshalling
reponses to machines.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This is a massive commit that restructures the code into modules:
db/
All functions related to modifying the Database
types/
All type definitions and methods that can be exclusivly used on
these types without dependencies
policy/
All Policy related code, now without dependencies on the Database.
policy/matcher/
Dedicated code to match machines in a list of FilterRules
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This is step one in detaching the Database layer from Headscale (h). The
ultimate goal is to have all function that does database operations in
its own package, and keep the business logic and writing separate.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
distroless has proven a mantenance burden for us, and it has caused headaches for user when trying to debug issues in the container.
And in 2023, 20MB of extra disk space are neglectible.
This commit simplifies the goreleaser configuration and then adds nfpm
support which allows us to build .deb and .rpm for each of the ARCH we
support.
The deb and rpm packages adds systemd services and users, creates
directories etc and should in general give the user a working
environment. We should be able to remove a lot of the complicated,
PEBCAK inducing documentation after this.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commits adds a test to verify that nodes get updated if a node in
their network expires.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit adds a default OpenID Connect expiry to 180d to align with
Tailscale SaaS (previously infinite or based on token expiry).
In addition, it adds an option use the expiry time from the Token sent
by the OpenID provider. This will typically cause really short expiry
and you should only turn on this option if you know what you are
desiring.
This fixes#1176.
Co-authored-by: Even Holthe <even.holthe@bekk.no>
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Align behaviour of dns_config.restricted_nameservers to tailscale.
Tailscale allows split DNS configuration without requiring global nameservers.
In addition, as per [the docs](https://tailscale.com/kb/1054/dns/#using-dns-settings-in-the-admin-console):
> These nameservers also configure search domains for your devices
This commit aligns headscale to tailscale by:
* honouring dns_config.restricted_nameservers regardless of whether any global resolvers are configured
* adding a search domain for each restricted_nameserver
- Save logs from control(headscale) on every run to tmp
- Upgrade nix-actions
- Cancel builds if new commit is pushed
- Fix a sorting bug in user command test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
As indicated by bradfitz in https://github.com/juanfont/headscale/issues/804#issuecomment-1399314002,
both routes for the exit node must be enabled at the same time. If a user tries to enable one of the exit node routes,
the other gets activated too.
This commit also reduces the API surface, making private a method that didnt need to be exposed.
The calls to AutoMigrate to other classes that refer to users will
create the table and it will break, it needs to be done before
everything else.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
While this truly breaks the point of the backwards compatible stuff with
protobuf, it does not seem worth it to attempt to glue together a
compatible API.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Currently the most "secret" way to specify the oidc client secret is via
an environment variable `OIDC_CLIENT_SECRET`, which is problematic[1].
Lets allow reading oidc client secret from a file. For extra convenience
the path to the secret will resolve the environment variables.
[1]: https://systemd.io/CREDENTIALS/
This commit sets the Headscale config from env instead of file for
integration tests, the main point is to make sure that when we add per
test config, it properly replaces the config key and not append it or
something similar.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
In TS2021 the MachineKey can be obtained from noiseConn.Peer() - contrary to what I thought before,
where I assumed MachineKey was dropped in TS2021.
By having a ts2021App and hanging from there the TS2021 handlers, we can fetch again the MachineKey.
When using Tailscale v1.34.1, enabling or disabling a route does not
effectively add or remove the route from the node's routing table.
We must restart tailscale on the node to have a netmap update.
Fix this by refreshing last state change so that a netmap diff is sent.
Also do not include secondary routes in allowedIPs, otherwise secondary
routes might be used by nodes instead of the primary route.
Signed-off-by: Fatih Acar <facar@scaleway.com>
Port routes tests to new model
Mark as primary the first instance of subnet + tests
In preparation for subnet failover, mark the initial occurrence of a subnet as the primary one.
This commit makes the initial SSH test a bit simpler:
- Use the same pattern/functions for all clients as other tests
- Only test within _one_ namespace/user to confirm the base case
- Use retry function, same as taildrop, there is some funky going on
there...
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Advertises the SSH capability, and parses the SSH ACLs to pass to the
tailscale client. Doesn’t support ‘autogroup’ ACL functionality.
Co-authored-by: Daniel Brooks <db48x@headline.com>
* Port OIDC integration tests to v2
* Move Tailscale old versions to TS2019 list
* Remove Alpine Linux container
* Updated changelog
* Releases: use flavor to set the tag suffix
* Added more debug messages in OIDC registration
* Added more logging
* Do not strip nodekey prefix on handle expired
* Updated changelog
* Add WithHostnameAsServerURL option func
* Reduce the number of namespaces and use hsic.WithHostnameAsServerURL
* Linting fix
* Fix linting issues
* Wait for ready outside the up goroutine
* Minor change in log message
* Add prefix to env var
* Remove unused env var
Co-authored-by: Juan Font <juan.font@esa.int>
Co-authored-by: Steven Honson <steven@honson.id.au>
Co-authored-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit injects the per-test-generated tls certs into the tailscale
container and makes sure all can ping all. It does not test any of the
DERP isolation yet.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit injects the per-test-generated tls certs into the tailscale
container and makes sure all can ping all. It does not test any of the
DERP isolation yet.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
When using running `tailscale up` in the AuthKey flow process, the tailscale client immediately enters PollMap after registration - avoiding a race condition.
When using the web auth (up -> go to the Control website -> CLI `register`) the client is polling checking if it has been authorized. If we immediately ask for the client IP, as done in CreateHeadscaleEnv() we might have the client in NotReady status.
This method provides a way to wait for the client to be ready.
Signed-off-by: Juan Font Alonso <juanfontalonso@gmail.com>
The retry has no real function as it will just fail on
"container exists" on the old tests and the new test will
just try forever before it eventually fails.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
As indicated by the comment, the default /var/lib/headscale path is not writable in the container. However the sample setting is not following that like `private_key_path`
0.16.0 introduced random suffixes to all machine given names
(DNS hostnames) regardless of collisions within a namespace.
This commit brings Headscale more inline with Tailscale by only
adding a suffix if the hostname will collide within the namespace.
The suffix generation differs from Tailscale.
See https://tailscale.com/kb/1098/machine-names/
Add a configuration flag (default true to preserve current behaviour) to
allow headscale to start without OIDC being able to initialise.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit makes headscale fall back to CLI authentication if oidc
fails to initialised and posts a warning to users.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
OIDC might be configured, but unable to be initialised, this only runs
the oidc cycle if it is actually successfully set up/initialised.
Prep for next commit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit addresses a potential issue where we allowed unsanitised
content to be passed through a go template without validation.
We now try to unmarshall the incoming node key and fails to render the
template if it is not a valid node key.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
It appears to be causing confusion for users on Discord when copying/pasting from the example here, if Headscale crashes on launch then the container will be removed and logs can't be viewed with `docker logs`.
It appears to be causing confusion for users on Discord when copying/pasting from the example here, if Headscale crashes on launch then the container will be removed and logs can't be viewed with `docker logs`.
Currently we exit the program if the setup does not work, this can cause
is to leave containers and other resources behind since we dont run
TearDown. This change will just fail the test if we cant set up, which
should mean that the TearDown runs aswell.
We currently have a bit of flaky logic which prevents the docker plugin
from cleaning up the containers if the tests or setup fatals or crashes,
this is due to a limitation in the save / passed stats handling.
This change makes it an environment variable which by default ditches
the logs and makes the containers clean up "correctly" in the teardown
method.
This commit makes the setLastStateChangeToNow function take a list of
namespaces instead of a single namespace. If no namespaces is passed,
all namespaces will be updated. This means that the argument acts like a
filter.
Implements #617.
Tailscale has changed the format of their ACLs to use a more firewall-y terms ("users" & "ports" -> "src" & "dst"). They have also started using all-lowercase tags. This PR applies these changes.
Extract LoadConfig from GetHeadscaleConfig, as they are conceptually
different operation, e.g.,
1) you can reload config through LoadConfig and do not get config
2) you can get config without reload config
This commit starts to wire up better signal handling, it starts with
handling shutdown a bit better, using the graceful shutdown for all the
listeners we use.
It also adds the initial switch case for handling config and acl reload,
which is to be implemented.
This commit makes isOutdated validate a nodes necessity to update
against all namespaces, and not just the nodes own namespace (which made
more sense before).
getLastStateChange is now uses the passed namespaces as a filter,
meaning that not requesting any namespace will give you the total last
updated state.
In addition, the sync.Map is exchanged for a variant that uses generics
which allows us to remove some casting logic.
When running in CI, I obtained the following error:
```
Running [/home/runner/golangci-lint-1.41.0-linux-amd64/golangci-lint run --out-format=github-actions --new-from-patch=/tmp/tmp-1795-28vaWZek2jfM/pull.patch --new=false --new-from-rev=] in [] ...
level=error msg="Running error: unknown linters: 'ireturn,maintidx', run 'golangci-lint linters' to see the list of supported linters"
```
Adds knobs to configure three aspects of the OpenID Connect flow:
* Custom scopes to override the default "openid profile email".
* Custom parameters to be added to the Authorize Endpoint request.
* Domain allowlisting for authenticated principals.
* User allowlisting for authenticated principals.
This commit fixes the issue of headscale crashing after sending on a
closed channel by moving the channel close to the sender side, instead
of the creator. closeChanWithLog is also implemented with generics now.
Fixes: https://github.com/juanfont/headscale/issues/342
Signed-off-by: Moritz Poldrack <git@moritz.sh>
- makes the html/template for /register follow the same formatting
as /apple and /windows
- adds a <title> element
- minor change for consistency's sake
This commit adds dockerbuild to flakes.nix:
```
nix build .#headscale-docker
```
This uses the Nix infra to build and _does not_ use Dockerfile.
It currently works on Linux (no macOS)
This commit adds a flake.nix build file, it can be used for three
things:
Build `headscale` from local or straight from git:
nix build
or
nix build github:juanfont/headscale
Run and Build `headscale` from local or straight from git:
nix run
or
nix run github:juanfont/headscale
Set up a development environment including all our tools,
- linters
- protobuf tooling
- compilers
nix develop
Websockets, in which DERP is based, requires a TLS certificate. At the same time,
if we use a certificate it must be valid... otherwise Tailscale wont connect (does not
have an Insecure option). So there is no option to expose insecure here
- registry file /windows/tailscale.reg is generated, filling in the
associated control server URL
- also includes CLI instructions
- fix /apple incorrect template: 'Url' is supposed to be '.URL'
This series of commit will be adding an embedded DERP server (and STUN) to Headscale,
thus making it completely self-contained and not dependant in other infrastructure.
logs looks like the following
```
2022-03-02T20:43:08Z DBG Expanding alias=app-test
2022-03-02T20:43:08Z DBG Expanding alias=kube-test
2022-03-02T20:43:08Z DBG Expanding alias=test
2022-03-02T20:43:08Z WRN No IPs found with the alias test
2022-03-02T20:43:08Z DBG Expanding alias=prod
2022-03-02T20:43:08Z WRN No IPs found with the alias prod
2022-03-02T20:43:08Z DBG Expanding alias=prod
2022-03-02T20:43:08Z WRN No IPs found with the alias prod
```
- `preauthkey`, `authkey`, `pre` are aliases for `preauthkey` command
- `ls`, `show` are aliases for `list` subcommand
- `c`, `new` are aliases for `create` subcommand
- `revoke`, `exp`, `e` are aliases for `expire` subcommand
- `apikey`, `api` are aliases for `apikeys` command
- `ls`, `show` are aliases for `list` subcommand
- `c`, `new` are aliases for `create` subcommand
- `revoke`, `exp`, `e` are aliases for the `expire` subcommand
- `namespace`, `ns`, `user`, `users` are aliases for `namespaces`
command
- `c`, `new` are aliases for the `create` subcommand
- `delete` is an alias for the `destroy` subcommand
- `mv` is an alias for the `rename` subcommand
- `ls`, `show` are aliases for the `list` subcommand
- `node`, `machine`, `machines` are aliases for `nodes` command
- `ls`, `show` aliases for `list` subcommand
- `logout`, `exp`, `e` are aliases for `expire` subcommand
- `del` is an alias for `delete` subcommand
This commit removes the need for datatypes.JSON and makes the code a bit
cleaner by allowing us to use proper types throughout the code when it
comes to hostinfo and other datatypes on the machine object.
This allows us to remove alot of unmarshal/marshal operations and remove
a lot of obsolete error checks.
This following commits will clean away a lot of untyped data and
uneccessary error checks.
This commit removes the two extra caches (oidc, requested time) and uses
the new central registration cache instead. The requested time is
unified into the main machine object and the oidc key is just added to
the same cache, as a string with the state as a key instead of machine
key.
This commit removes the field from the database and does a DB migration
**removing** all unregistered machines from headscale.
This means that from this version, all machines in the database is
considered registered.
current logic is not safe as it will allow an IP that isnt persisted to
the DB to be given out multiple times if machines joins in quick
succession.
This adds a lock around the "get ip" and machine registration and save
to DB so we ensure thiis isnt happning.
Currently this had to be done three places, which is silly, and outlined
in #294.
After some more tests in tailscale I couldn't replicate the behavior
described in there.
When adding a rule, allowing A to talk to B the reverse connection was
instantly added to B to allow communication to B.
The previous assumption was probably wrong.
Using h.ListAllMachines also listed the current machine in the result. It's unnecessary (I don't know if it's harmful).
Breaking the check with the `matchSourceAndDestinationWithRule` broke the tests. We have a specificity with the '*' destination that isn't symetrical.
I need to think of a better way to do this. It too hard to read.
Rewrite some function to get rid of the dependency on Headscale object. This allows us
to write succinct test that are more easy to review and implement.
The improvements of the tests allowed to write the removal of the tagged hosts
from the namespace as specified here: https://tailscale.com/kb/1068/acl-tags/
This call should be done quite at each modification of a server resources like RequestTags.
When a server changes it's tag we should rebuild the ACL rules.
When a server is added to headscale we also should update the ACLRules.
This commit change the default behaviour and remove the notion of namespaces between the hosts. It allows all namespaces to be only filtered by the ACLs. This behavior is closer to tailsnet.
A hidden thing was implied in this document is that each person should have his own namespace.
Hidden information in spicification isn't good.
Thank's @kradalby for pointing it out.
This commit adds a sponsor/funding section to headscale.
@juanfont and I have discussed this and this arrangement is agreed upon
and hopefully this can bring us to a place in the future were even more
features and prioritization can be put upon the project.
This commit removes the namespace kv worker and related code, now that
we talk over gRPC to the server, and not directly to the DB, we should
not need this anymore.
Headscale commands fail when running them as the current user instead of the user defined in the systemd file. This note provides 2 methods of how to correctly run the headscale commands.
The automatic migration did not pick up this change and it breaks
current setups as it only adds a new field which is empty.
This means that clients who dont request a new IP will not have any IP
at all.
This commit configures the CI to run specific parts of the CI when
relevant changes has been made.
This should help us not have to deal with the integration tests when we
do doc/admin changes.
This commit adds a command to generate a private key for headscale.
Mostly useful for systems were you drive the deployment from another
machine and use a secret management system.
This commit enables the existing gRPC and HTTP API from remote locations
as long as the user can provide a valid API key. This allows users to
control their headscale with the CLI from a workstation. 🎉
This commits introduces a new data model for holding api keys for the
API. The keys are stored in the database with a prefix and a hash and
bcrypt with 10 passes is used to store the hash and it is "one way
safe".
Api keys have an expiry logic similar to pre auth keys.
A key cannot be retrieved after it has created, only verified.
* Resolves an issue where sometimes attempted sends on a closed channel
happened by ensuring the channels remain open for the entire goroutine.
* May be of help with regards to issue #203
- Fix URLs referring to files in this repository
- Better explain that we are creating the headscale directory and running the commands on the host Docker node
- Place instructions to download example config file to use as config file, as recommended steps.
This commit starts restructuring the documentation and updating it to be
compliant with 0.12.x+ releases.
The main change is that the documentation has been rewritten for the
ground up, and hopefully simplified.
The documentation has been split into an official documentation for
running headscale as a binary under Linux with SystemD and a "community"
provided documentation for Docker.
This should make the two documents a lot easier to read and follow than
the mishmash document we had.
This commit converts all the uses of wgkey to the new key interfaces.
It now has specific machine, node and discovery keys and we now should
use them correctly.
Please note the new logic which strips a key prefix (in utils.go) that
is now standard inside tailscale.
In theory we could put it in the database, but to preserve backwards
compatibility and not spend a lot of resources on accounting for both,
we just strip them.
We dont seem to need the wireguard key anymore, we generate a key on
startup based on the new library and the users fetch it from /key.
Clean up app.go and update docs
a nil value, instead of the default value (1h). This resulted in the
preauthkey being created with expiration key '0001-01-01 00:00:00',
which meant the key would not work, because it was already expired.
This commit applies the default expiration time (1h) when a preauthkey
is created without a specific expiration. It also updates an integration
test to make sure this bug does not reoccur.
This commit adds a sentral cache to keep track of clients whom has
requested an expiry time, but were we need to keep hold of it until the
second request comes in.
We should never expose errors via web, it gives attackers a lot of info
(Insert OWASP guide).
Also handle error that didnt separate not found gorm issue and other
errors.
This commits tries to dismantle the complicated "if and or" in the
RegistrationHandler by factoring out the "is Registrated" into a root
if.
This, together with some new comments, should hopefully make it a bit
easier to follow what is happening in all the different cases that needs
to be handled when a Node contacts the registration endpoint.
This commit is trying to DRY up the initiation of the gRPC client in
each command:
It renames the function to CLI instead of GRPC as it actually set up a
CLI client, not a generic grpc client
It also moves the configuration of address, timeout (which is now
consistent) and api to use Viper, allowing users to set it via env vars
and configuration file
This PR adds a new part to the integration test suite which spins up a
new headscale and runs through a scenario of test cases for each
command.
The intent is to check that all commands work as intended and produce
the expected output.
I think they have been pretty well covered, but would appreciate
additional test cases if I have missed some.
Please note: headscale is set up, and teared down for _each_ "test
function" in this file, this means that its more suitable for specific
cases.
This commit is a first in a series of commits migrating the command
interfaces to use the new gRPC client.
As a part of this commit, they have been streamlined and each command
_should_ be a bit more similar and use consistent output.
By using the new output function, we now make sure its always json
(errors and everything) if the user asks for JSON.
This commit moves the routes lookup functions to be subcommands of
Machine, making them a lot simpler and more specific/composable.
It also moves the register command from cli.go into machine, so we can
clear out the extra file.
Finally a toProto function has been added to convert between the machine
database model and the proto/rpc model.
This commit adds a debug command tree, intended to host commands used
for debugging and testing.
It adds a create node/machine command which will be used later to create
machines that can be used to test the registration command.
This commit adds proto rpc definitions for the communication needed for
the CLI interface.
This will allow us to move the rest of the CLI interface over to gRPC
and in the future allow remote access
This commit changes the way CLI and grpc-gateway communicates with the
gRPC backend to socket, instead of localhost. Unauthenticated access now
goes on the socket, while the network interface will require API key (in
the future).
This commit sets up the API and gRPC endpoints and adds authentication
to them. Currently there is no actual authentication implemented but it
has been prepared for API keys.
In addition, there is a allow put in place for gRPC traffic over
localhost. This has two purposes:
1. grpc-gateway, which is the base of the API, connects to the gRPC
service over localhost.
2. We do not want to break current "on server" behaviour which allows
users to use the cli on the server without any fuzz
This does not have to be reviewed, here is some reasoning:
Go (and go mod) is designed for having code available and we need to
check in the generated code to make sure it is "go gettable". If we dont
we give ourselves a headache trying to setup all the ci, tests etc to
install and generate the code before it runs.
Because the code isnt there, the plugins needed to generate the code
fail to install...
I didnt find any good documentation for this, but there is this github
comment:
https://github.com/golang/go/issues/34514#issuecomment-535406759
This commit moves the TLS configuration into a seperate function.
It also wires up the gRPC interface and prepares handing the API
endpoints to the grpc gateway.
* when listing nodes, a namespace is now optional, when it is not
provided, all nodes are shown
* when deleting, and sharing a node, remove the `namespace` flag, it was
superfluous and unused
* when unsharing a node, specify the namespace as an argument not a
flag, making the UX the same as for sharing.
Also refactor the share/unshare code to reuse the shared bits.
This PR will promote fetching the derpmap directly from tailscale, so we
will remove our example, as it might easily get outdated.
Add a derp-example that shows how a user can also add their own derp
server.
description: Use this agent when you need to execute, analyze, or troubleshoot Headscale integration tests. This includes running specific test scenarios, investigating test failures, interpreting test artifacts, validating end-to-end functionality, or ensuring integration test quality before releases. Examples: <example>Context: User has made changes to the route management code and wants to validate the changes work correctly. user: 'I've updated the route advertisement logic in poll.go. Can you run the relevant integration tests to make sure everything still works?' assistant: 'I'll use the headscale-integration-tester agent to run the subnet routing integration tests and analyze the results.' <commentary>Since the user wants to validate route-related changes with integration tests, use the headscale-integration-tester agent to execute the appropriate tests and analyze results.</commentary></example> <example>Context: A CI pipeline integration test is failing and the user needs help understanding why. user: 'The TestSubnetRouterMultiNetwork test is failing in CI. The logs show some timing issues but I can't figure out what's wrong.' assistant: 'Let me use the headscale-integration-tester agent to analyze the test failure and examine the artifacts.' <commentary>Since this involves analyzing integration test failures and interpreting test artifacts, use the headscale-integration-tester agent to investigate the issue.</commentary></example>
color: green
---
You are a specialist Quality Assurance Engineer with deep expertise in Headscale's integration testing system. You understand the Docker-based test infrastructure, real Tailscale client interactions, and the complex timing considerations involved in end-to-end network testing.
## Integration Test System Overview
The Headscale integration test system uses Docker containers running real Tailscale clients against a Headscale server. Tests validate end-to-end functionality including routing, ACLs, node lifecycle, and network coordination. The system is built around the `hi` (Headscale Integration) test runner in `cmd/hi/`.
## Critical Test Execution Knowledge
### System Requirements and Setup
```bash
# ALWAYS run this first to verify system readiness
go run ./cmd/hi doctor
```
This command verifies:
- Docker installation and daemon status
- Go environment setup
- Required container images availability
- Sufficient disk space (critical - tests generate ~100MB logs per run)
- Network configuration
### Test Execution Patterns
**CRITICAL TIMEOUT REQUIREMENTS**:
- **NEVER use bash `timeout` command** - this can cause test failures and incomplete cleanup
- **ALWAYS use the built-in `--timeout` flag** with generous timeouts (minimum 15 minutes)
- **Increase timeout if tests ever time out** - infrastructure issues require longer timeouts
```bash
# Single test execution (recommended for development)
# ALWAYS use --timeout flag with minimum 15 minutes (900s)
go run ./cmd/hi run "TestSubnetRouterMultiNetwork" --timeout=900s
# Database-heavy tests require PostgreSQL backend and longer timeouts
go run ./cmd/hi run "TestExpireNode" --postgres --timeout=1800s
# Pattern matching for related tests - use longer timeout for multiple tests
go run ./cmd/hi run "TestSubnet*" --timeout=1800s
# Long-running individual tests need extended timeouts
go run ./cmd/hi run "TestNodeOnlineStatus" --timeout=2100s # Runs for 12+ minutes
# Full test suite (CI/validation only) - very long timeout required
- **Slow tests** (5+ min): Node expiration, HA failover
- **Long-running tests** (10+ min): `TestNodeOnlineStatus` runs for 12 minutes
**CONCURRENT EXECUTION**: Multiple tests CAN run simultaneously. Each test run gets a unique Run ID for isolation. See "Concurrent Execution and Run ID Isolation" section below.
## Test Artifacts and Log Analysis
### Artifact Structure
All test runs save comprehensive artifacts to `control_logs/TIMESTAMP-ID/`:
```
control_logs/20250713-213106-iajsux/
├── hs-testname-abc123.stderr.log # Headscale server error logs
├── hs-testname-abc123.stdout.log # Headscale server output logs
├── hs-testname-abc123.db # Database snapshot for post-mortem
3.**MapResponse JSON files**: Protocol-level debugging for network map generation issues
4.**Client status dumps** (`*_status.json`): Network state and peer connectivity information
5.**Database snapshots** (`.db` files): For data consistency and state persistence issues
## Concurrent Execution and Run ID Isolation
### Overview
The integration test system supports running multiple tests concurrently on the same Docker daemon. Each test run is isolated through a unique Run ID that ensures containers, networks, and cleanup operations don't interfere with each other.
### Run ID Format and Usage
Each test run generates a unique Run ID in the format: `YYYYMMDD-HHMMSS-{6-char-hash}`
## Common Failure Patterns and Root Cause Analysis
### CRITICAL MINDSET: Code Issues vs Infrastructure Issues
**⚠️ IMPORTANT**: When tests fail, it is ALMOST ALWAYS a code issue with Headscale, NOT infrastructure problems. Do not immediately blame disk space, Docker issues, or timing unless you have thoroughly investigated the actual error logs first.
### Systematic Debugging Process
1.**Read the actual error message**: Don't assume - read the stderr logs completely
2.**Check Headscale server logs first**: Most issues originate from server-side logic
3.**Verify client connectivity**: Only after ruling out server issues
4.**Check timing patterns**: Use proper `EventuallyWithT` patterns
5.**Infrastructure as last resort**: Only blame infrastructure after code analysis
### Real Failure Patterns
#### 1. Timing Issues (Common but fixable)
```go
// ❌ Wrong: Immediate assertions after async operations
},10*time.Second,100*time.Millisecond,"route should be advertised")
```
**Timeout Guidelines**:
- Route operations: 3-5 seconds
- Node state changes: 5-10 seconds
- Complex scenarios: 10-15 seconds
- Policy recalculation: 5-10 seconds
#### 2. NodeStore Synchronization Issues
Route advertisements must propagate through poll requests (`poll.go:420`). NodeStore updates happen at specific synchronization points after Hostinfo changes.
**Why focus on WHY**: Helps maintainers understand architectural decisions and security requirements.
## EventuallyWithT Pattern for External Calls
### Overview
EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
### External Calls That Must Be Wrapped
The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
- `headscale.ListNodes()` - Queries server state
- `client.Status()` - Gets client network status
- `client.Curl()` - Makes HTTP requests through the network
}, 10*time.Second, 100*time.Millisecond, "route should be approved")
```
## Your Core Responsibilities
1. **Test Execution Strategy**: Execute integration tests with appropriate configurations, understanding when to use `--postgres` and timing requirements for different test categories. Follow phase-based testing approach prioritizing route tests.
- **Why this priority**: Route tests are less infrastructure-sensitive and validate core security logic
2. **Systematic Test Analysis**: When tests fail, systematically examine artifacts starting with Headscale server logs, then client logs, then protocol data. Focus on CODE ISSUES first (99% of cases), not infrastructure. Use real-world failure patterns to guide investigation.
- **Why this approach**: Most failures are logic bugs, not environment issues - efficient debugging saves time
3. **Timing & Synchronization Expertise**: Understand asynchronous Headscale operations, particularly route advertisements, NodeStore synchronization at `poll.go:420`, and policy propagation. Fix timing with `EventuallyWithT` while preserving original test expectations.
- **Why preserve expectations**: Test assertions encode business requirements and security policies
- **Key Pattern**: Apply the EventuallyWithT pattern correctly for all external calls as documented above
4. **Root Cause Analysis**: Distinguish between actual code regressions (route approval logic, HA failover architecture), timing issues requiring `EventuallyWithT` patterns, and genuine infrastructure problems (DNS, Docker, container issues).
- **Why this distinction matters**: Different problem types require completely different solution approaches
- **EventuallyWithT Issues**: Often manifest as flaky tests or immediate assertion failures after async operations
5.**Security-Aware Quality Validation**: Ensure tests properly validate end-to-end functionality with realistic timing expectations and proper error handling. Never suggest security bypasses or test expectation changes. Add comprehensive godoc when you understand test business logic.
- **Why security focus**: Integration tests are the last line of defense against security regressions
- **EventuallyWithT Usage**: Proper use prevents race conditions without weakening security assertions
6.**Concurrent Execution Awareness**: Respect run ID isolation and never interfere with other agents' test sessions. Each test run has a unique run ID - only clean up YOUR containers (by run ID label), never perform global cleanup while tests may be running.
- **Why this matters**: Multiple agents/users may run tests concurrently on the same Docker daemon
- **Key Rule**: NEVER use global container cleanup commands - the test runner handles cleanup automatically per run ID
**CRITICAL PRINCIPLE**: Test expectations are sacred contracts that define correct system behavior. When tests fail, fix the code to match the test, never change the test to match broken code. Only timing and observability improvements are allowed - business logic expectations are immutable.
**ISOLATION PRINCIPLE**: Each test run is isolated by its unique Run ID. Never interfere with other test sessions. The system handles cleanup automatically - manual global cleanup commands are forbidden when other tests may be running.
**EventuallyWithT PRINCIPLE**: Every external call to headscale server or tailscale client must be wrapped in EventuallyWithT. Follow the five key rules strictly: one external call per block, proper variable scoping, no nesting, use CollectT for assertions, and provide descriptive messages.
**Remember**: Test failures are usually code issues in Headscale that need to be fixed, not infrastructure problems to be ignored. Use the specific debugging workflows and failure patterns documented above to efficiently identify root causes. Infrastructure issues have very specific signatures - everything else is code-related.
body: 'Nix build failed with wrong gosum, please update "vendorSha256" (${{ steps.build.outputs.OLD_HASH }}) for the "headscale" package in flake.nix with the new SHA: ${{ steps.build.outputs.NEW_HASH }}'
# This is an example .goreleaser.yml file with some sane defaults.
# Make sure to check the documentation at http://goreleaser.com
---
version:2
before:
hooks:
- go mod tidy
- go mod tidy -compat=1.25
- go mod vendor
release:
prerelease:auto
draft:true
header:|
## Upgrade
Please follow the steps outlined in the [upgrade guide](https://headscale.net/stable/setup/upgrade/) to update your existing Headscale installation.
**It's best to update from one stable version to the next** (e.g., 0.24.0 → 0.25.1 → 0.26.1) in case you are multiple releases behind. You should always pick the latest available patch release.
Be sure to check the changelog above for version-specific upgrade instructions and breaking changes.
### Backup Your Database
**Always backup your database before upgrading.** Here's how to backup a SQLite database:
name_template:'{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}{{ with .Arm }}v{{ . }}{{ end }}{{ with .Mips }}_{{ . }}{{ end }}{{ if not (eq .Amd64 "v1") }}{{ .Amd64 }}{{ end }}'
formats:
- binary
source:
enabled:true
name_template:"{{ .ProjectName }}_{{ .Version }}"
format:tar.gz
files:
- "vendor/"
nfpms:
# Configure nFPM for .deb and .rpm releases
#
# See https://nfpm.goreleaser.com/configuration/
# and https://goreleaser.com/customization/nfpm/
#
# Useful tools for debugging .debs:
# List file contents: dpkg -c dist/headscale...deb
Headscale is "Open Source, acknowledged contribution", this means that any contribution will have to be discussed with the maintainers before being added to the project.
This model has been chosen to reduce the risk of burnout by limiting the maintenance overhead of reviewing and validating third-party code.
## Why do we have this model?
Headscale has a small maintainer team that tries to balance working on the project, fixing bugs and reviewing contributions.
When we work on issues ourselves, we develop first hand knowledge of the code and it makes it possible for us to maintain and own the code as the project develops.
Code contributions are seen as a positive thing. People enjoy and engage with our project, but it also comes with some challenges; we have to understand the code, we have to understand the feature, we might have to become familiar with external libraries or services and we think about security implications. All those steps are required during the reviewing process. After the code has been merged, the feature has to be maintained. Any changes reliant on external services must be updated and expanded accordingly.
The review and day-1 maintenance adds a significant burden on the maintainers. Often we hope that the contributor will help out, but we found that most of the time, they disappear after their new feature was added.
This means that when someone contributes, we are mostly happy about it, but we do have to run it through a series of checks to establish if we actually can maintain this feature.
## What do we require?
A general description is provided here and an explicit list is provided in our pull request template.
All new features have to start out with a design document, which should be discussed on the issue tracker (not discord). It should include a use case for the feature, how it can be implemented, who will implement it and a plan for maintaining it.
All features have to be end-to-end tested (integration tests) and have good unit test coverage to ensure that they work as expected. This will also ensure that the feature continues to work as expected over time. If a change cannot be tested, a strong case for why this is not possible needs to be presented.
The contributor should help to maintain the feature over time. In case the feature is not maintained probably, the maintainers reserve themselves the right to remove features they redeem as unmaintainable. This should help to improve the quality of the software and keep it in a maintainable state.
## Bug fixes
Headscale is open to code contributions for bug fixes without discussion.
## Documentation
If you find mistakes in the documentation, please submit a fix to the documentation.
An open source, self-hosted implementation of the Tailscale coordination server.
An open source, self-hosted implementation of the Tailscale control server.
Join our [Discord](https://discord.gg/XcQxk2VHjx) server for a chat.
Join our [Discord server](https://discord.gg/c84AZQhmpx) for a chat.
## Overview
**Note:** Always select the same GitHub tag as the released version you use
to ensure you have the correct example configuration. The `main` branch might
contain unreleased changes. The documentation is available for stable and
development versions:
Tailscale is [a modern VPN](https://tailscale.com/) built on top of [Wireguard](https://www.wireguard.com/). It [works like an overlay network](https://tailscale.com/blog/how-tailscale-works/) between the computers of your networks - using all kinds of [NAT traversal sorcery](https://tailscale.com/blog/how-nat-traversal-works/).
- [Documentation for the stable version](https://headscale.net/stable/)
- [Documentation for the development version](https://headscale.net/development/)
Everything in Tailscale is Open Source, except the GUI clients for proprietary OS (Windows and macOS/iOS), and the 'coordination/control server'.
## What is Tailscale
The control server works as an exchange point of Wireguard public keys for the nodes in the Tailscale network. It also assigns the IP addresses of the clients, creates the boundaries between each user, enables sharing machines between users, and exposes the advertised routes of your nodes.
Tailscale is [a modern VPN](https://tailscale.com/) built on top of
[Wireguard](https://www.wireguard.com/).
It [works like an overlay network](https://tailscale.com/blog/how-tailscale-works/)
2. Use the authkey from your machine to register it
```shell
tailscale up --login-server YOUR_HEADSCALE_URL --authkey YOURAUTHKEY
```
If you create an authkey with the `--ephemeral` flag, that key will create ephemeral nodes. This implies that `--reusable` is true.
Please bear in mind that all the commands from headscale support adding `-o json` or `-o json-line` to get a nicely JSON-formatted output.
## Configuration reference
Headscale's configuration file is named `config.json` or `config.yaml`. Headscale will look for it in `/etc/headscale`, `~/.headscale` and finally the directory from where the Headscale binary is executed.
```
"server_url": "http://192.168.1.12:8080",
"listen_addr": "0.0.0.0:8080",
"ip_prefix": "100.64.0.0/10"
```
`server_url` is the external URL via which Headscale is reachable. `listen_addr` is the IP address and port the Headscale program should listen on. `ip_prefix` is the IP prefix (range) in which IP addresses for nodes will be allocated (default 100.64.0.0/10, e.g., 192.168.4.0/24, 10.0.0.0/8)
```
"log_level": "debug"
```
`log_level` can be used to set the Log level for Headscale, it defaults to `debug`, and the available levels are: `trace`, `debug`, `info`, `warn` and `error`.
```
"private_key_path": "private.key",
```
`private_key_path` is the path to the Wireguard private key. If the path is relative, it will be interpreted as relative to the directory the configuration file was read from.
```
"derp_map_path": "derp.yaml",
```
`derp_map_path` is the path to the [DERP](https://pkg.go.dev/tailscale.com/derp) map file. If the path is relative, it will be interpreted as relative to the directory the configuration file was read from.
```
"ephemeral_node_inactivity_timeout": "30m",
```
`ephemeral_node_inactivity_timeout` is the timeout after which inactive ephemeral node records will be deleted from the database. The default is 30 minutes. This value must be higher than 65 seconds (the keepalive timeout for the HTTP long poll is 60 seconds, plus a few seconds to avoid race conditions).
```
"db_host": "localhost",
"db_port": 5432,
"db_name": "headscale",
"db_user": "foo",
"db_pass": "bar",
```
The fields starting with `db_` are used for the PostgreSQL connection information.
### Running the service via TLS (optional)
```
"tls_cert_path": ""
"tls_key_path": ""
```
Headscale can be configured to expose its web service via TLS. To configure the certificate and key file manually, set the `tls_cert_path` and `tls_cert_path` configuration parameters. If the path is relative, it will be interpreted as relative to the directory the configuration file was read from.
```
"tls_letsencrypt_hostname": "",
"tls_letsencrypt_listen": ":http",
"tls_letsencrypt_cache_dir": ".cache",
"tls_letsencrypt_challenge_type": "HTTP-01",
```
To get a certificate automatically via [Let's Encrypt](https://letsencrypt.org/), set `tls_letsencrypt_hostname` to the desired certificate hostname. This name must resolve to the IP address(es) Headscale is reachable on (i.e., it must correspond to the `server_url` configuration parameter). The certificate and Let's Encrypt account credentials will be stored in the directory configured in `tls_letsencrypt_cache_dir`. If the path is relative, it will be interpreted as relative to the directory the configuration file was read from. The certificate will automatically be renewed as needed.
#### Challenge type HTTP-01
The default challenge type `HTTP-01` requires that Headscale is reachable on port 80 for the Let's Encrypt automated validation, in addition to whatever port is configured in `listen_addr`. By default, Headscale listens on port 80 on all local IPs for Let's Encrypt automated validation.
If you need to change the ip and/or port used by Headscale for the Let's Encrypt validation process, set `tls_letsencrypt_listen` to the appropriate value. This can be handy if you are running Headscale as a non-root user (or can't run `setcap`). Keep in mind, however, that Let's Encrypt will _only_ connect to port 80 for the validation callback, so if you change `tls_letsencrypt_listen` you will also need to configure something else (e.g. a firewall rule) to forward the traffic from port 80 to the ip:port combination specified in `tls_letsencrypt_listen`.
#### Challenge type TLS-ALPN-01
Alternatively, `tls_letsencrypt_challenge_type` can be set to `TLS-ALPN-01`. In this configuration, Headscale listens on the ip:port combination defined in `listen_addr`. Let's Encrypt will _only_ connect to port 443 for the validation callback, so if `listen_addr` is not set to port 443, something else (e.g. a firewall rule) will be required to forward the traffic from port 443 to the ip:port combination specified in `listen_addr`.
### Policy ACLs
Headscale implements the same policy ACLs as Tailscale.com, adapted to the self-hosted environment.
For instance, instead of referring to users when defining groups you must
use namespaces (which are the equivalent to user/logins in Tailscale.com).
Please check https://tailscale.com/kb/1018/acls/, and `./tests/acls/` in this repo for working examples.
### Apple devices
An endpoint with information on how to connect your Apple devices (currently macOS only) is available at `/apple` on your running instance.
- Fosdem 2023 (video): [Headscale: How we are using integration testing to reimplement Tailscale](https://fosdem.org/2023/schedule/event/goheadscale/)
- presented by Juan Font Alonso and Kristoffer Dalby
## Disclaimer
1. We have nothing to do with Tailscale, or Tailscale Inc.
2. The purpose of writing this was to learn how Tailscale works.
This project is not associated with Tailscale Inc.
## More on Tailscale
However, one of the active maintainers for Headscale [is employed by Tailscale](https://tailscale.com/blog/opensource) and he is allowed to spend work hours contributing to the project. Contributions from this maintainer are reviewed by other maintainers.
The maintainers work together on setting the direction for the project. The underlying principle is to serve the community of self-hosters, enthusiasts and hobbyists - while having a sustainable project.
## Contributing
Please read the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
### Requirements
To contribute to headscale you would need the latest version of [Go](https://golang.org)
and [Buf](https://buf.build) (Protobuf generator).
We recommend using [Nix](https://nixos.org/) to setup a development environment. This can
be done with `nix develop`, which will install the tools and give you a shell.
This guarantees that you will have the same dev env as `headscale` maintainers.
### Code style
To ensure we have some consistency with a growing number of contributions,
this project has adopted linting and style/formatting rules:
The **Go** code is linted with [`golangci-lint`](https://golangci-lint.run) and
formatted with [`golines`](https://github.com/segmentio/golines) (width 88) and
[`gofumpt`](https://github.com/mvdan/gofumpt).
Please configure your editor to run the tools while developing and make sure to
run `make lint` and `make fmt` before committing any code.
The **Proto** code is linted with [`buf`](https://docs.buf.build/lint/overview) and
formatted with [`clang-format`](https://clang.llvm.org/docs/ClangFormat.html).
The **rest** (Markdown, YAML, etc) is formatted with [`prettier`](https://prettier.io).
Check out the `.golangci.yaml` and `Makefile` to see the specific configuration.
### Install development tools
- Go
- Buf
- Protobuf tools
Install and activate:
```shell
nix develop
```
### Testing and building
Some parts of the project require the generation of Go code from Protobuf
(if changes are made in `proto/`) and it must be (re-)generated with:
```shell
make generate
```
**Note**: Please check in changes from `gen/` in a separate commit to make it easier to review.
To run the tests:
```shell
make test
```
To build the program:
```shell
make build
```
### Development workflow
We recommend using Nix for dependency management to ensure you have all required tools. If you prefer to manage dependencies yourself, you can use Make directly:
**With Nix (recommended):**
```shell
nix develop
make test
make build
```
**With your own dependencies:**
```shell
make test
make build
```
The Makefile will warn you if any required tools are missing and suggest running `nix develop`. Run `make help` to see all available targets.
This page provides <a href="https://support.apple.com/guide/mdm/mdm-overview-mdmbf9e668/web">configuration profiles</a> for the official Tailscale clients for <a href="https://apps.apple.com/us/app/tailscale/id1470499037?ls=1">iOS</a> and <a href="https://apps.apple.com/ca/app/tailscale/id1475387142?mt=12">macOS</a>.
</p>
<p>
The profiles will configure Tailscale.app to use {{.Url}} as its control server.
</p>
<h3>Caution</h3>
<p>You should always inspect the profile before installing it:</p>
approveRoutesCmd.Flags().StringSliceP("routes","r",[]string{},`List of routes that will be approved (comma-separated, e.g. "10.0.0.0/8,192.168.0.0/24" or empty string to remove all approved routes)`)
getPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
policyCmd.AddCommand(getPolicy)
setPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
setPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
policyCmd.AddCommand(setPolicy)
checkPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
// this is only a warning because there could be something sitting in front of headscale that redirects the traffic (e.g. an iptables rule)
log.Warn().
Msg("Warning: when using tls_letsencrypt_hostname with TLS-ALPN-01 as challenge type, headscale must be reachable on port 443, i.e. listen_addr should probably end in :443")
err=fmt.Errorf("ephemeral_node_inactivity_timeout (%s) is set too low, must be more than %s\n",viper.GetString("ephemeral_node_inactivity_timeout"),minInactivityTimeout)
Long story short, you can define the DNS servers you want to use in your tailnets, activate MagicDNS (so you don't have to remember the IP addresses of your nodes), define search domains, as well as predefined hosts. Headscale will inject that settings into your nodes.
## Configuration reference
The setup is done via the `config.json` file, under the `dns_config` key.
```json
{
"server_url":"http://127.0.0.1:8001",
"listen_addr":"0.0.0.0:8001",
"private_key_path":"private.key",
//...
"dns_config":{
"nameservers":["1.1.1.1","8.8.8.8"],
"domains":[],
"magic_dns":true,
"base_domain":"example.com"
}
}
```
-`nameservers`: The list of DNS servers to use.
-`domains`: Search domains to inject.
-`magic_dns`: Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/). Only works if there is at least a nameserver defined.
- `base_domain`: Defines the base domain to create the hostnames for MagicDNS. `base_domain` must be a FQDNs, without the trailing dot. The FQDN of the hosts will be `hostname.namespace.base_domain` (e.g., _myhost.mynamespace.example.com_).
Headscale aims to implement a self-hosted, open source alternative to the
[Tailscale](https://tailscale.com/) control server. Headscale's goal is to
provide self-hosters and hobbyists with an open-source server they can use for
their projects and labs. It implements a narrow scope, a _single_ Tailscale
network (tailnet), suitable for a personal use, or a small open-source
organisation.
## How can I contribute?
Headscale is "Open Source, acknowledged contribution", this means that any
contribution will have to be discussed with the Maintainers before being submitted.
Please see [Contributing](contributing.md) for more information.
## Why is 'acknowledged contribution' the chosen model?
Both maintainers have full-time jobs and families, and we want to avoid burnout. We also want to avoid frustration from contributors when their PRs are not accepted.
We are more than happy to exchange emails, or to have dedicated calls before a PR is submitted.
## When/Why is Feature X going to be implemented?
We don't know. We might be working on it. If you're interested in contributing, please post a feature request about it.
Please be aware that there are a number of reasons why we might not accept specific contributions:
- It is not possible to implement the feature in a way that makes sense in a self-hosted environment.
- Given that we are reverse-engineering Tailscale to satisfy our own curiosity, we might be interested in implementing the feature ourselves.
- You are not sending unit and integration tests with it.
## Do you support Y method of deploying headscale?
We currently support deploying headscale using our binaries and the DEB packages. Visit our [installation guide using
official releases](../setup/install/official.md) for more information.
In addition to that, you may use packages provided by the community or from distributions. Learn more in the
[installation guide using community packages](../setup/install/community.md).
For convenience, we also [build container images with headscale](../setup/install/container.md). But **please be aware that
we don't officially support deploying headscale using Docker**. On our [Discord server](https://discord.gg/c84AZQhmpx)
we have a "docker-issues" channel where you can ask for Docker-specific help to the community.
## What is the recommended update path? Can I skip multiple versions while updating?
Please follow the steps outlined in the [upgrade guide](../setup/upgrade.md) to update your existing Headscale
installation. Its best to update from one stable version to the next (e.g. 0.24.0 → 0.25.1 → 0.26.1) in case
you are multiple releases behind. You should always pick the latest available patch release.
Be sure to check the [changelog](https://github.com/juanfont/headscale/blob/main/CHANGELOG.md) for version specific
upgrade instructions and breaking changes.
## Scaling / How many clients does Headscale support?
It depends. As often stated, Headscale is not enterprise software and our focus
is homelabbers and self-hosters. Of course, we do not prevent people from using
it in a commercial/professional setting and often get questions about scaling.
Please note that when Headscale is developed, performance is not part of the
consideration as the main audience is considered to be users with a modest
amount of devices. We focus on correctness and feature parity with Tailscale
SaaS over time.
To understand if you might be able to use Headscale for your use case, I will
describe two scenarios in an effort to explain what is the central bottleneck
of Headscale:
1. An environment with 1000 servers
- they rarely "move" (change their endpoints)
- new nodes are added rarely
2. An environment with 80 laptops/phones (end user devices)
- nodes move often, e.g. switching from home to office
Headscale calculates a map of all nodes that need to talk to each other,
creating this "world map" requires a lot of CPU time. When an event that
requires changes to this map happens, the whole "world" is recalculated, and a
new "world map" is created for every node in the network.
This means that under certain conditions, Headscale can likely handle 100s
of devices (maybe more), if there is _little to no change_ happening in the
network. For example, in Scenario 1, the process of computing the world map is
extremely demanding due to the size of the network, but when the map has been
created and the nodes are not changing, the Headscale instance will likely
return to a very low resource usage until the next time there is an event
requiring the new map.
In the case of Scenario 2, the process of computing the world map is less
demanding due to the smaller size of the network, however, the type of nodes
will likely change frequently, which would lead to a constant resource usage.
Headscale will start to struggle when the two scenarios overlap, e.g. many nodes
with frequent changes will cause the resource usage to remain constantly high.
In the worst case scenario, the queue of nodes waiting for their map will grow
to a point where Headscale never will be able to catch up, and nodes will never
learn about the current state of the world.
We expect that the performance will improve over time as we improve the code
base, but it is not a focus. In general, we will never make the tradeoff to make
things faster on the cost of less maintainable or readable code. We are a small
team and have to optimise for maintainability.
## Which database should I use?
We recommend the use of SQLite as database for headscale:
- SQLite is simple to setup and easy to use
- It scales well for all of headscale's use cases
- Development and testing happens primarily on SQLite
- PostgreSQL is still supported, but is considered to be in "maintenance mode"
The headscale project itself does not provide a tool to migrate from PostgreSQL to SQLite. Please have a look at [the
related tools documentation](../ref/integration/tools.md) for migration tooling provided by the community.
The choice of database has little to no impact on the performance of the server,
see [Scaling / How many clients does Headscale support?](#scaling-how-many-clients-does-headscale-support) for understanding how Headscale spends its resources.
## Why is my reverse proxy not working with headscale?
We don't know. We don't use reverse proxies with headscale ourselves, so we don't have any experience with them. We have
[community documentation](../ref/integration/reverse-proxy.md) on how to configure various reverse proxies, and a
dedicated "reverse-proxy-issues" channel on our [Discord server](https://discord.gg/c84AZQhmpx) where you can ask for
help to the community.
## Can I use headscale and tailscale on the same machine?
Running headscale on a machine that is also in the tailnet can cause problems with subnet routers, traffic relay nodes, and MagicDNS. It might work, but it is not supported.
## Why do two nodes see each other in their status, even if an ACL allows traffic only in one direction?
A frequent use case is to allow traffic only from one node to another, but not the other way around. For example, the
workstation of an administrator should be able to connect to all nodes but the nodes themselves shouldn't be able to
connect back to the administrator's node. Why do all nodes see the administrator's workstation in the output of
`tailscale status`?
This is essentially how Tailscale works. If traffic is allowed to flow in one direction, then both nodes see each other
in their output of `tailscale status`. Traffic is still filtered according to the ACL, with the exception of `tailscale
ping` which is always allowed in either direction.
See also <https://tailscale.com/kb/1087/device-visibility>.
## My policy is stored in the database and Headscale refuses to start due to an invalid policy. How can I recover?
Headscale checks if the policy is valid during startup and refuses to start if it detects an error. The error message
indicates which part of the policy is invalid. Follow these steps to fix your policy:
- Dump the policy to a file: `headscale policy get --bypass-grpc-and-access-database-directly > policy.json`
- Edit and fixup `policy.json`. Use the command `headscale policy check --file policy.json` to validate the policy.
- Load the modified policy: `headscale policy set --bypass-grpc-and-access-database-directly --file policy.json`
- Start Headscale as usual.
!!! warning "Full server configuration required"
The above commands to get/set the policy require a complete server configuration file including database settings. A
minimal config to [control Headscale via remote CLI](../ref/api.md#grpc) is not sufficient. You may use `headscale
-c /path/to/config.yaml` to specify the path to an alternative configuration file.
## How can I avoid to send logs to Tailscale Inc?
A Tailscale client [collects logs about its operation and connection attempts with other
clients](https://tailscale.com/kb/1011/log-mesh-traffic#client-logs) and sends them to a central log service operated by
Tailscale Inc.
Headscale, by default, instructs clients to disable log submission to the central log service. This configuration is
applied by a client once it successfully connected with Headscale. See the configuration option `logtail.enabled` in the
[configuration file](../ref/configuration.md) for details.
Alternatively, logging can also be disabled on the client side. This is independent of Headscale and opting out of
client logging disables log submission early during client startup. The configuration is operating system specific and
is usually achieved by setting the environment variable `TS_NO_LOGS_NO_SUPPORT=true` or by passing the flag
`--no-logs-no-support` to `tailscaled`. See
<https://tailscale.com/kb/1011/log-mesh-traffic#opting-out-of-client-logging> for details.
An Atom/RSS feed of headscale releases is available [here](https://github.com/juanfont/headscale/releases.atom).
See the "announcements" channel on our [Discord server](https://discord.gg/c84AZQhmpx) for news about headscale.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
