Update documentation to reflect the new concurrent test execution
capabilities and add guidance on run ID isolation.
AGENTS.md:
- Add examples for running multiple tests concurrently
- Document run ID format and container naming conventions
- Update "Critical Notes" to explain isolation mechanisms
.claude/agents/headscale-integration-tester.md:
- Add "Concurrent Execution and Run ID Isolation" section
- Document forbidden and safe operations for cleanup
- Add "Agent Session Isolation Rules" for multi-agent environments
- Add 6th core responsibility about concurrent execution awareness
- Add ISOLATION PRINCIPLE to critical principles
- Update pre-test cleanup documentation
Remove the concurrent test prevention logic and update cleanup to use
run ID-based isolation, allowing multiple tests to run simultaneously.
Changes:
- cleanup: Add killTestContainersByRunID() to clean only containers
belonging to a specific run, add cleanupStaleTestContainers() to
remove only stopped/exited containers without affecting running tests
- docker: Remove RunningTestInfo, checkForRunningTests(), and related
error types, update cleanupAfterTest() to use run ID-based cleanup
- run: Remove Force flag and concurrent test prevention check
The test runner now:
- Allows multiple concurrent test runs on the same Docker daemon
- Cleans only stale containers before tests (not running ones)
- Cleans only containers with matching run ID after tests
- Prints run ID and monitoring info for operator visibility
Add run ID-based isolation to container naming and network setup to
enable multiple integration tests to run concurrently on the same
Docker daemon without conflicts.
Changes:
- hsic: Add run ID prefix to headscale container names and use dynamic
port allocation for metrics endpoint (port 0 lets kernel assign)
- tsic: Add run ID prefix to tailscale container names
- dsic: Add run ID prefix to DERP container names
- scenario: Use run ID-aware test suite container name for network setup
Container naming now follows: {type}-{runIDShort}-{identifier}-{hash}
Example: ts-mdjtzx-1-74-fgdyls, hs-mdjtzx-pingallbyip-abc123
The run ID is obtained from HEADSCALE_INTEGRATION_RUN_ID environment
variable via dockertestutil.GetIntegrationRunID().
This commit replaces the ChangeSet with a simpler bool based
change model that can be directly used in the map builder to
build the appropriate map response based on the change that
has occured. Previously, we fell back to sending full maps
for a lot of changes as that was consider "the safe" thing to
do to ensure no updates were missed.
This was slightly problematic as a node that already has a list
of peers will only do full replacement of the peers if the list
is non-empty, meaning that it was not possible to remove all
nodes (if for example policy changed).
Now we will keep track of last seen nodes, so we can send remove
ids, but also we are much smarter on how we send smaller, partial
maps when needed.
Fixes#2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit adds tests to validate that there are
issues with how we propagate tag changes in the system.
This replicates #2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit changes so that node changes to the policy is
calculated if any of the nodes has changed in a way that might
affect the policy.
Previously we just checked if the number of nodes had changed,
which meant that if a node was added and removed, we would be
in a bad state.
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This PR restructures the integration tests and prebuilds all common assets used in all tests:
Headscale and Tailscale HEAD image
hi binary that is used to run tests
go cache is warmed up for compilation of the test
This essentially means we spend 6-10 minutes building assets before any tests starts, when that is done, all tests can just sprint through.
It looks like we are saving 3-9 minutes per test, and since we are limited to running max 20 concurrent tests across the repo, that means we had a lot of double work.
There is currently 113 checks, so we have to do five runs of 20, and the saving should be quite noticeable! I think the "worst case" saving would be 20+min and "best case" probably towards an hour.
Fixes#2927
In v0.27.0, the list-routes command with -i flag and -o json output
was returning all nodes instead of just the specified node.
The issue was that JSON output was happening before the identifier
filtering logic. This change moves the JSON output to after both
the identifier filter and route existence filter are applied,
ensuring the correct filtered results are returned.
This restores the v0.26.1 behavior where:
headscale nodes list-routes -i 12 -o json
correctly returns only node 12's route information.
This PR investigates, adds tests and aims to correctly implement Tailscale's model for how Tags should be accepted, assigned and used to identify nodes in the Tailscale access and ownership model.
When evaluating in Headscale's policy, Tags are now only checked against a nodes "tags" list, which defines the source of truth for all tags for a given node. This simplifies the code for dealing with tags greatly, and should help us have less access bugs related to nodes belonging to tags or users.
A node can either be owned by a user, or a tag.
Next, to ensure the tags list on the node is correctly implemented, we first add tests for every registration scenario and combination of user, pre auth key and pre auth key with tags with the same registration expectation as observed by trying them all with the Tailscale control server. This should ensure that we implement the correct behaviour and that it does not change or break over time.
Lastly, the missing parts of the auth has been added, or changed in the cases where it was wrong. This has in large parts allowed us to delete and simplify a lot of code.
Now, tags can only be changed when a node authenticates or if set via the CLI/API. Tags can only be fully overwritten/replaced and any use of either auth or CLI will replace the current set if different.
A user owned device can be converted to a tagged device, but it cannot be changed back. A tagged device can never remove the last tag either, it has to have a minimum of one.
This PR changes tags to be something that exists on nodes in addition to users, to being its own thing. It is part of moving our tags support towards the correct tailscale compatible implementation.
There are probably rough edges in this PR, but the intention is to get it in, and then start fixing bugs from 0.28.0 milestone (long standing tags issue) to discover what works and what doesnt.
Updates #2417Closes#2619
This improves security and explicitly fails on startup when a user picks
the wrong directory to store its data.
- Run in read-only mode
- Make /var/run/headscale a read-write tmpfs
- Mount the config volume read-only
- Use the /health endpoint to check if Headscale is up
Previously we tested migrations on schemas and dumps
of old databases.
The problems with testing migrations against the schemas
is that the migration table is empty, so we try to run
migrations that are already ran on that schema, which might
blow up.
This commit removes the schema approach and just leaves all
the dumps, which include the migration table.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Document the API endpoint and the built-in swagger docs at /swagger. The
remote control docs are just a use case for gRPC - move it in the API
docs and update links to it.
The former is a no-op in the base images (45491f2c5c/scripts/debuerreotype-minimizing-config (L87-L109)), and `apt-get dist-clean` is a safer/better version of the `rm -rf /var/lib/apt/lists/*` that keeps the cryptographic bits that help prevent downgrade attacks.
Move favicon.png, style.css, and headscale.svg to hscontrol/assets/
and create a single assets.go file with all embed directives.
Update hscontrol/handlers.go and hscontrol/templates/general.go to
use the centralized assets package.
Add test to validate HTML template output consistency across all
templates (OIDC callback, registration, Windows, Apple).
Verifies all templates produce valid HTML5 with:
- Proper DOCTYPE declaration
- HTML5 lang attribute
- UTF-8 charset
- Viewport meta tag
- Semantic HTML structure
Ensures template refactoring maintains standards compliance.
Refactor template system to use go:embed for external assets and
CSS classes for styling instead of inline styles:
- general.go: Add go:embed directives for style.css and headscale.svg,
replace inline styles with CSS classes (H1, H2, H3, P, etc.),
add mdTypesetBody wrapper with Material for MkDocs styling
- apple.go, oidc_callback.go, register_web.go, windows.go:
Update to use new CSS-based helper functions (H1, H2, P, etc.)
and mdTypesetBody for consistent layout
This separates content from presentation, making templates easier
to maintain and update. All styling is now centralized in style.css
with Material for MkDocs design system.
Add design system assets for HTML templates:
- headscale.svg: Logo with optimized viewBox for proper alignment
- style.css: Material for MkDocs CSS variables and typography
- design.go: Design system constants for consistent styling
The logo viewBox is adjusted to 32.92 0 1247.08 640 to eliminate
whitespace from the original export and ensure left alignment with
text content.
Replace html/template with type-safe elem-go templating for OIDC
callback page. Improves consistency with other templates and provides
compile-time safety. All UI elements and styling preserved.
When tailscaled restarts, it sends RegisterRequest with Auth=nil and
Expiry=zero. Previously this was treated as a logout because
time.Time{}.Before(time.Now()) returns true.
Add early return in handleRegister() to detect this case and preserve
the existing node state without modification.
Fixes#2862
Changed UpdateUser and re-registration flows to use Updates() which only
writes modified fields, preventing unintended overwrites of unchanged fields.
Also updated UsePreAuthKey to use Model().Update() for single field updates
and removed unused NodeSave wrapper.
Fixes a regression introduced in v0.27.0 where node expiry times were
being reset to zero when tailscaled restarts and sends a MapRequest.
The issue was caused by using GORM's Save() method in persistNodeToDB(),
which overwrites ALL fields including zero values. When a MapRequest
updates a node (without including expiry information), Save() would
overwrite the database expiry field with a zero value.
Changed to use Updates() which only updates non-zero values, preserving
existing database values when struct pointer fields are nil.
In BackfillNodeIPs, we need to explicitly update IPv4/IPv6 fields even
when nil (to remove IPs), so we use Select() to specify those fields.
Added regression test that validates expiry is preserved after MapRequest.
Fixes#2862
Skip auth key validation for existing nodes re-registering with the same
NodeKey. Pre-auth keys are only required for initial authentication.
NodeKey rotation still requires a valid auth key as it is a security-sensitive
operation that changes the node's cryptographic identity.
Fixes#2830
When we encounter a source we cannot resolve, we skipped the whole rule,
even if some of the srcs could be resolved. In this case, if we had one user
that exists and one that does not.
In the regular policy, we log this, and still let a rule be created from what
does exist, while in the SSH policy we did not.
This commit fixes it so the behaviour is the same.
Fixes#2863
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
On Windows, if the user clicks the Tailscale icon in the system tray,
it opens a login URL in the browser.
When the login URL is opened, `state/nonce` cookies are set for that particular URL.
If the user clicks the icon again, a new login URL is opened in the browser,
and new cookies are set.
If the user proceeds with auth in the first tab,
the redirect results in a "state did not match" error.
This patch ensures that each opened login URL sets an individual cookie
that remains valid on the `/oidc/callback` page.
`TestOIDCMultipleOpenedLoginUrls` illustrates and tests this behavior.
When we fixed the issue of node visibility of nodes
that only had access to eachother because of a subnet
route, we gave all nodes access to all exit routes by
accident.
This commit splits exit nodes and subnet routes in the
access.
If a matcher indicates that the node should have access to
any part of the subnet routes, we do not remove it from the
node list.
If a matcher destination is equal to the internet, and the
target node is an exit node, we also do not remove the access.
Fixes#2784Fixes#2788
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
There are situations where the subnet routes and exit nodes
must be treated differently. This splits it so SubnetRoutes
only returns routes that are not exit routes.
It adds `IsExitRoutes` and `AllApprovedRoutes` for convenience.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Correctly identify Viper's ConfigFileNotFoundError in LoadConfig to log a warning and use defaults, unifying behavior with empty config files. Fixes fatal error when no config file is present for CLI commands relying on environment variables.
This PR addresses some consistency issues that was introduced or discovered with the nodestore.
nodestore:
Now returns the node that is being put or updated when it is finished. This closes a race condition where when we read it back, we do not necessarily get the node with the given change and it ensures we get all the other updates from that batch write.
auth:
Authentication paths have been unified and simplified. It removes a lot of bad branches and ensures we only do the minimal work.
A comprehensive auth test set has been created so we do not have to run integration tests to validate auth and it has allowed us to generate test cases for all the branches we currently know of.
integration:
added a lot more tooling and checks to validate that nodes reach the expected state when they come up and down. Standardised between the different auth models. A lot of this is to support or detect issues in the changes to nodestore (races) and auth (inconsistencies after login and reaching correct state)
This PR was assisted, particularly tests, by claude code.
- tailscale client gets a new AuthUrl and sets entry in the regcache
- regcache entry expires
- client doesn't know about that
- client always polls followup request а gets error
When user clicks "Login" in the app (after cache expiry), they visit
invalid URL and get "node not found in registration cache". Some clients
on Windows for e.g. can't get a new AuthUrl without restart the app.
To fix that we can issue a new reg id and return user a new valid
AuthUrl.
RegisterNode is refactored to be created with NewRegisterNode() to
autocreate channel and other stuff.
When the node notifier was replaced with batcher, we removed
its closing, but forgot to add the batchers so it was never
stopping node connections and waiting forever.
Fixes#2751
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
The UserInfo endpoint is always queried since 5d8a2c2.
This allows to use all OIDC related features without any extra
configuration on Authelia.
For Keycloak, its sufficient to add the groups mapper to the userinfo
endpoint.
the client will send a lot of fields as `nil` if they have
not changed. NetInfo, which is inside Hostinfo, is one of those
fields and we often would override the whole hostinfo meaning that
we would remove netinfo if it hadnt changed.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Initial work on a nodestore which stores all of the nodes
and their relations in memory with relationship for peers
precalculated.
It is a copy-on-write structure, replacing the "snapshot"
when a change to the structure occurs. It is optimised for reads,
and while batches are not fast, they are grouped together
to do less of the expensive peer calculation if there are many
changes rapidly.
Writes will block until commited, while reads are never
blocked.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Before this patch, we would send a message to each "node stream"
that there is an update that needs to be turned into a mapresponse
and sent to a node.
Producing the mapresponse is a "costly" afair which means that while
a node was producing one, it might start blocking and creating full
queues from the poller and all the way up to where updates where sent.
This could cause updates to time out and being dropped as a bad node
going away or spending too time processing would cause all the other
nodes to not get any updates.
In addition, it contributed to "uncontrolled parallel processing" by
potentially doing too many expensive operations at the same time:
Each node stream is essentially a channel, meaning that if you have 30
nodes, we will try to process 30 map requests at the same time. If you
have 8 cpu cores, that will saturate all the cores immediately and cause
a lot of wasted switching between the processing.
Now, all the maps are processed by workers in the mapper, and the number
of workers are controlable. These would now be recommended to be a bit
less than number of CPU cores, allowing us to process them as fast as we
can, and then send them to the poll.
When the poll recieved the map, it is only responsible for taking it and
sending it to the node.
This might not directly improve the performance of Headscale, but it will
likely make the performance a lot more consistent. And I would argue the
design is a lot easier to reason about.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
- Fix directory hierarchy flattening by using full paths instead of filepath.Base()
- Remove redundant container hostname prefixes from directory names
- Strip top-level directory from tar extraction to avoid nested structure
- Ensure parent directories exist before creating files
- Results in clean structure: control_logs/mapresponses/1-ts-client/file.json
Share/Contribute Headscale Zabbix Monitoring scripts and templates.
Thank you for the awesome application to everyone involved in Headscale's development!
Previously, nil regions were not properly handled. This change allows users to disable regions in DERPMaps.
Particularly useful to disable some official regions.
This patch includes some changes to the OIDC integration in particular:
- Make sure that userinfo claims are queried *before* comparing the
user with the configured allowed groups, email and email domain.
- Update user with group claim from the userinfo endpoint which is
required for allowed groups to work correctly. This is essentially a
continuation of #2545.
- Let userinfo claims take precedence over id token claims.
With these changes I have verified that Headscale works as expected
together with Authelia without the documented escape hatch [0], i.e.
everything works even if the id token only contain the iss and sub
claims.
[0]: https://www.authelia.com/integration/openid-connect/headscale/#configuration-escape-hatch
There was a bug in HA subnet router handover where we used stale node data
from the longpoll session that we handed to Connect. This meant that we got
some odd behaviour where routes would not be deactivated correctly.
This commit changes to the nodeview is used through out, and we load the
current node to be updated in the write path and then handle it all there
to be consistent.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit changes most of our (*)types.Node to
types.NodeView, which is a readonly version of the
underlying node ensuring that there is no mutations
happening in the read path.
Based on the migration, there didnt seem to be any, but the
idea here is to prevent it in the future and simplify other
new implementations.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
We are already being punished by github actions, there seem to be
little value in running all the tests for both databases, so only
run a few key tests to check postgres isnt broken.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Restructure and rewrite the OpenID Connect documentation. Start from the
most minimal configuration and describe what needs to be done both in
Headscale and the identity provider. Describe additional features such
as PKCE and authorization filters in a generic manner with examples.
Document how Headscale populates its user profile and how it relates to
OIDC claims. This is a revised version from the table in the changelog.
Document the validation rules for fields and extend known limitations.
Sort the provider specific section alphabetically and add a section for
Authelia, Authentik, Kanidm and Keycloak. Also simplify and rename Azure
to Entra ID.
Update the description for the oidc section in the example
configuration. Give a short explanation of each configuration setting.
All documentend features were tested with Headscale 0.26 (using a fresh
database each time) using the following identity providers:
* Authelia
* Authentik
* Kanidm
* Keycloak
Fixes: #2295
this commit moves all of the read and write logic, and all different parts
of headscale that manages some sort of persistent and in memory state into
a separate package.
The goal of this is to clearly define the boundry between parts of the app
which accesses and modifies data, and where it happens. Previously, different
state (routes, policy, db and so on) was used directly, and sometime passed to
functions as pointers.
Now all access has to go through state. In the initial implementation,
most of the same functions exists and have just been moved. In the future
centralising this will allow us to optimise bottle necks with the database
(in memory state) and make the different parts talking to eachother do so
in the same way across headscale components.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* cmd/hi: add integration test runner CLI tool
Add a new CLI tool 'hi' for running headscale integration tests
with Docker automation. The tool replaces manual Docker command
composition with an automated solution.
Features:
- Run integration tests in golang:1.24 containers
- Docker context detection (supports colima and other contexts)
- Test isolation with unique run IDs and isolated control_logs
- Automatic Docker image pulling and container management
- Comprehensive cleanup operations for containers, networks, images
- Docker volume caching for Go modules
- Verbose logging and detailed test artifact reporting
- Support for PostgreSQL/SQLite selection and various test flags
Usage: go run ./cmd/hi run TestPingAllByIP --verbose
The tool uses creachadair/command and flax for CLI parsing and
provides cleanup subcommands for Docker resource management.
Updates flake.nix vendorHash for new Go dependencies.
* ci: update integration tests to use hi CLI tool
Replace manual Docker command composition in GitHub Actions
workflow with the new hi CLI tool for running integration tests.
Changes:
- Replace complex docker run command with simple 'go run ./cmd/hi run'
- Remove manual environment variable setup (handled by hi tool)
- Update artifact paths for new timestamped log directory structure
- Simplify command from 15+ lines to 3 lines
- Maintain all existing functionality (postgres/sqlite, timeout, test patterns)
The hi tool automatically handles Docker context detection, container
management, volume mounting, and environment variable setup that was
previously done manually in the workflow.
* makefile: remove test integration
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* feat: add verify client config for embedded DERP
* refactor: embedded DERP no longer verify clients via HTTP
- register the `headscale://` protocol in `http.DefaultTransport` to intercept network requests
- update configuration to use a single boolean option `verify_clients`
* refactor: use `http.HandlerFunc` for type definition
* refactor: some renaming and restructuring
* chore: some renaming and fix lint
* test: fix TestDERPVerifyEndpoint
- `tailscale debug derp` use random node private key
* test: add verify clients integration test for embedded DERP server
* fix: apply code review suggestions
* chore: merge upstream changes
* fix: apply code review suggestions
---------
Co-authored-by: Kristoffer Dalby <kristoffer@dalby.cc>
* Improve map auth logic
* Bugfix
* Add comment, improve error message
* noise: make func, get by node
this commit splits the additional validation into a
separate function so it can be reused if we add more
endpoints in the future.
It swaps the check, so we still look up by NodeKey, but before
accepting the connection, we validate the known machinekey from
the db against the noise connection.
The reason for this is that when a node logs in or out, the node key
is replaced and it will no longer be possible to look it up, breaking
reauthentication.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* noise: add comment to remind future use of getAndVal
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* changelog: add entry
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
The systemd target "syslog.target" and not required because syslog is
socket activated.
The directory /var/run is usually a symlink to /run and its created by
systemd via the RuntimeDirectory=headscale option. System creates and
handles permissions, no need to manually mark it as a read-write path.
Move files for packaging outside the docs directory into its own
packaging directory. Replace the existing postinstall and postremove
scripts with Debian maintainerscripts to behave more like a typical
Debian package:
* Start and enable the headscale systemd service by default
* Does not print informational messages
* No longer stop and disable the service on updates
This package also performs migrations for all changes done in previous
package versions on upgrade:
* Set login shell to /usr/sbin/nologin
* Set home directory to /var/lib/headscale
* Migrate to system UID/GID
The package is lintian-clean with a few exceptions that are documented
as excludes and it passes puipars (both tested on Debian 12).
The following scenarious were tested on Ubuntu 22.04, Ubuntu 24.04,
Debian 11, Debian 12:
* Install
* Install same version again
* Install -> Remove -> Install
* Install -> Purge -> Install
* Purge
* Update from 0.22.0
* Update from 0.26.0
See: #2278
See: #2133Fixes: #2311
This change makes editing the generated command easier.
For example, after pasting into a terminal, the cursor position will be
near the username portion which requires editing.
* policy/v2: allow Username as ssh source
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy/v2: validate that no undefined group or tag is used
Fixes#2570
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy: fixup tests which violated tag constraing
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Link to stable and development docs in the README
* Add Tailscale SSH and autogroup:nonroot to features page
* Use @ when referencing users in policy
* Remove unmaintained headscale-webui
The project seems to be unmaintained (last commit: 2023-05-08) and it
only supports Headscale 0.22 or earlier.
* Use full image URL in container docs
This makes it easy to switch the container runtime from docker <->
podman.
* Remove version from docker-compose.yml example
This is now deprecated and yields a warning.
* Add documentation for routes
* Rename exit-node to routes and add redirects
* Add a new section on subnet routers
* Extend the existing exit-node documentation
* Describe auto approvers for subnet routers and exit nodes
* Provide ACL examples for subnet routers and exit nodes
* Describe HA and its current limitations
* Add a troubleshooting section with IP forwarding
* Update features page for 0.26
Add auto approvers and link to our documentation if available.
* Prefer the console lexer when commandline and output mixed
If we assume someone doesn't already have the required go package, they might also not have the required git package installed either, so pkg_add both of them.
* Make matchers part of the Policy interface
* Prevent race condition between rules and matchers
* Test also matchers in tests for Policy.Filter
* Compute `filterChanged` in v2 policy correctly
* Fix nil vs. empty list issue in v2 policy test
* policy/v2: always clear ssh map
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Aras Ergus <aras.ergus@tngtech.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
* integration: ensure route is set before node joins, reproduce
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* auth: ensure that routes are autoapproved when the node is stored
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Tailscale allows to override the local DNS settings of a node via
"Override local DNS" [1]. Restore this flag with the same config setting
name `dns.override_local_dns` but disable it by default to align it with
Tailscale's default behaviour.
Tested with Tailscale 1.80.2 and systemd-resolved on Debian 12.
With `dns.override_local_dns: false`:
```
Link 12 (tailscale0)
Current Scopes: DNS
Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 100.100.100.100
DNS Domain: tn.example.com ~0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa [snip]
```
With `dns.override_local_dns: true`:
```
Link 12 (tailscale0)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 100.100.100.100
DNS Domain: tn.example.com ~.
```
[1] https://tailscale.com/kb/1054/dns#override-local-dnsFixes: #2256
* ensure final dot on node name
This ensures that nodes which have a base domain set, will have a dot appended to their FQDN.
Resolves: https://github.com/juanfont/headscale/issues/2501
* improve OIDC TTL expire test
Waiting a bit more than the TTL of the OIDC token seems to remove some flakiness of this test. This furthermore makes use of a go func safe buffer which should avoid race conditions.
* Only read relevant nodes from database in PeerChangedResponse
* Rework to ensure transactional consistency in PeerChangedResponse again
* An empty nodeIDs list should return an empty nodes list
* Add test to ListNodesSubset
* Link PR in CHANGELOG.md
* combine ListNodes and ListNodesSubset into one function
* query for all nodes in ListNodes if no parameter is given
* also add optional filtering for relevant nodes to ListPeers
* fix issue auto approve route on register bug
This commit fixes an issue where routes where not approved
on a node during registration. This cause the auto approval
to require the node to readvertise the routes.
Fixes#2497Fixes#2485
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* hsic: only set db policy if exist
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* policy: calculate changed based on policy and filter
v1 is a bit simpler than v2, it does not pre calculate the auto approver map
and we cannot tell if it is changed.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* populate serving from primary routes
Depends on #2464Fixes#2480
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* also exit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix route update outside of connection
there was a bug where routes would not be updated if
they changed while a node was connected and it was not part of an
autoapprove.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update expected test output, cli only shows service node
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Some endpoints in /debug send JSON data as string. Set the Content-Type
header to "application/json" which renders nicely in Firefox.
Mention the /debug route in the example configuration.
* utility iterator for ipset
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* split policy -> policy and v1
This commit split out the common policy logic and policy implementation
into separate packages.
policy contains functions that are independent of the policy implementation,
this typically means logic that works on tailcfg types and generic formats.
In addition, it defines the PolicyManager interface which the v1 implements.
v1 is a subpackage which implements the PolicyManager using the "original"
policy implementation.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use polivyv1 definitions in integration tests
These can be marshalled back into JSON, which the
new format might not be able to.
Also, just dont change it all to JSON strings for now.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* formatter: breaks lines
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove compareprefix, use tsaddr version
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove getacl test, add back autoapprover
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use policy manager tag handling
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* rename display helper for user
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* introduce policy v2 package
policy v2 is built from the ground up to be stricter
and follow the same pattern for all types of resolvers.
TODO introduce
aliass
resolver
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* wire up policyv2 in integration testing
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* split policy v2 tests into seperate workflow to work around github limit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add policy manager output to /debug
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* handle register auth errors
This commit handles register auth errors as the
Tailscale clients expect. It returns the error as
part of a tailcfg.RegisterResponse and not as a
http error.
In addition it fixes a nil pointer panic triggered
by not handling the errors as part of this chain.
Fixes#2434
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This PR switches the homegrown debug endpoint to using tsweb.Debugger, a neat toolkit with batteries included for pprof and friends, and making it easy to add additional debug info:
I've started out by adding a bunch of "introspect" endpoints
image
So users can see the acl, filter, config, derpmap and connected nodes as headscale sees them.
This helps preventing messages being sent with the wrong update type
and payload combination, and it is shorter/neater.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Separate the term "tailnet" from user and be more explicit about
providing a single tailnet.
Also be more explicit about users. Refer to "headscale users" when
mentioning commandline invocations and use the term "local users" when
discussing unix accounts.
Fixes: #2335
* add dedicated http error to propagate to user
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* classify user errors in http handlers
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* move validation of pre auth key out of db
This move separates the logic a bit and allow us to
write specific errors for the caller, in this case the web
layer so we can present the user with the correct error
codes without bleeding web stuff into a generic validate.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* set state and nounce in oidc to prevent csrf
Fixes#2276
* try to fix new postgres issue
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure valid tags is populated on user gets too
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure forced tags are added
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* remove unused envvar in test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* debug log auth/unauth tags in policy man
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* defer shutdown in tags test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add tag test with groups
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add email, display name, picture to create user
Updates #2166
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add ability to set display and email to cli
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add email to test users in integration
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix issue where tags were only assigned to email, not username
Fixes#2300Fixes#2307
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* expand principles to correct login name
and if fix an issue where nodeip principles might not expand to all
relevant IPs instead of taking the first in a prefix.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix ssh unit test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update cli and oauth tests for users with email
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* index by test email
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix last test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Describe both ways to add extra DNS records
* Use "extra" instead of "custom" to align with the configuration file
* Include dns.extra_records_path in the configuration file
* Add -race flag to Makefile and integration tests; fix data race in CreateTailscaleNodesInUser
* Fix data race in ExecuteCommand by using local buffers and mutex
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
* lint
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
---------
Signed-off-by: Dongjun Na <kmu5544616@gmail.com>
* Fix excess error message during writes
Fixes#2290
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* retry filewatcher on removed files
This should handled if files are deleted and added again, and for rename
scenarios.
Fixes#2289
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* test more write and remove in filewatcher
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Upgrade the use of dns.use_username_in_magic_dns or
dns_config.use_username_in_magic_dns to a fatal error and remove the
option from the example configuration and integration tests.
Fixes: #2219
Docker releases a patch release which changed the required permissions to be able to do tun devices in containers, this caused all containers to fail in tests causing us to fail all tests. This fixes it, and adds some tools for debugging in the future.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Setup mike to provide versioned builds of the documentation.
The goal is to have versioned docs for stable releases (0.23.0, 0.24.0)
and development docs that can progress along with the code. This allows
us to tailor docs to the next upcoming version as we no longer need to
care about diversion between rendered docs and the latest release.
Versions:
* development (alias: unstable) on each push to the main branch
* MAJOR.MINOR.PATCH (alias: stable, latest for the newest version)
* for each "final" release tag
* for each push to doc maintenance branches: doc/MAJOR.MINOR.PATCH
The default version should the current stable version. The doc
maintenance branches may be used to update the version specific
documentation when issues arise after a release.
This commit fixes the constraint syntax so it is both valid for
sqlite and postgres.
To validate this, I've added a new postgres testing library and a
helper that will spin up local postgres, setup a db and use it in
the constraints tests. This should also help testing db stuff in
the future.
postgres has been added to the nix dev shell and is now required
for running the unit tests.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit hardens the migration part of the OIDC from
the old username based approach to the new sub based approach
and makes it possible for the operator to opt out entirely.
Fixes#1990
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Use a trailing slash
recommended by mkdocs-material
* Update doc requirements
Let mkdocs-material resolve its imaging dependencies (cairosvg and
pillow) and fix a dependabot warning along the way.
Reference compatible versions by major.minor.
* config: loosen up BaseDomain and ServerURL checks
Requirements [here][1]:
> OK:
> server_url: headscale.com, base: clients.headscale.com
> server_url: headscale.com, base: headscale.net
>
> Not OK:
> server_url: server.headscale.com, base: headscale.com
>
> Essentially we have to prevent the possibility where the headscale
> server has a URL which can also be assigned to a node.
>
> So for the Not OK scenario:
>
> if the server is: server.headscale.com, and a node joins with the name
> server, it will be assigned server.headscale.com and that will break
> the connection for nodes which will now try to connect to that node
> instead of the headscale server.
Fixes#2210
[1]: https://github.com/juanfont/headscale/issues/2210#issuecomment-2488165187
* server_url and base_domain: re-word error message, fix a one-off bug and add a test case for the bug.
* lint
* lint again
* integration testing: add and validate build-time options for tailscale head
* fixup! integration testing: add and validate build-time options for tailscale head
integration testing: comply with linter
* fixup! fixup! integration testing: add and validate build-time options for tailscale head
integration testing: tsic.New must never return nil
* fixup! fixup! fixup! integration testing: add and validate build-time options for tailscale head
* minor fixes
* Document to either use a minimal configuration file or environment
variables to connect with a remote headscale instance.
* Document a workaround specific for headscale 0.23.0.
* Remove reference to ancient headscale version.
* Use `cli.insecure: true` or `HEADSCALE_CLI_INSECURE=1` to skip
certificate verification.
* Style and typo fixes
Ref: #2193
* Remove status from web-ui docs
Rename the title to indicate that there multiple web interfaces
available. Do not track the status of each web interface here as their
status is subject to change over time.
* Add page for third-party tools and scripts
According to 15fc6cd966
the routes `/derp/probe` and `/derp/latency-check` are the same and
different versions of the tailscale client use one or the other
endpoint.
Also handle /derp/latency-check
Fixes: #2211
* #2140 Fixed updating of hostname and givenName when it is updated in HostInfo
* #2140 Added integration tests
* #2140 Fix unit tests
* Changed IsAutomaticNameMode to GivenNameHasBeenChanged. Fixed errors in files according to golangci-lint rules
* Setup mkdocs-redirects
* Restructure existing documentation
* Move client OS support into the documentation
* Move existing Client OS support table into its own documentation page
* Link from README.md to the rendered documentation
* Document minimum Tailscale client version
* Reuse CONTRIBUTING.md" in the documentation
* Include "CONTRIBUTING.md" from the repository root
* Update FAQ and index page and link to the contributing docs
* Add configuration reference
* Add a getting started page and explain the first steps with headscale
* Use the existing "Using headscale" sections and combine them into a
single getting started guide with a little bit more explanation.
* Explain how to get help from the command line client.
* Remove duplicated sections from existing installation guides
* Document requirements and assumptions
* Document packages provided by the community
* Move deb install guide to official releases
* Move manual install guide to official releases
* Move container documentation to setup section
* Move sealos documentation to cloud install page
* Move OpenBSD docs to build from source
* Simplify DNS documentation
* Add sponsor page
* Add releases page
* Add features page
* Add help page
* Add upgrading page
* Adjust mkdocs nav
* Update wording
Use the term headscale for the project, Headscale on the beginning of a
sentence and `headscale` when refering to the CLI.
* Welcome to headscale
* Link to existing documentation in the FAQ
* Remove the goal header and use the text as opener
* Indent code block in OIDC
* Make a few pages linter compatible
Also update ignored files for prettier
* Recommend HTTPS on port 443
Fixes: #2164
* Use hosts in acl documentation
thx @efficacy38 for noticing this
Ref: #1863
* Use mkdocs-macros to set headscale version once
* Changed all the HTML into go using go-elem
Created templates package in ./hscontrol/templates.
Moved the registerWebAPITemplate into the templates package as a function to be called.
Replaced the apple and windows html files with go-elem.
* update flake
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make reauth test compat with tailscale head
tailscale/tailscale@1eaad7d broke our reauth test as it makes the client
retry with https/443 if it reconnects within 2 minutes.
This commit fixes this by running the test as a two part,
- with https, to confirm instant reconnect works
- with http, and a 3 min wait, to check that it work without.
The change is not a general consern as headscale in prod is ran
with https.
Updates #2164
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* sort test for stable order
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
expand user, add claims to user
This commit expands the user table with additional fields that
can be retrieved from OIDC providers (and other places) and
uses this data in various tailscale response objects if it is
available.
This is the beginning of implementing
https://docs.google.com/document/d/1X85PMxIaVWDF6T_UPji3OeeUqVBcGj_uHRM5CI-AwlY/edit
trying to make OIDC more coherant and maintainable in addition
to giving the user a better experience and integration with a
provider.
remove usernames in magic dns, normalisation of emails
this commit removes the option to have usernames as part of MagicDNS
domains and headscale will now align with Tailscale, where there is a
root domain, and the machine name.
In addition, the various normalisation functions for dns names has been
made lighter not caring about username and special character that wont
occur.
Email are no longer normalised as part of the policy processing.
untagle oidc and regcache, use typed cache
This commits stops reusing the registration cache for oidc
purposes and switches the cache to be types and not use any
allowing the removal of a bunch of casting.
try to make reauth/register branches clearer in oidc
Currently there was a function that did a bunch of stuff,
finding the machine key, trying to find the node, reauthing
the node, returning some status, and it was called validate
which was very confusing.
This commit tries to split this into what to do if the node
exists, if it needs to register etc.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Update headscale user creation settings in .deb
Update the headscale user settings to:
- shell = /usr/sbin/nologin
- home-dir = /var/lib/headscale
This syncs the .deb installation behavior with the current Linux docs:
fe68f50328/docs/running-headscale-linux-manual.md (L39-L45)Fixesjuanfont/headscale#2133
* slight refactor to use existing variables.
* Fixup for HOME_DIR var
this commit denormalises the Tags related to a Pre auth key
back onto the preauthkey table and struct as a string list.
There was not really any real normalisation here as we just added
a bunch of duplicate tags with new IDs and preauthkeyIDs, lots of
GORM cermony but no actual advantage.
This work is the start to fixup tags which currently are not working
as they should.
Updates #1369
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Some commands such as `nodes delete` require user interaction and they
fail if `-it` is no supplied to `docker exec`. Use `docker exec -it` in
documentation examples to also make them work in interactive commands.
* handle control protocol through websocket
The necessary behaviour is already in place,
but the wasm build only issued GETs, and the handler was not invoked.
* get DERP-over-websocket working for wasm clients
* Prepare for testing builtin websocket-over-DERP
Still needs some way to assert that clients are connected through websockets,
rather than the TCP hijacking version of DERP.
* integration tests: properly differentiate between DERP transports
* do not touch unrelated code
* linter fixes
* integration testing: unexport common implementation of derp server scenario
* fixup! integration testing: unexport common implementation of derp server scenario
* dockertestutil/logs: remove unhelpful comment
* update changelog
---------
Co-authored-by: Csaba Sarkadi <sarkadicsa@tutanota.de>
* Rename docs/ios-client.md to docs/apple-client.md. Add instructions
for macOS; those are copied from the /apple endpoint and slightly
modified. Fix doc links in the README.
* Move infoboxes for /apple and /windows under the "Goal" section to the
top. Those should be seen by users first as they contain *their*
specific headscale URL.
* Swap order of macOS and iOS to move "Profiles" further down.
* Remove apple configuration profiles
* Remove Tailscale versions hints
* Mention /apple and /windows in the README along with their docs
See: #2096
* move logic for validating node names
this commits moves the generation of "given names" of nodes
into the registration function, and adds validation of renames
to RenameNode using the same logic.
Fixes#2121
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix double arg
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add shutdown that asserts if headscale had panics
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add test case producing 2118 panic
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make stream shutdown if self-node has been removed
Currently we will read the node from database, and since it is
deleted, the id might be set to nil. Keep the node around and
just shutdown, so it is cleanly removed from notifier.
Fixes#2118
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Simplify /windows to the bare minimum. Also remove the
/windows/tailscale.reg endpoint as its generated file is no longer
valid for current Tailscale versions.
* Update and simplify the windows documentation accordingly.
* Add a "Unattended mode" section to the troubleshooting section
explaining how to enable "Unattended mode" in the via the Tailscale
tray icon.
* Add infobox about /windows to the docs
Tested on Windows 10, 22H2 with Tailscale 1.72.0
Replaces: #1995
See: #2096
* replace deprecated golangci-lint output format
CI was producing this kind of messages:
> [config_reader] The output format `github-actions` is deprecated, please use `colored-line-number`
* Actually lint files on CI
* Add support for service reload and sync service file
* Copy the systemd.service file to the manual linux docs and adjust the
path to the headscale binary to match with the previous documentation
blocks. Unfortunately, there seems to be no easy way to include a
file in mkdocs.
* Remove a redundant "deprecation" block. The beginning of the
documentation already states that.
* Add `ExecReload` to the systemd.service file.
Fixes: #2016
* Its called systemd
* Fix link to systemd homepage
* docs/acl: fix path to policy file
* docs/exit-node: fixup for 0.23
* Add newlines between commands to improve readability
* Use nodes instead on name
* Remove query parameter from link to Tailscale docs
* docs/remote-cli: fix formatting
* Indent blocks below line numbers to restore numbering
* Fix minor typos
* docs/reverse-proxy: remove version information
* Websocket support is always required now
* s/see detail/see details
* docs/exit-node: add warning to manual documentation
* Replace the warning section with a warning admonition
* Fix TODO link back to the regular linux documentation
* docs/openbsd: fix typos
* the database is created on-the-fly
* docs/sealos: fix typos
* docs/container: various fixes
* Remove a stray sentence
* Remove "headscale" before serve
* Indent line continuation
* Replace hardcoded 0.22 with <VERSION>
* Fix path in debug image to /ko-app/headscale
Fixes: #1822
aa
* Fix KeyExpiration when a zero time value has a timezone
When a zero time value is loaded from JSON or a DB in a way that
assigns it the local timezone, it does not roudtrip in JSON as a
value for which IsZero returns true. This causes KeyExpiry to be
treated as a far past value instead of a nilish value.
See https://github.com/golang/go/issues/57040
* Fix whitespace
* Ensure that postgresql is used for all tests when env var is set
* Pass through value of HEADSCALE_INTEGRATION_POSTGRES env var
* Add option to set timezone on headscale container
* Add test for registration with auth key in alternate timezone
* validate policy against nodes, error if not valid
this commit aims to improve the feedback of "runtime" policy
errors which would only manifest when the rules are compiled to
filter rules with nodes.
this change will in;
file-based mode load the nodes from the db and try to compile the rules on
start up and return an error if they would not work as intended.
database-based mode prevent a new ACL being written to the database if
it does not compile with the current set of node.
Fixes#2073Fixes#2044
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* ensure stderr can be used in err checks
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* test policy set validation
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add new integration test to ghaction
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add back defer for cli tst
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Requiring someone to write a design doc/contribute to the feature shouldn't be a requirement for raising a feature request as users may lack the skills required to do this.
Code Rabbit is one of these new fancy LLM code review tools. I am skeptical
but we can try it for free and it might provide us with some value to let
people get feedback while waiting for other people.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
this commit changes and streamlines the dns_config into a new
key, dns. It removes a combination of outdates and incompatible
configuration options that made it easy to confuse what headscale
could and could not do, or what to expect from ones configuration.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Fix data race issues in EphemeralGarbageCollector tests
* Add defer for mutex unlock in TestEphemeralGarbageCollectorOrder
* Fix mutex unlock order in closure by updating defer placement
* reformat code
This is mostly an automated change with `make lint`.
I had to manually please golangci-lint in routes_test because of a short
variable name.
* fix start -> strategy which was wrongly corrected by linter
* replace ephemeral deletion logic
this commit replaces the way we remove ephemeral nodes,
currently they are deleted in a loop and we look at last seen
time. This time is now only set when a node disconnects and
there was a bug (#2006) where nodes that had never disconnected
was deleted since they did not have a last seen.
The new logic will start an expiry timer when the node disconnects
and delete the node from the database when the timer is up.
If the node reconnects within the expiry, the timer is cancelled.
Fixes#2006
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use uint64 as authekyid and ptr helper in tests
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add test db helper
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add list ephemeral node func
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* schedule ephemeral nodes for removal on startup
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix gorm query for postgres
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* add godoc
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Added a new function `RegisterMethodToV1Enum()` to Node, converting the internal register method string to the corresponding V1 Enum value. Included corresponding unit test in `node_test.go` to ensure correct conversion for various register methods.
* correctly enable WAL log for sqlite
this commit makes headscale correctly enable write-ahead-log for
sqlite and adds an option to turn it on and off.
WAL is enabled by default and should make sqlite perform a lot better,
even further eliminating the need to use postgres.
It also adds a couple of other useful defaults.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
most of the time we dont even check this error and checking
the string for particular errors is very flake as different
databases (sqlite and psql) use different error messages, and
some users might have it in other languages.
Fixes#1956
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This PR removes the complicated session management introduced in https://github.com/juanfont/headscale/pull/1791 which kept track of the sessions in a map, in addition to the channel already kept track of in the notifier.
Instead of trying to close the mapsession, it will now be replaced by the new one and closed after so all new updates goes to the right place.
The map session serve function is also split into a streaming and a non-streaming version for better readability.
RemoveNode in the notifier will not remove a node if the channel is not matching the one that has been passed (e.g. it has been replaced with a new one).
A new tuning parameter has been added to added to set timeout before the notifier gives up to send an update to a node.
Add a keep alive resetter so we wait with sending keep alives if a node has just received an update.
In addition it adds a bunch of env debug flags that can be set:
- `HEADSCALE_DEBUG_HIGH_CARDINALITY_METRICS`: make certain metrics include per node.id, not recommended to use in prod.
- `HEADSCALE_DEBUG_PROFILING_ENABLED`: activate tracing
- `HEADSCALE_DEBUG_PROFILING_PATH`: where to store traces
- `HEADSCALE_DEBUG_DUMP_CONFIG`: calls `spew.Dump` on the config object startup
- `HEADSCALE_DEBUG_DEADLOCK`: enable go-deadlock to dump goroutines if it looks like a deadlock has occured, enabled in integration tests.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
this directory is unmaintained and not verified, if it should be restored, it should end up
under the community docs effort.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* Add test stage to docs
Add new file with docs tets
Run only in pulls
* set explicit python version
* Revert "set explicit python version"
This reverts commit 4dd7b81f26.
* docs/requirements: update mkdocs-material
---------
Co-authored-by: ohdearaugustin <ohdearaugustin@users.noreply.github.com>
jagottsicher's fork fixed a bug in Windows implementation. While Windows may be not intended as a target platform,
some contributors may prefer it for development.
Also ran go mod tidy, thus two more unnecessary packages are removed from go.sum
This commit restructures the map session in to a struct
holding the state of what is needed during its lifetime.
For streaming sessions, the event loop is structured a
bit differently not hammering the clients with updates
but rather batching them over a short, configurable time
which should significantly improve cpu usage, and potentially
flakyness.
The use of Patch updates has been dialed back a little as
it does not look like its a 100% ready for prime time. Nodes
are now updated with full changes, except for a few things
like online status.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Fixes the issue reported in #1712. In Tailscale SaaS, ephemeral keys can be single-user or reusable. Until now, our ephemerals were only reusable. This PR makes us adhere to the .com behaviour.
A lot of things are breaking in 0.23 so instead of having this
be a long process, just rip of the plaster.
Updates #1758
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* rework docker tags
This commit tries to align the new docker tags with the old schema
A prerelease will end up with the following tags:
- unstable
- v0.23.0-alpha3
- 0.23.0.alpha3
- sha-1234adsfg
A release will end up with:
- latest
- stable
- v0.23.0
- v0.23
- v0
- 0.23.0
- 0.23
- 0
- sha-1234adsfg
All of the builds will also have a `-debug` version.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* update changelog
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* TLS documentation updates
Move "Bring your own certificates" to the top
since the letsencrypt section is now much longer, it seems wrong to
keep such a short section way down at the bottom.
Restructure "Challenge types" into separate sections
Add technical description of letsencrypt renewals
this aims to answer:
- what can be expected in terms of renewals
- what logs can be expected (none)
- how to validate that renewal happened successfully
- the reason for some of the 'acme/autocert' logs, or at least
some best-effort assumptions
* +prettier
* Add test because of issue 1604
* Add peer for routes
* Revert previous change to try different way to add peer
* Add traces
* Remove traces
* Make sure tests have IPPrefix comparator
* Get allowedIps before loop
* Remove comment
* Add composite literals :)
We currently do not have a way to clean up api keys. There may be cases
where users of headscale may generate a lot of api keys and these may
end up accumulating in the database. This commit adds the command to
delete an api key given a prefix.
When Postgres is used as the backing database for headscale,
it does not set a limit on maximum open and idle connections
which leads to hundreds of open connections to the Postgres
server.
This commit introduces the configuration variables to set those
values and also sets default while opening a new postgres connection.
This commits removes the locks used to guard data integrity for the
database and replaces them with Transactions, turns out that SQL had
a way to deal with this all along.
This reduces the complexity we had with multiple locks that might stack
or recurse (database, nofitifer, mapper). All notifications and state
updates are now triggered _after_ a database change.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix#1706 - failover should disregard disabled routes during failover
* fixe tests for failover; all current tests assume routes to be enabled
* add testcase for #1706 - failover to disabled route
* upgrade tailscale
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make Node object use actualy tailscale key types
This commit changes the Node struct to have both a field for strings
to store the keys in the database and a dedicated Key for each type
of key.
The keys are populated and stored with Gorm hooks to ensure the data
is stored in the db.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use key types throughout the code
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* make sure machinekey is concistently used
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use machine key in auth url
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix web register
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* use key type in notifier
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
* fix relogin with webauth
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
---------
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit rearranges the poll handler to immediatly accept
updates and notify its peers and return, not travel down the
function for a bit. This reduces the DB calls and other
holdups that isnt necessary to send a "lite response", a
map response without peers, or accepting an endpoint update.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This field is no longer used, it was used in our old state
"algorithm" to determine if we should send an update.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit changes the internals of the mapper to
track all the changes to peers over its lifetime.
This means that it no longer depends on the database
and this should hopefully help with locks and timing issues.
When the mapper is created, it needs the current list of peers,
the world view, when the polling session was started. Then as
update changes are called, it tracks the changes and generates
responses based on its internal list.
As a side, the types.Machines and types.MachinesP, as well as
types.Machine being passed as a full struct and pointer has been
changed to always be pointers, everywhere.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Previously we did not update the packet filter
when nodes changed, which would cause new nodes
to be missing from packet filters of old nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commits extends the mapper with functions for creating "delta"
MapResponses for different purposes (peer changed, peer removed, derp).
This wires up the new state management with a new StateUpdate struct
letting the poll worker know what kind of update to send to the
connected nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit replaces the timestamp based state system with a new
one that has update channels directly to the connected nodes. It
will send an update to all listening clients via the polling
mechanism.
It introduces a new package notifier, which has a concurrency safe
manager for all our channels to the connected nodes.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
There was a lot of tests that actually threw a lot of errors and that did
not pass all the way because we didnt check everything. This commit should
fix all of these cases.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Also bumps tailscale version to trigger build and fixes a CLI test
that had the wrong capitalisation
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit makes a wrapper function round the normalisation requiring
"stripEmailDomain" which has to be passed in almost all functions of
headscale by loading it from Viper instead.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit allows SSH rules to be assigned to each relevant not and
by doing that allow SSH to be rejected, completing the initial SSH
support.
This commit enables SSH by default and removes the experimental flag.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit renames a bunch of files to try to make it a bit less confusing;
protocol_ is now auth as they contained registration, auth and login/out flow
protocol_.*_poll is now poll.go
api.go and other generic handlers are now handlers.go
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Prior to the code reorg, we would generate rules from the Policy and
store it on the global object. Now we generate it on the fly for each node
and this commit cleans up the old variables to make sure we have no
unexpected side effects.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
The mapper package contains functions related to creating and marshalling
reponses to machines.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This is a massive commit that restructures the code into modules:
db/
All functions related to modifying the Database
types/
All type definitions and methods that can be exclusivly used on
these types without dependencies
policy/
All Policy related code, now without dependencies on the Database.
policy/matcher/
Dedicated code to match machines in a list of FilterRules
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This is step one in detaching the Database layer from Headscale (h). The
ultimate goal is to have all function that does database operations in
its own package, and keep the business logic and writing separate.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
distroless has proven a mantenance burden for us, and it has caused headaches for user when trying to debug issues in the container.
And in 2023, 20MB of extra disk space are neglectible.
This commit simplifies the goreleaser configuration and then adds nfpm
support which allows us to build .deb and .rpm for each of the ARCH we
support.
The deb and rpm packages adds systemd services and users, creates
directories etc and should in general give the user a working
environment. We should be able to remove a lot of the complicated,
PEBCAK inducing documentation after this.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commits adds a test to verify that nodes get updated if a node in
their network expires.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit adds a default OpenID Connect expiry to 180d to align with
Tailscale SaaS (previously infinite or based on token expiry).
In addition, it adds an option use the expiry time from the Token sent
by the OpenID provider. This will typically cause really short expiry
and you should only turn on this option if you know what you are
desiring.
This fixes#1176.
Co-authored-by: Even Holthe <even.holthe@bekk.no>
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Align behaviour of dns_config.restricted_nameservers to tailscale.
Tailscale allows split DNS configuration without requiring global nameservers.
In addition, as per [the docs](https://tailscale.com/kb/1054/dns/#using-dns-settings-in-the-admin-console):
> These nameservers also configure search domains for your devices
This commit aligns headscale to tailscale by:
* honouring dns_config.restricted_nameservers regardless of whether any global resolvers are configured
* adding a search domain for each restricted_nameserver
- Save logs from control(headscale) on every run to tmp
- Upgrade nix-actions
- Cancel builds if new commit is pushed
- Fix a sorting bug in user command test
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
As indicated by bradfitz in https://github.com/juanfont/headscale/issues/804#issuecomment-1399314002,
both routes for the exit node must be enabled at the same time. If a user tries to enable one of the exit node routes,
the other gets activated too.
This commit also reduces the API surface, making private a method that didnt need to be exposed.
The calls to AutoMigrate to other classes that refer to users will
create the table and it will break, it needs to be done before
everything else.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
While this truly breaks the point of the backwards compatible stuff with
protobuf, it does not seem worth it to attempt to glue together a
compatible API.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Currently the most "secret" way to specify the oidc client secret is via
an environment variable `OIDC_CLIENT_SECRET`, which is problematic[1].
Lets allow reading oidc client secret from a file. For extra convenience
the path to the secret will resolve the environment variables.
[1]: https://systemd.io/CREDENTIALS/
This commit sets the Headscale config from env instead of file for
integration tests, the main point is to make sure that when we add per
test config, it properly replaces the config key and not append it or
something similar.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
In TS2021 the MachineKey can be obtained from noiseConn.Peer() - contrary to what I thought before,
where I assumed MachineKey was dropped in TS2021.
By having a ts2021App and hanging from there the TS2021 handlers, we can fetch again the MachineKey.
When using Tailscale v1.34.1, enabling or disabling a route does not
effectively add or remove the route from the node's routing table.
We must restart tailscale on the node to have a netmap update.
Fix this by refreshing last state change so that a netmap diff is sent.
Also do not include secondary routes in allowedIPs, otherwise secondary
routes might be used by nodes instead of the primary route.
Signed-off-by: Fatih Acar <facar@scaleway.com>
Port routes tests to new model
Mark as primary the first instance of subnet + tests
In preparation for subnet failover, mark the initial occurrence of a subnet as the primary one.
This commit makes the initial SSH test a bit simpler:
- Use the same pattern/functions for all clients as other tests
- Only test within _one_ namespace/user to confirm the base case
- Use retry function, same as taildrop, there is some funky going on
there...
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Advertises the SSH capability, and parses the SSH ACLs to pass to the
tailscale client. Doesn’t support ‘autogroup’ ACL functionality.
Co-authored-by: Daniel Brooks <db48x@headline.com>
* Port OIDC integration tests to v2
* Move Tailscale old versions to TS2019 list
* Remove Alpine Linux container
* Updated changelog
* Releases: use flavor to set the tag suffix
* Added more debug messages in OIDC registration
* Added more logging
* Do not strip nodekey prefix on handle expired
* Updated changelog
* Add WithHostnameAsServerURL option func
* Reduce the number of namespaces and use hsic.WithHostnameAsServerURL
* Linting fix
* Fix linting issues
* Wait for ready outside the up goroutine
* Minor change in log message
* Add prefix to env var
* Remove unused env var
Co-authored-by: Juan Font <juan.font@esa.int>
Co-authored-by: Steven Honson <steven@honson.id.au>
Co-authored-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit injects the per-test-generated tls certs into the tailscale
container and makes sure all can ping all. It does not test any of the
DERP isolation yet.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit injects the per-test-generated tls certs into the tailscale
container and makes sure all can ping all. It does not test any of the
DERP isolation yet.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
When using running `tailscale up` in the AuthKey flow process, the tailscale client immediately enters PollMap after registration - avoiding a race condition.
When using the web auth (up -> go to the Control website -> CLI `register`) the client is polling checking if it has been authorized. If we immediately ask for the client IP, as done in CreateHeadscaleEnv() we might have the client in NotReady status.
This method provides a way to wait for the client to be ready.
Signed-off-by: Juan Font Alonso <juanfontalonso@gmail.com>
The retry has no real function as it will just fail on
"container exists" on the old tests and the new test will
just try forever before it eventually fails.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
As indicated by the comment, the default /var/lib/headscale path is not writable in the container. However the sample setting is not following that like `private_key_path`
0.16.0 introduced random suffixes to all machine given names
(DNS hostnames) regardless of collisions within a namespace.
This commit brings Headscale more inline with Tailscale by only
adding a suffix if the hostname will collide within the namespace.
The suffix generation differs from Tailscale.
See https://tailscale.com/kb/1098/machine-names/
Add a configuration flag (default true to preserve current behaviour) to
allow headscale to start without OIDC being able to initialise.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit makes headscale fall back to CLI authentication if oidc
fails to initialised and posts a warning to users.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
OIDC might be configured, but unable to be initialised, this only runs
the oidc cycle if it is actually successfully set up/initialised.
Prep for next commit
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
This commit addresses a potential issue where we allowed unsanitised
content to be passed through a go template without validation.
We now try to unmarshall the incoming node key and fails to render the
template if it is not a valid node key.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
It appears to be causing confusion for users on Discord when copying/pasting from the example here, if Headscale crashes on launch then the container will be removed and logs can't be viewed with `docker logs`.
It appears to be causing confusion for users on Discord when copying/pasting from the example here, if Headscale crashes on launch then the container will be removed and logs can't be viewed with `docker logs`.
Currently we exit the program if the setup does not work, this can cause
is to leave containers and other resources behind since we dont run
TearDown. This change will just fail the test if we cant set up, which
should mean that the TearDown runs aswell.
We currently have a bit of flaky logic which prevents the docker plugin
from cleaning up the containers if the tests or setup fatals or crashes,
this is due to a limitation in the save / passed stats handling.
This change makes it an environment variable which by default ditches
the logs and makes the containers clean up "correctly" in the teardown
method.
This commit makes the setLastStateChangeToNow function take a list of
namespaces instead of a single namespace. If no namespaces is passed,
all namespaces will be updated. This means that the argument acts like a
filter.
Implements #617.
Tailscale has changed the format of their ACLs to use a more firewall-y terms ("users" & "ports" -> "src" & "dst"). They have also started using all-lowercase tags. This PR applies these changes.
Extract LoadConfig from GetHeadscaleConfig, as they are conceptually
different operation, e.g.,
1) you can reload config through LoadConfig and do not get config
2) you can get config without reload config
This commit starts to wire up better signal handling, it starts with
handling shutdown a bit better, using the graceful shutdown for all the
listeners we use.
It also adds the initial switch case for handling config and acl reload,
which is to be implemented.
This commit makes isOutdated validate a nodes necessity to update
against all namespaces, and not just the nodes own namespace (which made
more sense before).
getLastStateChange is now uses the passed namespaces as a filter,
meaning that not requesting any namespace will give you the total last
updated state.
In addition, the sync.Map is exchanged for a variant that uses generics
which allows us to remove some casting logic.
When running in CI, I obtained the following error:
```
Running [/home/runner/golangci-lint-1.41.0-linux-amd64/golangci-lint run --out-format=github-actions --new-from-patch=/tmp/tmp-1795-28vaWZek2jfM/pull.patch --new=false --new-from-rev=] in [] ...
level=error msg="Running error: unknown linters: 'ireturn,maintidx', run 'golangci-lint linters' to see the list of supported linters"
```
Adds knobs to configure three aspects of the OpenID Connect flow:
* Custom scopes to override the default "openid profile email".
* Custom parameters to be added to the Authorize Endpoint request.
* Domain allowlisting for authenticated principals.
* User allowlisting for authenticated principals.
This commit fixes the issue of headscale crashing after sending on a
closed channel by moving the channel close to the sender side, instead
of the creator. closeChanWithLog is also implemented with generics now.
Fixes: https://github.com/juanfont/headscale/issues/342
Signed-off-by: Moritz Poldrack <git@moritz.sh>
- makes the html/template for /register follow the same formatting
as /apple and /windows
- adds a <title> element
- minor change for consistency's sake
This commit adds dockerbuild to flakes.nix:
```
nix build .#headscale-docker
```
This uses the Nix infra to build and _does not_ use Dockerfile.
It currently works on Linux (no macOS)
This commit adds a flake.nix build file, it can be used for three
things:
Build `headscale` from local or straight from git:
nix build
or
nix build github:juanfont/headscale
Run and Build `headscale` from local or straight from git:
nix run
or
nix run github:juanfont/headscale
Set up a development environment including all our tools,
- linters
- protobuf tooling
- compilers
nix develop
Websockets, in which DERP is based, requires a TLS certificate. At the same time,
if we use a certificate it must be valid... otherwise Tailscale wont connect (does not
have an Insecure option). So there is no option to expose insecure here
- registry file /windows/tailscale.reg is generated, filling in the
associated control server URL
- also includes CLI instructions
- fix /apple incorrect template: 'Url' is supposed to be '.URL'
This series of commit will be adding an embedded DERP server (and STUN) to Headscale,
thus making it completely self-contained and not dependant in other infrastructure.
logs looks like the following
```
2022-03-02T20:43:08Z DBG Expanding alias=app-test
2022-03-02T20:43:08Z DBG Expanding alias=kube-test
2022-03-02T20:43:08Z DBG Expanding alias=test
2022-03-02T20:43:08Z WRN No IPs found with the alias test
2022-03-02T20:43:08Z DBG Expanding alias=prod
2022-03-02T20:43:08Z WRN No IPs found with the alias prod
2022-03-02T20:43:08Z DBG Expanding alias=prod
2022-03-02T20:43:08Z WRN No IPs found with the alias prod
```
- `preauthkey`, `authkey`, `pre` are aliases for `preauthkey` command
- `ls`, `show` are aliases for `list` subcommand
- `c`, `new` are aliases for `create` subcommand
- `revoke`, `exp`, `e` are aliases for `expire` subcommand
- `apikey`, `api` are aliases for `apikeys` command
- `ls`, `show` are aliases for `list` subcommand
- `c`, `new` are aliases for `create` subcommand
- `revoke`, `exp`, `e` are aliases for the `expire` subcommand
- `namespace`, `ns`, `user`, `users` are aliases for `namespaces`
command
- `c`, `new` are aliases for the `create` subcommand
- `delete` is an alias for the `destroy` subcommand
- `mv` is an alias for the `rename` subcommand
- `ls`, `show` are aliases for the `list` subcommand
- `node`, `machine`, `machines` are aliases for `nodes` command
- `ls`, `show` aliases for `list` subcommand
- `logout`, `exp`, `e` are aliases for `expire` subcommand
- `del` is an alias for `delete` subcommand
This commit removes the need for datatypes.JSON and makes the code a bit
cleaner by allowing us to use proper types throughout the code when it
comes to hostinfo and other datatypes on the machine object.
This allows us to remove alot of unmarshal/marshal operations and remove
a lot of obsolete error checks.
This following commits will clean away a lot of untyped data and
uneccessary error checks.
This commit removes the two extra caches (oidc, requested time) and uses
the new central registration cache instead. The requested time is
unified into the main machine object and the oidc key is just added to
the same cache, as a string with the state as a key instead of machine
key.
This commit removes the field from the database and does a DB migration
**removing** all unregistered machines from headscale.
This means that from this version, all machines in the database is
considered registered.
current logic is not safe as it will allow an IP that isnt persisted to
the DB to be given out multiple times if machines joins in quick
succession.
This adds a lock around the "get ip" and machine registration and save
to DB so we ensure thiis isnt happning.
Currently this had to be done three places, which is silly, and outlined
in #294.
After some more tests in tailscale I couldn't replicate the behavior
described in there.
When adding a rule, allowing A to talk to B the reverse connection was
instantly added to B to allow communication to B.
The previous assumption was probably wrong.
Using h.ListAllMachines also listed the current machine in the result. It's unnecessary (I don't know if it's harmful).
Breaking the check with the `matchSourceAndDestinationWithRule` broke the tests. We have a specificity with the '*' destination that isn't symetrical.
I need to think of a better way to do this. It too hard to read.
Rewrite some function to get rid of the dependency on Headscale object. This allows us
to write succinct test that are more easy to review and implement.
The improvements of the tests allowed to write the removal of the tagged hosts
from the namespace as specified here: https://tailscale.com/kb/1068/acl-tags/
This call should be done quite at each modification of a server resources like RequestTags.
When a server changes it's tag we should rebuild the ACL rules.
When a server is added to headscale we also should update the ACLRules.
This commit change the default behaviour and remove the notion of namespaces between the hosts. It allows all namespaces to be only filtered by the ACLs. This behavior is closer to tailsnet.
A hidden thing was implied in this document is that each person should have his own namespace.
Hidden information in spicification isn't good.
Thank's @kradalby for pointing it out.
description: Use this agent when you need to execute, analyze, or troubleshoot Headscale integration tests. This includes running specific test scenarios, investigating test failures, interpreting test artifacts, validating end-to-end functionality, or ensuring integration test quality before releases. Examples: <example>Context: User has made changes to the route management code and wants to validate the changes work correctly. user: 'I've updated the route advertisement logic in poll.go. Can you run the relevant integration tests to make sure everything still works?' assistant: 'I'll use the headscale-integration-tester agent to run the subnet routing integration tests and analyze the results.' <commentary>Since the user wants to validate route-related changes with integration tests, use the headscale-integration-tester agent to execute the appropriate tests and analyze results.</commentary></example> <example>Context: A CI pipeline integration test is failing and the user needs help understanding why. user: 'The TestSubnetRouterMultiNetwork test is failing in CI. The logs show some timing issues but I can't figure out what's wrong.' assistant: 'Let me use the headscale-integration-tester agent to analyze the test failure and examine the artifacts.' <commentary>Since this involves analyzing integration test failures and interpreting test artifacts, use the headscale-integration-tester agent to investigate the issue.</commentary></example>
color: green
---
You are a specialist Quality Assurance Engineer with deep expertise in Headscale's integration testing system. You understand the Docker-based test infrastructure, real Tailscale client interactions, and the complex timing considerations involved in end-to-end network testing.
## Integration Test System Overview
The Headscale integration test system uses Docker containers running real Tailscale clients against a Headscale server. Tests validate end-to-end functionality including routing, ACLs, node lifecycle, and network coordination. The system is built around the `hi` (Headscale Integration) test runner in `cmd/hi/`.
## Critical Test Execution Knowledge
### System Requirements and Setup
```bash
# ALWAYS run this first to verify system readiness
go run ./cmd/hi doctor
```
This command verifies:
- Docker installation and daemon status
- Go environment setup
- Required container images availability
- Sufficient disk space (critical - tests generate ~100MB logs per run)
- Network configuration
### Test Execution Patterns
**CRITICAL TIMEOUT REQUIREMENTS**:
- **NEVER use bash `timeout` command** - this can cause test failures and incomplete cleanup
- **ALWAYS use the built-in `--timeout` flag** with generous timeouts (minimum 15 minutes)
- **Increase timeout if tests ever time out** - infrastructure issues require longer timeouts
```bash
# Single test execution (recommended for development)
# ALWAYS use --timeout flag with minimum 15 minutes (900s)
go run ./cmd/hi run "TestSubnetRouterMultiNetwork" --timeout=900s
# Database-heavy tests require PostgreSQL backend and longer timeouts
go run ./cmd/hi run "TestExpireNode" --postgres --timeout=1800s
# Pattern matching for related tests - use longer timeout for multiple tests
go run ./cmd/hi run "TestSubnet*" --timeout=1800s
# Long-running individual tests need extended timeouts
go run ./cmd/hi run "TestNodeOnlineStatus" --timeout=2100s # Runs for 12+ minutes
# Full test suite (CI/validation only) - very long timeout required
- **Slow tests** (5+ min): Node expiration, HA failover
- **Long-running tests** (10+ min): `TestNodeOnlineStatus` runs for 12 minutes
**CONCURRENT EXECUTION**: Multiple tests CAN run simultaneously. Each test run gets a unique Run ID for isolation. See "Concurrent Execution and Run ID Isolation" section below.
## Test Artifacts and Log Analysis
### Artifact Structure
All test runs save comprehensive artifacts to `control_logs/TIMESTAMP-ID/`:
```
control_logs/20250713-213106-iajsux/
├── hs-testname-abc123.stderr.log # Headscale server error logs
├── hs-testname-abc123.stdout.log # Headscale server output logs
├── hs-testname-abc123.db # Database snapshot for post-mortem
3.**MapResponse JSON files**: Protocol-level debugging for network map generation issues
4.**Client status dumps** (`*_status.json`): Network state and peer connectivity information
5.**Database snapshots** (`.db` files): For data consistency and state persistence issues
## Concurrent Execution and Run ID Isolation
### Overview
The integration test system supports running multiple tests concurrently on the same Docker daemon. Each test run is isolated through a unique Run ID that ensures containers, networks, and cleanup operations don't interfere with each other.
### Run ID Format and Usage
Each test run generates a unique Run ID in the format: `YYYYMMDD-HHMMSS-{6-char-hash}`
## Common Failure Patterns and Root Cause Analysis
### CRITICAL MINDSET: Code Issues vs Infrastructure Issues
**⚠️ IMPORTANT**: When tests fail, it is ALMOST ALWAYS a code issue with Headscale, NOT infrastructure problems. Do not immediately blame disk space, Docker issues, or timing unless you have thoroughly investigated the actual error logs first.
### Systematic Debugging Process
1.**Read the actual error message**: Don't assume - read the stderr logs completely
2.**Check Headscale server logs first**: Most issues originate from server-side logic
3.**Verify client connectivity**: Only after ruling out server issues
4.**Check timing patterns**: Use proper `EventuallyWithT` patterns
5.**Infrastructure as last resort**: Only blame infrastructure after code analysis
### Real Failure Patterns
#### 1. Timing Issues (Common but fixable)
```go
// ❌ Wrong: Immediate assertions after async operations
},10*time.Second,100*time.Millisecond,"route should be advertised")
```
**Timeout Guidelines**:
- Route operations: 3-5 seconds
- Node state changes: 5-10 seconds
- Complex scenarios: 10-15 seconds
- Policy recalculation: 5-10 seconds
#### 2. NodeStore Synchronization Issues
Route advertisements must propagate through poll requests (`poll.go:420`). NodeStore updates happen at specific synchronization points after Hostinfo changes.
**Why focus on WHY**: Helps maintainers understand architectural decisions and security requirements.
## EventuallyWithT Pattern for External Calls
### Overview
EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
### External Calls That Must Be Wrapped
The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
- `headscale.ListNodes()` - Queries server state
- `client.Status()` - Gets client network status
- `client.Curl()` - Makes HTTP requests through the network
}, 10*time.Second, 100*time.Millisecond, "route should be approved")
```
## Your Core Responsibilities
1. **Test Execution Strategy**: Execute integration tests with appropriate configurations, understanding when to use `--postgres` and timing requirements for different test categories. Follow phase-based testing approach prioritizing route tests.
- **Why this priority**: Route tests are less infrastructure-sensitive and validate core security logic
2. **Systematic Test Analysis**: When tests fail, systematically examine artifacts starting with Headscale server logs, then client logs, then protocol data. Focus on CODE ISSUES first (99% of cases), not infrastructure. Use real-world failure patterns to guide investigation.
- **Why this approach**: Most failures are logic bugs, not environment issues - efficient debugging saves time
3. **Timing & Synchronization Expertise**: Understand asynchronous Headscale operations, particularly route advertisements, NodeStore synchronization at `poll.go:420`, and policy propagation. Fix timing with `EventuallyWithT` while preserving original test expectations.
- **Why preserve expectations**: Test assertions encode business requirements and security policies
- **Key Pattern**: Apply the EventuallyWithT pattern correctly for all external calls as documented above
4. **Root Cause Analysis**: Distinguish between actual code regressions (route approval logic, HA failover architecture), timing issues requiring `EventuallyWithT` patterns, and genuine infrastructure problems (DNS, Docker, container issues).
- **Why this distinction matters**: Different problem types require completely different solution approaches
- **EventuallyWithT Issues**: Often manifest as flaky tests or immediate assertion failures after async operations
5.**Security-Aware Quality Validation**: Ensure tests properly validate end-to-end functionality with realistic timing expectations and proper error handling. Never suggest security bypasses or test expectation changes. Add comprehensive godoc when you understand test business logic.
- **Why security focus**: Integration tests are the last line of defense against security regressions
- **EventuallyWithT Usage**: Proper use prevents race conditions without weakening security assertions
6.**Concurrent Execution Awareness**: Respect run ID isolation and never interfere with other agents' test sessions. Each test run has a unique run ID - only clean up YOUR containers (by run ID label), never perform global cleanup while tests may be running.
- **Why this matters**: Multiple agents/users may run tests concurrently on the same Docker daemon
- **Key Rule**: NEVER use global container cleanup commands - the test runner handles cleanup automatically per run ID
**CRITICAL PRINCIPLE**: Test expectations are sacred contracts that define correct system behavior. When tests fail, fix the code to match the test, never change the test to match broken code. Only timing and observability improvements are allowed - business logic expectations are immutable.
**ISOLATION PRINCIPLE**: Each test run is isolated by its unique Run ID. Never interfere with other test sessions. The system handles cleanup automatically - manual global cleanup commands are forbidden when other tests may be running.
**EventuallyWithT PRINCIPLE**: Every external call to headscale server or tailscale client must be wrapped in EventuallyWithT. Follow the five key rules strictly: one external call per block, proper variable scoping, no nesting, use CollectT for assertions, and provide descriptive messages.
**Remember**: Test failures are usually code issues in Headscale that need to be fixed, not infrastructure problems to be ignored. Use the specific debugging workflows and failure patterns documented above to efficiently identify root causes. Infrastructure issues have very specific signatures - everything else is code-related.
body: 'Nix build failed with wrong gosum, please update "vendorSha256" (${{ steps.build.outputs.OLD_HASH }}) for the "headscale" package in flake.nix with the new SHA: ${{ steps.build.outputs.NEW_HASH }}'
Please follow the steps outlined in the [upgrade guide](https://headscale.net/stable/setup/upgrade/) to update your existing Headscale installation.
**It's best to update from one stable version to the next** (e.g., 0.24.0 → 0.25.1 → 0.26.1) in case you are multiple releases behind. You should always pick the latest available patch release.
Be sure to check the changelog above for version-specific upgrade instructions and breaking changes.
### Backup Your Database
**Always backup your database before upgrading.** Here's how to backup a SQLite database:
name_template:'{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}{{ with .Arm }}v{{ . }}{{ end }}{{ with .Mips }}_{{ . }}{{ end }}{{ if not (eq .Amd64 "v1") }}{{ .Amd64 }}{{ end }}'
formats:
- binary
source:
enabled:true
name_template:"{{ .ProjectName }}_{{ .Version }}"
format:tar.gz
files:
- "vendor/"
nfpms:
# Configure nFPM for .deb and .rpm releases
#
# See https://nfpm.goreleaser.com/configuration/
# and https://goreleaser.com/customization/nfpm/
#
# Useful tools for debugging .debs:
# List file contents: dpkg -c dist/headscale...deb
Headscale is "Open Source, acknowledged contribution", this means that any contribution will have to be discussed with the maintainers before being added to the project.
This model has been chosen to reduce the risk of burnout by limiting the maintenance overhead of reviewing and validating third-party code.
## Why do we have this model?
Headscale has a small maintainer team that tries to balance working on the project, fixing bugs and reviewing contributions.
When we work on issues ourselves, we develop first hand knowledge of the code and it makes it possible for us to maintain and own the code as the project develops.
Code contributions are seen as a positive thing. People enjoy and engage with our project, but it also comes with some challenges; we have to understand the code, we have to understand the feature, we might have to become familiar with external libraries or services and we think about security implications. All those steps are required during the reviewing process. After the code has been merged, the feature has to be maintained. Any changes reliant on external services must be updated and expanded accordingly.
The review and day-1 maintenance adds a significant burden on the maintainers. Often we hope that the contributor will help out, but we found that most of the time, they disappear after their new feature was added.
This means that when someone contributes, we are mostly happy about it, but we do have to run it through a series of checks to establish if we actually can maintain this feature.
## What do we require?
A general description is provided here and an explicit list is provided in our pull request template.
All new features have to start out with a design document, which should be discussed on the issue tracker (not discord). It should include a use case for the feature, how it can be implemented, who will implement it and a plan for maintaining it.
All features have to be end-to-end tested (integration tests) and have good unit test coverage to ensure that they work as expected. This will also ensure that the feature continues to work as expected over time. If a change cannot be tested, a strong case for why this is not possible needs to be presented.
The contributor should help to maintain the feature over time. In case the feature is not maintained probably, the maintainers reserve themselves the right to remove features they redeem as unmaintainable. This should help to improve the quality of the software and keep it in a maintainable state.
## Bug fixes
Headscale is open to code contributions for bug fixes without discussion.
## Documentation
If you find mistakes in the documentation, please submit a fix to the documentation.
An open source, self-hosted implementation of the Tailscale coordination server.
An open source, self-hosted implementation of the Tailscale control server.
Join our [Discord](https://discord.gg/XcQxk2VHjx) server for a chat.
Join our [Discord server](https://discord.gg/c84AZQhmpx) for a chat.
**Note:** Always select the same GitHub tag as the released version you use to ensure you have the correct example configuration and documentation. The `main` branch might contain unreleased changes.
**Note:** Always select the same GitHub tag as the released version you use
to ensure you have the correct example configuration. The `main` branch might
contain unreleased changes. The documentation is available for stable and
development versions:
## Overview
- [Documentation for the stable version](https://headscale.net/stable/)
- [Documentation for the development version](https://headscale.net/development/)
Tailscale is [a modern VPN](https://tailscale.com/) built on top of [Wireguard](https://www.wireguard.com/). It [works like an overlay network](https://tailscale.com/blog/how-tailscale-works/) between the computers of your networks - using [NAT traversal](https://tailscale.com/blog/how-nat-traversal-works/).
## What is Tailscale
Everything in Tailscale is Open Source, except the GUI clients for proprietary OS (Windows and macOS/iOS), and the 'coordination/control server'.
Tailscale is [a modern VPN](https://tailscale.com/) built on top of
[Wireguard](https://www.wireguard.com/).
It [works like an overlay network](https://tailscale.com/blog/how-tailscale-works/)
The control server works as an exchange point of Wireguard public keys for the nodes in the Tailscale network. It also assigns the IP addresses of the clients, creates the boundaries between each user, enables sharing machines between users, and exposes the advertised routes of your nodes.
Everything in Tailscale is Open Source, except the GUI clients for proprietary OS
(Windows and macOS/iOS), and the control server.
headscale implements this coordination server.
The control server works as an exchange point of Wireguard public keys for the
nodes in the Tailscale network. It assigns the IP addresses of the clients,
creates the boundaries between each user, enables sharing machines between users,
and exposes the advertised routes of your nodes.
## Support
A [Tailscale network (tailnet)](https://tailscale.com/kb/1136/tailnet/) is private
network which Tailscale assigns to a user in terms of private users or an
organisation.
If you like `headscale` and find it useful, there is sponsorship and donation buttons available in the repo.
## Design goal
If you would like to sponsor features, bugs or prioritisation, reach out to one of the maintainers.
Headscale aims to implement a self-hosted, open source alternative to the
[Tailscale](https://tailscale.com/) control server. Headscale's goal is to
provide self-hosters and hobbyists with an open-source server they can use for
their projects and labs. It implements a narrow scope, a _single_ Tailscale
network (tailnet), suitable for a personal use, or a small open-source
organisation.
## Status
## Supporting Headscale
- [x] Base functionality (nodes can communicate with each other)
- [x] Node registration through the web flow
- [x] Network changes are relayed to the nodes
- [x] Namespaces support (~tailnets in Tailscale.com naming)
- [x] Routing (advertise & accept, including exit nodes)
- [x] Node registration via pre-auth keys (including reusable keys, and ephemeral node support)
- [x] JSON-formatted output
- [x] ACLs
- [x] Taildrop (File Sharing)
- [x] Support for alternative IP ranges in the tailnets (default Tailscale's 100.64.0.0/10)
- [x] DNS (passing DNS servers to nodes)
- [x] Single-Sign-On (via Open ID Connect)
- [x] Share nodes between namespaces
- [x] MagicDNS (see `docs/`)
If you like `headscale` and find it useful, there is a sponsorship and donation
buttons available in the repo.
## Features
Please see ["Features" in the documentation](https://headscale.net/stable/about/features/).
| macOS | Yes (see `/apple` on your headscale for more information) |
| Windows | Yes [docs](./docs/windows-client.md) |
| Android | [You need to compile the client yourself](https://github.com/juanfont/headscale/issues/58#issuecomment-885255270) |
| iOS | Not yet |
## Roadmap
Suggestions/PRs welcomed!
Please see ["Client and operating system support" in the documentation](https://headscale.net/stable/about/clients/).
## Running headscale
Please have a look at the documentation under [`docs/`](docs/).
**Please note that we do not support nor encourage the use of reverse proxies
and container to run Headscale.**
Please have a look at the [`documentation`](https://headscale.net/stable/).
For NixOS users, a module is available in [`nix/`](./nix/).
## Talks
- Fosdem 2023 (video): [Headscale: How we are using integration testing to reimplement Tailscale](https://fosdem.org/2023/schedule/event/goheadscale/)
- presented by Juan Font Alonso and Kristoffer Dalby
## Disclaimer
1. We have nothing to do with Tailscale, or Tailscale Inc.
2. The purpose of Headscale is maintaining a working, self-hosted Tailscale control panel.
This project is not associated with Tailscale Inc.
However, one of the active maintainers for Headscale [is employed by Tailscale](https://tailscale.com/blog/opensource) and he is allowed to spend work hours contributing to the project. Contributions from this maintainer are reviewed by other maintainers.
The maintainers work together on setting the direction for the project. The underlying principle is to serve the community of self-hosters, enthusiasts and hobbyists - while having a sustainable project.
## Contributing
To contribute to Headscale you would need the lastest version of [Go](https://golang.org) and [Buf](https://buf.build)(Protobuf generator).
Please read the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
### Requirements
To contribute to headscale you would need the latest version of [Go](https://golang.org)
and [Buf](https://buf.build) (Protobuf generator).
We recommend using [Nix](https://nixos.org/) to setup a development environment. This can
be done with `nix develop`, which will install the tools and give you a shell.
This guarantees that you will have the same dev env as `headscale` maintainers.
### Code style
To ensure we have some consistency with a growing number of contributions, this project has adopted linting and style/formatting rules:
To ensure we have some consistency with a growing number of contributions,
this project has adopted linting and style/formatting rules:
The **Go** code is linted with [`golangci-lint`](https://golangci-lint.run) and
formatted with [`golines`](https://github.com/segmentio/golines) (width 88) and
@@ -90,15 +113,18 @@ Check out the `.golangci.yaml` and `Makefile` to see the specific configuration.
- Go
- Buf
- Protobuf tools:
- Protobuf tools
Install and activate:
```shell
make install-protobuf-plugins
nix develop
```
### Testing and building
Some parts of the project require the generation of Go code from Protobuf (if changes are made in `proto/`) and it must be (re-)generated with:
Some parts of the project require the generation of Go code from Protobuf
(if changes are made in `proto/`) and it must be (re-)generated with:
```shell
make generate
@@ -118,169 +144,31 @@ To build the program:
make build
```
### Development workflow
We recommend using Nix for dependency management to ensure you have all required tools. If you prefer to manage dependencies yourself, you can use Make directly:
**With Nix (recommended):**
```shell
nix develop
make test
make build
```
**With your own dependencies:**
```shell
make test
make build
```
The Makefile will warn you if any required tools are missing and suggest running `nix develop`. Run `make help` to see all available targets.
This page provides <a href="https://support.apple.com/guide/mdm/mdm-overview-mdmbf9e668/web">configuration profiles</a> for the official Tailscale clients for <a href="https://apps.apple.com/us/app/tailscale/id1470499037?ls=1">iOS</a> and <a href="https://apps.apple.com/ca/app/tailscale/id1475387142?mt=12">macOS</a>.
</p>
<p>
The profiles will configure Tailscale.app to use {{.Url}} as its control server.
</p>
<h3>Caution</h3>
<p>You should always inspect the profile before installing it:</p>
getPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
policyCmd.AddCommand(getPolicy)
setPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
setPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
policyCmd.AddCommand(setPolicy)
checkPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
// this is only a warning because there could be something sitting in front of headscale that redirects the traffic (e.g. an iptables rule)
log.Warn().
Msg("Warning: when using tls_letsencrypt_hostname with TLS-ALPN-01 as challenge type, headscale must be reachable on port 443, i.e. listen_addr should probably end in :443")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
