komorebi/docs/dev/ci.md

Continuous integration

Status: written together with the workflows, and none of them has run on GitHub: they were checked with actionlint and shellcheck, and every nix attribute they name was evaluated (see "How to verify before merging"). What a hosted runner does (KVM, memory, disk, the macOS image, the Windows ARM image, Cachix) is not known until the first run.

The workflows are in .github/workflows/; the paths that say which platform a change concerns are in .github/ci-paths.yaml; the action that installs nix is .github/actions/nix.

The ladder (ci.yaml, named ci)

Each stage is cheaper than the next and starts only when the one before it passed, so a typo costs two minutes and not an hour. A stage that a change does not concern is skipped, and a skipped job does not stop the stages after it (their conditions start with !cancelled() && !failure()). ci / required needs every job, and passes when each is success or skipped: it is the only check a ruleset needs to require.

stage	jobs	runner	what it covers	when it runs
(gate)	changes	ubuntu-24.04	dorny/paths-filter over .github/ci-paths.yaml: linux, macos, windows	always; a manual run concerns all platforms
0	hygiene	ubuntu-24.04, nix develop .#hygiene	no CRLF; the macros just lint bans (eyre!, dbg!, println! in the Linux and macOS daemons); actionlint; shellcheck of scripts/linux/wsl2-probe.sh and tools/*; tools/kde-closure-check; tools/test-partition check; dependencies.json is what just depgen writes	always
0	fmt	ubuntu-24.04, nix develop .#hygiene	the nightly rustfmt of the flake over every file (tools/format-all --check), all platforms	always
0	deny	ubuntu-24.04, nix develop .#hygiene	cargo deny check over the workspace	always
1	nix-eval	ubuntu-24.04, nix, swap	nix flake check --no-build --all-systems (every output of every system evaluates: the Windows, Linux and macOS packages and checks); builds checkSets.fmt, checkSets.linux-options, checkSets.macos-options (the macOS options and the nix-darwin module, evaluated on Linux), and the Windows e2e Python	always
2	lint (4 legs)	ubuntu-24.04, nix	checkSets.{windows-lint,linux-lint,recorder-lint,nixgen-lint}: clippy and rustdoc, Windows by cross build	always
2	mac-typecheck	ubuntu-24.04, nix	tools/mac-check.sh: the macOS packages type-checked for aarch64-apple-darwin, no link, no run	macOS paths on a pull request; always on a push
3	test (4 legs)	ubuntu-24.04, nix	checkSets.{linux-test,windows-test,recorder-test,nixgen-test}: Linux tests, the Windows test executables cross-built (not run), the recorder's and nixgen's tests	always
3	windows-native-test	windows-latest	cargo nextest of the windows and os_sensitive classes of tools/test-partition	Windows or shared paths
3	windows-native-test-arm	windows-11-arm	the same on ARM	label ci:win-arm; a push to master with Windows paths; by hand
4	discover, vm-fast	ubuntu-24.04, KVM	the fast tier of the KDE VM suites, one machine each (max-parallel: 8, 45 minutes), a machine with no KVM drawn again (_vm.yaml, see KVM)	Linux paths; label ci:vm; every push to master
5	macos-hosted	macos-14 (_macos.yaml)	the macOS checks, the recorder's macOS tests, the binaries, the schemas	label ci:macos on a pull request; macOS paths on a push

Beside the ladder:

workflow	jobs	runner	what it covers	when it runs
macos-selfhosted.yaml	macos (_macos.yaml)	the Mac Mini	the same as macos-hosted, on the machine of the owner	variable MACOS_SELF_HOSTED=true, and: by hand, a push to master, or label ci:macos-self on a pull request from this repository
sweeps.yaml	vm-slow	ubuntu-24.04, KVM	the slow tier of the VM suites, 60 minutes each, logs kept 7 days, a machine with no KVM drawn again (see KVM)	label ci:vm-slow; every push to master; by hand
	nixos-eval, linux-demo	ubuntu-24.04, swap	every NixOS system of the module and of the Plasma guests (about 11.6 GB to evaluate), and the demo guest	the same
	canary (3 legs)	ubuntu-24.04, KVM	every check against the tip of nixpkgs-unstable, one leg for the checks, one for each VM tier; it opens an issue when one fails	Saturday 05:20 UTC; by hand
update-flake-lock.yaml	update	ubuntu-24.04	a pull request that moves flake.lock	Monday 03:00 UTC; by hand
windows-guest.yaml	guest	ubuntu-24.04, KVM	EXPERIMENTAL: builds the Windows guest from the evaluation ISO and runs nix run .#windows-nextest in it	by hand only

ci runs on a pull request (opened, pushed to, reopened, labeled), in the merge queue, on a push to master, and by hand. A pull request cancels its own earlier run; nothing else cancels. The token of every workflow is read-only (contents: read), and no workflow uses pull_request_target: a pull request from a fork gets no secret and no self-hosted runner.

The tools of stage 0 are the latest, and the flake is how CI gets them

Every step of hygiene, fmt and deny runs in nix develop .#hygiene (nix/common.nix), not with the copy of a tool that the runner image or an action ships, so that a local nix develop .#hygiene and CI give the same answer. The policy for those tools is to run the latest release and to change the code to satisfy it, not to pin an older one or to silence a finding: when a mismatch shows up, move the flake forward first. The shell holds the toolchain of rust-toolchain.toml (stable), the nightly rustfmt, cargo-deny, shellcheck, actionlint, jq and ripgrep, and hygiene prints their versions at the start of every run, so a stale pin is visible in the log. When this was written every one of them was the latest upstream release (actionlint 1.7.12, shellcheck 0.11.0, cargo-deny 0.20.2, ripgrep 15.2.0, jq 1.8.2, cargo 1.99.1) and the nightly rustfmt was the one of 2026-11-04.

They stay current in two ways: the weekly update-flake-lock workflow (Mondays) moves every input, and a tool that is behind the release in between is brought forward by hand with nix flake update rust-overlay (the toolchains and the nightly rustfmt) or nix flake update nixpkgs (the rest; this one also moves Plasma, which the VM suites test, so check them after it). Two tools are not in the shell and keep their own versions: taiki-e/install-action installs nextest on the Windows runners, which have no nix, and windows.yaml (the release workflow) runs its own cargo-deny action.

What is and is not covered

	Windows	Linux (KDE)	macOS
format	fmt, checkSets.fmt	the same	the same
clippy	lint (cross build)	lint	macos-hosted only (macos-lint); mac-typecheck type-checks on Linux
unit tests	windows-native-test on Windows, and windows.yaml on every event	test (linux-test)	macos-hosted only
integration	none (the guest is experimental: windows-guest.yaml)	vm-fast, vm-slow	none
options and schemas	linux-options, macos-options on Linux; the Windows schemas are checked by the Windows e2e Python	linux-options	macos-options on Linux; the schemas by macos-hosted

The gaps:

• No Windows e2e: the guest needs the evaluation ISO and about 20 minutes of Windows Setup, and nothing says a hosted runner can do it.
• macOS is run only when asked for (the label, the paths on a push) or on the Mac Mini: hosted macOS minutes cost ten times those of Linux and the macOS image may not have the nix store warm. Until then a macOS break is caught by mac-typecheck (types) and nix-eval (evaluation), not by a test.
• The nightly rustfmt of fmt is the nightly of the day; checkSets.fmt in stage 1 uses the one the flake pins. When only fmt fails, the nightly moved.
• mac-typecheck takes the std of aarch64-apple-darwin from the toolchain of the runner (rustup target add), because the dev shell of the flake only has the Windows target. clippy could run the same way (cargo clippy --target aarch64-apple-darwin); tools/mac-check.sh does not.
• A change that touches no platform path (documentation, README.md) still runs stages 0 to 3, because they are cheap with the cache.

How this lives next to windows.yaml

windows.yaml is the release workflow of the Windows build and is not touched: it keeps running cargo test for every crate of the workspace on every pull request and push to master (about 40 to 50 Windows minutes per event) and builds and publishes the nightly, the MSI and winget. The new workflows do not publish anything, use no write-all, and use their own concurrency groups and artifact names (ci-*).

• needs: does not cross workflows, so windows-native-test waits for lint of ci, and not for the build of windows.yaml. The two run at the same time.
• windows.yaml has no merge_group trigger, so a ruleset cannot require it in a merge queue: it would never start. Require ci / required only.
• The partition makes windows-native-test run fewer tests than windows.yaml (just test-windows-full and windows-nextest-full run all of them by hand). When it has been green for a while, the test step of windows.yaml can be removed by the owner of that file.
• The partition (tools/test-partition.json) puts komorebi-windows in the class windows and komorebi-config, -protocol, -transport and komorebic-core in os_sensitive: those are what windows-native-test runs. komorebi-client, komorebi-bar and komorebi-recorder are mixed, so a platform job names them with -p. The nix set windows-test builds the same subset; the whole suite is the set windows-test-full (not in the default sets, so not in ci) and just test-windows-full.
• Both run cargo-deny; ci / deny is the cheaper copy and stops the ladder first.
• actionlint in hygiene leaves windows.yaml out: it reads the PowerShell of that file as bash. just lint runs it on every workflow and fails on that file.

Repository settings that the owner has to change

None of these can be set from a workflow.

1. A ruleset on master that requires the status check ci / required (and nothing else), with the merge queue on if wanted.
2. Settings, Actions, General: the default token permission is read-only; approval to run workflows from outside collaborators (first-time contributors at least).
3. Already created, nothing to do: the public Cachix cache komorebi and the variable CACHIX_CACHE=komorebi, the secret CACHIX_AUTH_TOKEN (read and write), the secret CACHIX_AUTH_TOKEN_RO and the variable CACHIX_PUBLIC_KEY. The workflows use only CACHIX_CACHE and CACHIX_AUTH_TOKEN: the cache is public, and cachix/cachix-action configures the substituter and its key with no token at all, so a pull request, a fork and the merge queue read the cache without one and CACHIX_AUTH_TOKEN_RO and CACHIX_PUBLIC_KEY are not needed (they would only be for a hand-written extra-substituters). Every job that runs nix goes through .github/actions/nix (the update-flake-lock job too), whose callers pass vars.CACHIX_CACHE and secrets.CACHIX_AUTH_TOKEN (the reusable _macos.yaml gets the secrets with secrets: inherit). The write token is handed to cachix-action (as its authToken input) on a push (master, and the temporary komomono trigger, which no expression names), on workflow_dispatch and on schedule; on a pull_request or in the merge queue it is an empty string and the action logs Pushing is disabled. The only job that reads and never writes is the nixpkgs canary (push: "false"): it builds against another nixpkgs, which nothing else needs. A job that has no CACHIX_CACHE builds with no cache. Still the owner's: the variable MACOS_SELF_HOSTED (true once the Mac Mini has a runner with the labels self-hosted, macOS, ARM64).
4. The labels: ci:macos, ci:macos-self, ci:vm, ci:vm-slow, ci:win-arm.
5. Settings, Actions: allow the actions the workflows use (cachix/*, dorny/paths-filter, taiki-e/install-action, Swatinem/rust-cache, DeterminateSystems/update-flake-lock, EmbarkStudios/cargo-deny-action) if the repository restricts them.

KVM on hosted runners

What GitHub documents: hardware acceleration (KVM) on the Linux runners, with the udev rule that .github/actions/nix writes (changelog of 2024-04-02, and of 2023-02-23 for the first release on larger runners). What it does not document: that every host of the ubuntu-24.04 pool has it. The pages "GitHub-hosted runners" and "Larger runners" say nothing about nested virtualization, and no service level covers it. Nothing in the repository can choose the host: a job is given whichever one the pool has. In run 42182554717 (all 16 hosts the same image, ubuntu24/20261101.360, spread over six Azure regions) 3 of 16 failed the check of /dev/kvm, 13 had it; what is known of those 3 is only that the check failed 42 to 46 ms after the udev trigger, which cannot tell a missing device from a rule that was not applied yet. The step now logs which of the two it is.

What the repository does:

• .github/actions/nix, step "Enable KVM" (the only place that enables it: enable_kvm of cachix/install-nix-action writes the same rule, ignores the outcome, and is turned off): writes the rule, loads kvm_intel or kvm_amd, reloads and triggers udev, waits for udevadm settle, polls /dev/kvm for 30 seconds for read and write access, and as the last resort sets the mode by hand. It prints ls -l /dev/kvm, id, systemd-detect-virt, the count of vmx/svm flags and the loaded kvm modules, before and after.
• _vm.yaml calls _vm-attempt.yaml up to three times. In the first two a host with no KVM does not fail its leg: the leg uploads the artifact kvm-missing-<run attempt>-<attempt>-<suite> and ends green without having run the suite, and the next attempt (a job plan that lists the artifacts) runs those suites again, each on a new machine. In the third, a host with no KVM fails the suite. So a suite is run, or the call fails: three draws without KVM are the only way a suite is not run and the check is red for it.
• vm-fast (ci.yaml) and vm-slow (sweeps.yaml) are calls of _vm.yaml; required reads the result of the call. It works on any branch and any event, because it is inside the one run: it needs no workflow on the default branch and no rerun. (A rerun of the failed jobs of a run cannot be started by the run itself while it is in progress, and a workflow_run workflow runs only from the file on the default branch; that is why the earlier rerun-no-kvm.yaml is gone.)
• Not covered: the three legs of canary and windows-guest.yaml (one machine each, hours long): a host with no KVM fails the step "Enable KVM", with the message this runner has no usable /dev/kvm; run them again by hand.

When GitHub gives no KVM at all for a long stretch (every leg of three draws fails, the log of the step says the device is absent and systemd-detect-virt or the vmx count says why): the check cannot pass on hosted runners. The ways out are a larger runner (runs-on with a label of a larger-runner group: billed per minute also on a public repository, and nothing in the documentation promises nested virtualization there either, only that the Android emulator is supported), or a self-hosted x86-64 Linux machine with KVM, with runs-on changed in _vm-attempt.yaml.

Not verified: none of this has run on GitHub. In particular the artifact listing of plan, the nesting of two reusable workflows and the result of a call whose legs are skipped.

The komomono removal checklist

ci.yaml and macos-selfhosted.yaml run on a push to the temporary branch komomono, so that the stack can be tried before it is merged to master. Each has a line - komomono under the comment # TEMPORARY: remove when komomono is merged to master. When it is merged:

• delete the - komomono line and its comment in .github/workflows/ci.yaml
• delete the same two lines in .github/workflows/macos-selfhosted.yaml
• sweeps.yaml never runs on that branch (its push trigger is master only): nothing to delete there

grep -rn komomono .github finds nothing afterwards. No expression in a workflow names the branch: the conditions say "a push" and "a push to master".

How to verify before merging

Nothing can be run on GitHub from a checkout, so the first run is the test.

1. nix shell nixpkgs#actionlint nixpkgs#shellcheck -c actionlint over the workflows other than windows.yaml.
2. nix eval of every attribute the workflows build, for example nix eval .#legacyPackages.x86_64-linux.checkSets.linux-lint.name and nix eval --json .#legacyPackages.x86_64-linux.vmTiers; nix flake check --no-build --all-systems.
3. Push the stack to a scratch branch, add that branch to the push list of ci.yaml in the scratch branch, and run ci with workflow_dispatch (all platforms). Then: a pull request with no label (stages 0 to 3), one with ci:vm, one with ci:macos, and one with ci:win-arm.
4. Break a check on purpose and see that ci / required fails; skip a stage with a path and see that it passes.
5. Run sweeps and windows-guest by hand once.

Verify the cache after the first push to komomono

Pushes happen in the post step of cachix-action (the daemon registers a post-build hook, so only paths the job builds are pushed, not those substituted from cache.nixos.org; the post step runs when the job fails too).

• the log of the Post Run ./.github/actions/nix step of hygiene shows Cachix: push and Pushing lines (and not Pushing is disabled)
• the Cachix dashboard of komorebi lists new store paths and a size
• a second run of the same commit is faster, and its log shows copying path ... from 'https://komorebi.cachix.org'
• a pull request run of the same jobs shows Pushing is disabled and still copies from komorebi.cachix.org
• the size of the cache against the plan of the account: the xwin Windows SDK (over 1 GB) and the NixOS guest closures (about 10 GB each for Plasma) are pushed unfiltered; if the free tier is too small, a pushFilter on the cachix-action step is the lever, at the price of rebuilding what it excludes

Open items

• Cachix: whether the free tier of the cache komorebi is large enough for the store of this flake (several GB per system) is not verified.
• KVM on hosted runners: documented for public repositories; see KVM below: GitHub does not promise it of every host, so the workflows draw again, up to three times in all.
• The macOS image: macos-14 is Apple silicon with its own nix install each run; whether the cache makes the checks fit the 2 hours of the job is not known.
• The first run of stage 0 sets up the hygiene shell, which is a download of about 1 GB until the Cachix cache has it (a push and a manual run push it).
• The memory of nix flake check --no-build over the nixosConfigurations (about 7 GB) against a runner that has 7 GB (private repositories) or 16 GB (public): the swap of the action is the mitigation.
• windows-native-test-arm assumes the windows-11-arm runner exists for this repository and installs Rust itself when the image has none.