komorebi/docs/dev/flake-checks.md

The checks of the flake are separate sets

Design note for the checks of the one root flake.nix (the three platforms). Measurements were made on the work-in-progress flake of wip-cutover-flake (a copy of its working tree, taken 2026-11-01 09:10; the first table ran under a 20 GB cap, its largest peak 10.3 GB, and the later ones under 12 GB), x86_64-linux, Lix 2.94, --no-eval-cache, each evaluation alone in a systemd-run --user --scope with a memory cap; the memory is the cgroup's memory.peak, which includes page cache of the store reads, so it is an upper bound of the evaluator's heap. Numbers marked (inferred) were not run.

What the checks are

At wip-main the three flakes had these checks (x86_64-linux): root (Windows) komorebi-workspace-{clippy, deny,fmt,nextest,toml-fmt}, e2e-tests, pre-commit, treefmt; the nested KDE flake the same komorebi-workspace-* plus doc, komorebi-bridge-js, keybindings-fixtures, nix-options-validation, nixos-module-eval, plasma-guests-eval, schema-fresh, run-kde*, wsl2-probe*, the two effect install checks and 35 vm-* (34 suites + vm-lib; 16 of tier fast, 18 of tier slow); the nested Mac flake darwin-module-validation, nix-options-validation and komorebi-workspace-*. The root flake re-exported the KDE and Mac checks as kfde-* and kfm-* but not the 34 VM suites. The one flake has every one of them under platform-prefixed names (windows-*, linux-*, macos-*, vm-*): 62 checks on x86_64-linux, 28 on aarch64-darwin, 9 on x86_64-darwin (nix eval .#checks.<system>).

Why evaluation needed more than 28 GB

Cost of evaluating checks.x86_64-linux.<name>.drvPath alone (peak, wall time):

check	peak	time
every Rust check, fmt, deny, bridge-js, schema-fresh, options-validation, keybindings, vm-lib, treefmt	0.14-0.25 GB	1-3 s
pre-commit, wsl2-probe, effect install checks	0.4-0.5 GB	3-4 s
linux-run-kde, -doctor, -dry-run (they instantiate the demo guest)	1.5 GB	17 s
one VM suite (vm-kwin, vm-module)	1.0 GB	12.5 s
2 / 4 / 8 VM suites in one evaluation	1.6 / 2.7 / 5.2 GB	29 / 64 / 83 s
linux-nixos-module-eval (26 nixosSystems)	10.3 GB	138 s
linux-plasma-guests-eval (9 guests)	9.0 GB	107 s

So the memory is linear in the number of NixOS systems an evaluation instantiates, about 0.55 GB per VM suite and 0.35-0.4 GB per nixosSystem of the module check, and nothing is shared between them. A flake check holds every check's value, so the total is the sum: 34 suites (about 19 GB) + nixos-module-eval (10 GB) + plasma-guests-eval (9 GB) + the nine nixosConfigurations, which nix flake check also evaluates (7 GB when forced alone, measured), about 47 GB (sum of those figures) against a host with 31 GB. The nested KDE flake needed 15-16 GB for one system because the same sum was taken before the suites were 34.

Where the 0.55 GB of one suite goes (each row is an nix eval of a stand-alone expression, 4 copies against 1, the difference divided by 3):

what is evaluated	per copy
nixosSystem with Plasma, no test framework	about 0.2 GB
testers.runNixOSTest, Plasma node, framework's shared pkgs	0.14 GB
the same with node.pkgsReadOnly = false and an overlay (as the suites need)	0.18 GB
the same, importing nixosModules.forVmTests and enabling services.komorebi	0.40 GB
the real suite (its node files, the library, the script)	0.55 GB

The module costs most: its options file (nix/linux/komorebi-options.nix, 1351 lines of typed submodules) is evaluated once per NixOS system, and its overlay (overlays.linux, overlays.forVmTests which applies the first as well) ran mkPkgs (a new import nixpkgs with the rust overlay) and the whole crane mkKomorebiPackages once per node, twice. Doing that once per system (a genAttrs of the systems, shared by every overlay application; tried on a scratch copy, not committed) lowered the 4-suite stand-alone expression from 2.15 to 1.50 GB, 8 real suites from 5.18 to 4.79 GB and linux-nixos-module-eval from 10.3 to 8.5 GB. That is worth doing (D-checks-3) and does not change the picture: 34 + 26 + 9 + 9 full NixOS evaluations cost tens of GB however they are written, because each is a separate module-system evaluation whose thunks stay reachable from the check attribute.

--all-systems multiplies by the systems that have the checks (three here, each with the Windows and platform checks; the VM suites exist on the two Linux systems), so it is not the cheap way to see a problem either.

The sets

A check keeps its name (linux-clippy), so no command that names a check changes. A SET is a list of members, and every set is also one derivation (pkgs.linkFarm) that builds its members:

set	members (x86_64-linux names)	evaluation
fmt	treefmt, pre-commit, windows-fmt, windows-toml-fmt, linux-fmt, linux-toml-fmt (macOS systems: macos-fmt, macos-toml-fmt)	cheap
deny	windows-deny, linux-deny (macos-deny)	cheap
windows-lint	windows-clippy	cheap
windows-test	windows-e2e-tests, windows-nextest-build (the Windows test executables, cross-built and not run; they run in the guest with nix run .#windows-nextest, below)	cheap
linux-lint	linux-clippy, linux-doc	cheap
linux-test	linux-nextest, linux-bridge-js, linux-keybindings-fixtures, vm-lib, linux-wsl2-probe, linux-wsl2-probe-dry-run, the two effect install checks	cheap
linux-demo	linux-run-kde, linux-run-kde-doctor, linux-run-kde-dry-run	1.5 GB
linux-options	linux-nix-options-validation, linux-schema-fresh	cheap
linux-nixos-eval	linux-nixos-module-eval, linux-plasma-guests-eval	19 GB
macos-lint, macos-test, macos-options	macos-clippy; macos-nextest; macos-nix-options-validation, macos-darwin-module-validation	cheap
vm-fast	the suites whose tier is fast (16), computed from suite.tier	9 GB
vm-slow	the suites of tier slow (18)	10 GB

The Windows end-to-end run is an app (nix run .#e2e-windows), and so is windows-nextest, not a check: it needs a golden image. The sets that exist on a system are the ones with members on it (a Mac has no windows-* build host except Apple silicon's cross build, which the member lists already say).

Two more sets are made of the others: cheap is the sets of defaultCheckSets (what checks has, as one derivation), and all is every check. A check that is in no set, or in two, fails the evaluation of checks (nix/checks.nix), so a new check has to be placed.

How they are exposed

nix/checks.nix is the one place that knows the sets. Each platform module only says which derivations it has, as allChecks.<name> (a per-system flake-parts option, an attrset of derivations that merges across the platform modules; nix/common.nix adds treefmt and pre-commit), and nix/checks.nix does the rest with plain functions:

# nix/checks.nix (shape)
sets = { fmt = [ "treefmt" ... ]; deny = [ ... ]; ... };   # the names of each set; vm-fast and vm-slow from the tiers
defaultCheckSets = [ "fmt" "deny" "windows-lint" "windows-test" "linux-lint" "linux-test"
                     "linux-options" "macos-lint" "macos-test" "macos-options" ];   # the one list to edit
checks = members of the sets of defaultCheckSets;
legacyPackages.checkMembers = members;      # nix build .#checkMembers.vm-fast.vm-kwin
legacyPackages.checkSets    = mapAttrs (set: ms: pkgs.linkFarm "checks-${set}" ...) (members // { cheap; all; });

• One set alone: nix build -L .#checkSets.linux-lint. Only that set's members are evaluated: sets are attributes of legacyPackages, which nix flake check does not walk, and a set that is not named is a thunk nothing forces.
• One check alone, as now: nix build -L .#checks.x86_64-linux.linux-clippy for a check of the default sets and .#checkMembers.<set>.<name> for any member (a heavy one is not in checks).
• The default composition, nix flake check and CI's pull request run: defaultCheckSets. Choosing what nix flake check runs is an edit of that list.
• Everything: nix build -L .#checkSets.all (cheap sets and heavy ones; the VM suites need KVM and hours). One set of sets: nix build .#checkSets.cheap.
• The existing legacyPackages.<system>.vm, vmTiers, vmGuests, vmTcg stay (the Mac and CI use them).

nixosConfigurations (the nine Plasma guests) is the other thing a flake check could evaluate: forced alone, their toplevel costs 7.0 GB and 76 s, but nix flake check of the whole flake peaks at 4.5 GB, so it does not force them that deeply and they stay where they are.

Alternatives considered

• A second flake in nix/checks/flake.nix with this flake as an input: the evaluation of the checks would be separate by construction, but self becomes an input with a lock entry that has to be updated (--override-input for the local tree) and every check would be a different attribute path than the one the CI and the docs use. More machinery for what one defaultCheckSets list does.
• --override-input: it replaces an input, not the set of outputs; it cannot choose checks.
• A checkSets function of flake-parts options (the one chosen) versus a templating generator: rejected by the brief; the sets are attrsets and linkFarm.
• Keeping flake-one's legacyPackages.<system>.heavyChecks.<name>: a single bucket with no way to build "the fast VM suites" or "the options" alone; the sets replace it and keep its property (nothing under it is evaluated by nix flake check).
• Lowering the cost of one NixOS evaluation instead of not evaluating it: done where cheap (the shared mkPkgs/crane per system), but the cost is the module system's, so it is not a substitute.

Measured with the sets (x86_64-linux unless said, one evaluation at a time, 12 GB cap, 31 GB host)

nix flake check --no-build before the sets (the tree of wip-cutover-flake, where the heavy checks were already out of checks as legacyPackages.<system>.heavyChecks) and after:

	peak	time
before, this system	4.54 GB	32 s
after, this system	4.54 GB	33 s
before, --all-systems	6.56 GB	45 s
after, --all-systems	6.55 GB	40 s

The sets do not make nix flake check cheaper than flake-one's split did (they are the same checks); what they add is that every heavy group is a set that can be built alone, and the default is one edited list. The first measurement before the split, killed at 28 GB, was not repeated.

Evaluating one set alone (nix eval .#checkSets.<set>.drvPath, peak / time):

set	peak	time
fmt, deny, windows-lint, windows-test, linux-lint, linux-options	0.19-0.40 GB	2-4 s
linux-test	0.54 GB	6 s
cheap (the ten default sets; same on aarch64-linux and aarch64-darwin)	0.55 GB	10 s
linux-demo	1.46 GB	20 s
vm-fast (16 suites)	8.6 GB	109 s
vm-slow (18 suites)	9.7 GB	127 s
linux-nixos-eval (35 NixOS systems)	11.6 GB	191 s
macos-lint, macos-test, macos-options (aarch64-darwin); all on x86_64-darwin	0.26-0.53 GB	2-6 s

With the per-system sharing of nixpkgs and the crane build (D-checks-3) linux-nixos-eval fits in 12 GB where its two checks took 10.3 + 9.0 GB alone before. Every set of one platform is far below the cap, and no set was killed.

Nothing is lost: the check names of every system (checks plus what heavyChecks had) before, and the union of checkMembers after, are the same sets except windows-nextest (below), which is now an app, and the new check windows-nextest-build: that was 61 names on x86_64-linux and aarch64-linux, 27 on aarch64-darwin, 9 on x86_64-darwin, and each system that builds the Windows crates has one more now (not re-counted). The drvPath of every one of them is the same before and after, except treefmt and pre-commit, which hash the whole source tree and so change with any edit of it.

windows-nextest and macos-nextest ran no test

cargoNextest of crane runs the tests in its check phase, and the Windows and macOS individualCrateArgs have doCheck = false (Linux passes doCheck = true), so the derivation ran cargo nextest --version and nothing else. A Windows program's tests cannot run on a Linux builder, so the check windows-nextest was removed (a check that runs nothing under the name of one that runs tests is worse than no check). The tests now run in the Windows e2e guest, which is the answer to "can the Windows nextest run in a VM from NixOS":

• windows-nextest-build (a check, set windows-test) cross-compiles every test executable of the Windows workspace (cargo test --workspace --release --no-run, xwin, 48 executables) and keeps them, so that they are proved to compile and link; about 90 s with warm dependencies.

• windows-nextest (an app, the name of the old check) boots the golden image like a scenario, uploads the executables and the data files they read at the path the build baked in (the schemas, the fixtures), runs every test in a process of its own (as nextest does) and prints a line per executable and the failures; OUT/cargo-tests.json has every result. The guest needs KVM and the golden image, so it is not a check:

  E2E_GOLDEN=golden KOMOREBI_E2E_SCALE=1.5 KOMOREBI_E2E_STATE=state-nextest nix run .#windows-nextest -- OUT
  

  964 tests ran and passed (the first run also saves the guest state, 163 s; after that about 3.5 minutes, of which the tests are 171 s). It is a release build: debug assertions are off, as in linux-nextest.

The neutral crates (komorebi-layouts, komorebi-scrolling, komorebi-themes, the shared crates/*) now also run in linux-nextest (1532 tests with the Linux packages), which needs the schemas of every platform and docs/macos/cli in the Linux source (nix/workspace-src.nix). macos-nextest is the same shape as the old Windows one and is kept, unchanged and still named for what it does not do, only until a Mac run decides whether doCheck = true is right there (the tests of komorebi-macos run natively on an Apple machine).