komorebi/docs/dev/flake-checks.md

# The checks of the flake are separate sets

Design note for the checks of the one root `flake.nix` (the three platforms). Measurements were made on
the work-in-progress flake of `wip-cutover-flake` (a copy of its working tree, taken 2026-11-01 09:10; the first table ran under a 20 GB cap, its largest peak 10.3 GB, and the later ones under 12 GB),
x86_64-linux, Lix 2.94, `--no-eval-cache`, each evaluation alone in a `systemd-run --user --scope`
with a memory cap; the memory is the cgroup's `memory.peak`, which includes page cache of the store
reads, so it is an upper bound of the evaluator's heap. Numbers marked (inferred) were not run.

## What the checks are

At `wip-main` the three flakes had these checks (x86_64-linux): root (Windows) `komorebi-workspace-{clippy,
deny,fmt,nextest,toml-fmt}`, `e2e-tests`, `pre-commit`, `treefmt`; the nested KDE flake the same
`komorebi-workspace-*` plus `doc`, `komorebi-bridge-js`, `keybindings-fixtures`, `nix-options-validation`,
`nixos-module-eval`, `plasma-guests-eval`, `schema-fresh`, `run-kde*`, `wsl2-probe*`, the two effect
install checks and 35 `vm-*` (34 suites + `vm-lib`; 16 of tier `fast`, 18 of tier `slow`); the nested Mac
flake `darwin-module-validation`, `nix-options-validation` and `komorebi-workspace-*`. The root flake
re-exported the KDE and Mac checks as `kfde-*` and `kfm-*` but not the 34 VM suites. The one flake has
every one of them under platform-prefixed names (`windows-*`, `linux-*`, `macos-*`, `vm-*`): 62 checks on
x86_64-linux, 28 on aarch64-darwin, 9 on x86_64-darwin (`nix eval .#checks.<system>`).

## Why evaluation needed more than 28 GB

Cost of evaluating `checks.x86_64-linux.<name>.drvPath` alone (peak, wall time):

| check | peak | time |
| --- | --- | --- |
| every Rust check, fmt, deny, bridge-js, schema-fresh, options-validation, keybindings, vm-lib, treefmt | 0.14-0.25 GB | 1-3 s |
| pre-commit, wsl2-probe, effect install checks | 0.4-0.5 GB | 3-4 s |
| linux-run-kde, -doctor, -dry-run (they instantiate the demo guest) | 1.5 GB | 17 s |
| one VM suite (`vm-kwin`, `vm-module`) | 1.0 GB | 12.5 s |
| 2 / 4 / 8 VM suites in one evaluation | 1.6 / 2.7 / 5.2 GB | 29 / 64 / 83 s |
| linux-nixos-module-eval (26 `nixosSystem`s) | 10.3 GB | 138 s |
| linux-plasma-guests-eval (9 guests) | 9.0 GB | 107 s |

So the memory is linear in the number of NixOS systems an evaluation instantiates, about 0.55 GB per VM
suite and 0.35-0.4 GB per `nixosSystem` of the module check, and nothing is shared between them. A flake
check holds every check's value, so the total is the sum: 34 suites (about 19 GB) + nixos-module-eval
(10 GB) + plasma-guests-eval (9 GB) + the nine `nixosConfigurations`, which `nix flake check` also
evaluates (7 GB when forced alone, measured), about 47 GB
(sum of those figures) against a host with 31 GB. The nested KDE flake needed 15-16 GB for one system because the
same sum was taken before the suites were 34.

Where the 0.55 GB of one suite goes (each row is an `nix eval` of a stand-alone expression, 4 copies
against 1, the difference divided by 3):

| what is evaluated | per copy |
| --- | --- |
| `nixosSystem` with Plasma, no test framework | about 0.2 GB |
| `testers.runNixOSTest`, Plasma node, framework's shared pkgs | 0.14 GB |
| the same with `node.pkgsReadOnly = false` and an overlay (as the suites need) | 0.18 GB |
| the same, importing `nixosModules.forVmTests` and enabling `services.komorebi` | 0.40 GB |
| the real suite (its node files, the library, the script) | 0.55 GB |

The module costs most: its options file (`nix/linux/komorebi-options.nix`, 1351 lines of typed
submodules) is evaluated once per NixOS system, and its overlay (`overlays.linux`, `overlays.forVmTests`
which applies the first as well) ran `mkPkgs` (a new `import nixpkgs` with the rust overlay) and the whole
crane `mkKomorebiPackages` once per node, twice. Doing that once per system (a `genAttrs` of the systems,
shared by every overlay application; tried on a scratch copy, not committed) lowered the 4-suite
stand-alone expression from 2.15 to 1.50 GB, 8 real suites from 5.18 to 4.79 GB and
linux-nixos-module-eval from 10.3 to 8.5 GB. That is worth doing (D-checks-3) and does not change the
picture: 34 + 26 + 9 + 9 full NixOS evaluations cost tens of GB however they are written, because each is a
separate module-system evaluation whose thunks stay reachable from the check attribute.

`--all-systems` multiplies by the systems that have the checks (three here, each with the Windows and
platform checks; the VM suites exist on the two Linux systems), so it is not the cheap way to see a
problem either.

## The sets

A check keeps its name (`linux-clippy`), so no command that names a check changes. A SET is a list of
members, and every set is also one derivation (`pkgs.linkFarm`) that builds its members:

| set | members (x86_64-linux names) | evaluation |
| --- | --- | --- |
| `fmt` | `treefmt`, `pre-commit`, `windows-fmt`, `windows-toml-fmt`, `linux-fmt`, `linux-toml-fmt` (macOS systems: `macos-fmt`, `macos-toml-fmt`) | cheap |
| `deny` | `windows-deny`, `linux-deny` (`macos-deny`) | cheap |
| `windows-lint` | `windows-clippy` | cheap |
| `windows-test` | `windows-e2e-tests` (the Rust tests of the Windows crates cannot run on a Linux builder; the old `windows-nextest` ran none, see below) | cheap |
| `linux-lint` | `linux-clippy`, `linux-doc` | cheap |
| `linux-test` | `linux-nextest`, `linux-bridge-js`, `linux-keybindings-fixtures`, `vm-lib`, `linux-wsl2-probe`, `linux-wsl2-probe-dry-run`, the two effect install checks | cheap |
| `linux-demo` | `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run` | 1.5 GB |
| `linux-options` | `linux-nix-options-validation`, `linux-schema-fresh` | cheap |
| `linux-nixos-eval` | `linux-nixos-module-eval`, `linux-plasma-guests-eval` | 19 GB |
| `macos-lint`, `macos-test`, `macos-options` | `macos-clippy`; `macos-nextest`; `macos-nix-options-validation`, `macos-darwin-module-validation` | cheap |
| `vm-fast` | the suites whose `tier` is `fast` (16), computed from `suite.tier` | 9 GB |
| `vm-slow` | the suites of tier `slow` (18) | 10 GB |

The Windows end-to-end run is an app (`nix run .#e2e-windows`), not a check: it needs a golden image. The
sets that exist on a system are the ones with members on it (a Mac has no `windows-*` build host except
Apple silicon's cross build, which the member lists already say).

Two more sets are made of the others: `cheap` is the sets of `defaultCheckSets` (what `checks` has, as one
derivation), and `all` is every check. A check that is in no set, or in two, fails the evaluation of `checks`
(`nix/checks.nix`), so a new check has to be placed.

## How they are exposed

`nix/checks.nix` is the one place that knows the sets. Each platform module only says which derivations it
has, as `allChecks.<name>` (a per-system flake-parts option, an attrset of derivations that merges across the
platform modules; `nix/common.nix` adds treefmt and pre-commit), and `nix/checks.nix` does the rest with plain
functions:

```nix
# nix/checks.nix (shape)
sets = { fmt = [ "treefmt" ... ]; deny = [ ... ]; ... };   # the names of each set; vm-fast and vm-slow from the tiers
defaultCheckSets = [ "fmt" "deny" "windows-lint" "windows-test" "linux-lint" "linux-test"
                     "linux-options" "macos-lint" "macos-test" "macos-options" ];   # the one list to edit
checks = members of the sets of defaultCheckSets;
legacyPackages.checkMembers = members;      # nix build .#checkMembers.vm-fast.vm-kwin
legacyPackages.checkSets    = mapAttrs (set: ms: pkgs.linkFarm "checks-${set}" ...) (members // { cheap; all; });
```

* One set alone: `nix build -L .#checkSets.linux-lint`. Only that set's members are evaluated: sets are
  attributes of `legacyPackages`, which `nix flake check` does not walk, and a set that is not named is a
  thunk nothing forces.
* One check alone, as now: `nix build -L .#checks.x86_64-linux.linux-clippy` for a check of the default
  sets and `.#checkMembers.<set>.<name>` for any member (a heavy one is not in `checks`).
* The default composition, `nix flake check` and CI's pull request run: `defaultCheckSets`. Choosing what
  `nix flake check` runs is an edit of that list.
* Everything: `nix build -L .#checkSets.all` (cheap sets and heavy ones; the VM suites need KVM and hours).
  One set of sets: `nix build .#checkSets.cheap`.
* The existing `legacyPackages.<system>.vm`, `vmTiers`, `vmGuests`, `vmTcg` stay (the Mac and CI use them).

`nixosConfigurations` (the nine Plasma guests) is the other thing a flake check could evaluate: forced alone,
their `toplevel` costs 7.0 GB and 76 s, but `nix flake check` of the whole flake peaks at 4.5 GB, so it does not
force them that deeply and they stay where they are.

## Alternatives considered

* A second flake in `nix/checks/flake.nix` with this flake as an input: the evaluation of the checks
  would be separate by construction, but `self` becomes an input with a lock entry that has to be updated
  (`--override-input` for the local tree) and every check would be a different attribute path than the
  one the CI and the docs use. More machinery for what one `defaultCheckSets` list does.
* `--override-input`: it replaces an input, not the set of outputs; it cannot choose checks.
* A `checkSets` function of flake-parts options (the one chosen) versus a templating generator: rejected
  by the brief; the sets are attrsets and `linkFarm`.
* Keeping flake-one's `legacyPackages.<system>.heavyChecks.<name>`: a single bucket with no way to build
  "the fast VM suites" or "the options" alone; the sets replace it and keep its property (nothing under it is
  evaluated by `nix flake check`).
* Lowering the cost of one NixOS evaluation instead of not evaluating it: done where cheap (the shared
  `mkPkgs`/crane per system), but the cost is the module system's, so it is not a substitute.

## Measured with the sets (x86_64-linux unless said, one evaluation at a time, 12 GB cap, 31 GB host)

`nix flake check --no-build` before the sets (the tree of `wip-cutover-flake`, where the heavy checks were
already out of `checks` as `legacyPackages.<system>.heavyChecks`) and after:

| | peak | time |
| --- | --- | --- |
| before, this system | 4.54 GB | 32 s |
| after, this system | 4.54 GB | 33 s |
| before, `--all-systems` | 6.56 GB | 45 s |
| after, `--all-systems` | 6.55 GB | 40 s |

The sets do not make `nix flake check` cheaper than flake-one's split did (they are the same checks); what
they add is that every heavy group is a set that can be built alone, and the default is one edited list.
The first measurement before the split, killed at 28 GB, was not repeated.

Evaluating one set alone (`nix eval .#checkSets.<set>.drvPath`, peak / time):

| set | peak | time |
| --- | --- | --- |
| fmt, deny, windows-lint, windows-test, linux-lint, linux-options | 0.19-0.40 GB | 2-4 s |
| linux-test | 0.54 GB | 6 s |
| cheap (the ten default sets; same on aarch64-linux and aarch64-darwin) | 0.55 GB | 10 s |
| linux-demo | 1.46 GB | 20 s |
| vm-fast (16 suites) | 8.6 GB | 109 s |
| vm-slow (18 suites) | 9.7 GB | 127 s |
| linux-nixos-eval (35 NixOS systems) | 11.6 GB | 191 s |
| macos-lint, macos-test, macos-options (aarch64-darwin); `all` on x86_64-darwin | 0.26-0.53 GB | 2-6 s |

With the per-system sharing of nixpkgs and the crane build (D-checks-3) `linux-nixos-eval` fits in 12 GB
where its two checks took 10.3 + 9.0 GB alone before. Every set of one platform is far below the cap, and no
set was killed.

Nothing is lost: the check names of every system (`checks` plus what `heavyChecks` had) before, and the union
of `checkMembers` after, are the same sets except `windows-nextest` (below): 61 names on x86_64-linux and
aarch64-linux, 27 on aarch64-darwin, 9 on x86_64-darwin. The `drvPath` of every one of them is the same before and
after, except `treefmt` and `pre-commit`, which hash the whole source tree and so change with any edit of it.

## `windows-nextest` and `macos-nextest` ran no test

`cargoNextest` of crane runs the tests in its check phase, and the Windows and macOS `individualCrateArgs`
have `doCheck = false` (Linux passes `doCheck = true`), so the derivation ran `cargo nextest --version` and
nothing else. A Windows program's tests cannot run on a Linux builder, so `windows-nextest` is gone (a check
that runs nothing under the name of one that runs tests is worse than no check) and `windows-clippy
--all-targets` is what compiles them; the Windows CI runs `cargo test`. `macos-nextest` is the same shape and is
kept, unchanged and still named for what it does not do, only until a Mac run decides whether `doCheck = true`
is right there (the tests of `komorebi-macos` run natively on an Apple machine). Open: the tests of the
platform-neutral crates `komorebi-layouts` and `komorebi-scrolling` run in the Windows CI and nowhere in nix
(`linux-nextest` names `-p komorebi -p komorebi-linux -p komorebic -p komorebi-client -p komorebi-hotkeys`).