diff --git a/docs/dev/CUTOVER-PLAN.md b/docs/dev/CUTOVER-PLAN.md index f8a24996..59bb1353 100644 --- a/docs/dev/CUTOVER-PLAN.md +++ b/docs/dev/CUTOVER-PLAN.md @@ -617,7 +617,7 @@ jobs: - uses: actions/checkout@v7 - uses: ./.github/actions/nix with: { kvm: "true" } - - run: nix build -L .#checks.x86_64-linux.vm-${{ matrix.suite }} + - run: nix build -L .#legacyPackages.x86_64-linux.vm.${{ matrix.suite }} vm-slow: needs: discover diff --git a/docs/dev/build-and-test.md b/docs/dev/build-and-test.md index 4ea54af0..9847ccaa 100644 --- a/docs/dev/build-and-test.md +++ b/docs/dev/build-and-test.md @@ -63,6 +63,16 @@ nix build -L --no-link .#checks.x86_64-linux.windows-fmt .#checks.x86_64-linux.w nix build -L --no-link .#checks.x86_64-linux.windows-e2e-tests # the Python of the e2e (pytest), no guest needed ``` +The checks are in sets (docs/dev/flake-checks.md): one set alone builds without the others being evaluated. + +```sh +nix build -L --no-link .#checkSets.fmt .#checkSets.deny # a set (a linkFarm of its members); also linux-lint, linux-test, windows-lint, ... +nix build -L --no-link .#checkSets.cheap # what `nix flake check` runs (defaultCheckSets in nix/checks.nix) +nix build -L --no-link .#checkSets.vm-fast # the heavy sets: vm-fast, vm-slow, linux-nixos-eval, linux-demo +nix build -L --no-link .#checkMembers.vm-fast.vm-kwin # one member of any set (what a heavy check is called) +nix flake check --no-build # evaluates the default sets and every other output +``` + Quick loop without nix builds (what `tools/fast-check` runs; clippy of the Windows tree from the nix dev shell): ```sh diff --git a/docs/dev/flake-checks.md b/docs/dev/flake-checks.md new file mode 100644 index 00000000..1affc2e9 --- /dev/null +++ b/docs/dev/flake-checks.md @@ -0,0 +1,191 @@ +# The checks of the flake are separate sets + +Design note for the checks of the one root `flake.nix` (the three platforms). Measurements were made on +the work-in-progress flake of `wip-cutover-flake` (a copy of its working tree, taken 2026-11-01 09:10; the first table ran under a 20 GB cap, its largest peak 10.3 GB, and the later ones under 12 GB), +x86_64-linux, Lix 2.94, `--no-eval-cache`, each evaluation alone in a `systemd-run --user --scope` +with a memory cap; the memory is the cgroup's `memory.peak`, which includes page cache of the store +reads, so it is an upper bound of the evaluator's heap. Numbers marked (inferred) were not run. + +## What the checks are + +At `wip-main` the three flakes had these checks (x86_64-linux): root (Windows) `komorebi-workspace-{clippy, +deny,fmt,nextest,toml-fmt}`, `e2e-tests`, `pre-commit`, `treefmt`; the nested KDE flake the same +`komorebi-workspace-*` plus `doc`, `komorebi-bridge-js`, `keybindings-fixtures`, `nix-options-validation`, +`nixos-module-eval`, `plasma-guests-eval`, `schema-fresh`, `run-kde*`, `wsl2-probe*`, the two effect +install checks and 35 `vm-*` (34 suites + `vm-lib`; 16 of tier `fast`, 18 of tier `slow`); the nested Mac +flake `darwin-module-validation`, `nix-options-validation` and `komorebi-workspace-*`. The root flake +re-exported the KDE and Mac checks as `kfde-*` and `kfm-*` but not the 34 VM suites. The one flake has +every one of them under platform-prefixed names (`windows-*`, `linux-*`, `macos-*`, `vm-*`): 62 checks on +x86_64-linux, 28 on aarch64-darwin, 9 on x86_64-darwin (`nix eval .#checks.`). + +## Why evaluation needed more than 28 GB + +Cost of evaluating `checks.x86_64-linux..drvPath` alone (peak, wall time): + +| check | peak | time | +| --- | --- | --- | +| every Rust check, fmt, deny, bridge-js, schema-fresh, options-validation, keybindings, vm-lib, treefmt | 0.14-0.25 GB | 1-3 s | +| pre-commit, wsl2-probe, effect install checks | 0.4-0.5 GB | 3-4 s | +| linux-run-kde, -doctor, -dry-run (they instantiate the demo guest) | 1.5 GB | 17 s | +| one VM suite (`vm-kwin`, `vm-module`) | 1.0 GB | 12.5 s | +| 2 / 4 / 8 VM suites in one evaluation | 1.6 / 2.7 / 5.2 GB | 29 / 64 / 83 s | +| linux-nixos-module-eval (26 `nixosSystem`s) | 10.3 GB | 138 s | +| linux-plasma-guests-eval (9 guests) | 9.0 GB | 107 s | + +So the memory is linear in the number of NixOS systems an evaluation instantiates, about 0.55 GB per VM +suite and 0.35-0.4 GB per `nixosSystem` of the module check, and nothing is shared between them. A flake +check holds every check's value, so the total is the sum: 34 suites (about 19 GB) + nixos-module-eval +(10 GB) + plasma-guests-eval (9 GB) + the nine `nixosConfigurations`, which `nix flake check` also +evaluates (7 GB when forced alone, measured), about 47 GB +(sum of those figures) against a host with 31 GB. The nested KDE flake needed 15-16 GB for one system because the +same sum was taken before the suites were 34. + +Where the 0.55 GB of one suite goes (each row is an `nix eval` of a stand-alone expression, 4 copies +against 1, the difference divided by 3): + +| what is evaluated | per copy | +| --- | --- | +| `nixosSystem` with Plasma, no test framework | about 0.2 GB | +| `testers.runNixOSTest`, Plasma node, framework's shared pkgs | 0.14 GB | +| the same with `node.pkgsReadOnly = false` and an overlay (as the suites need) | 0.18 GB | +| the same, importing `nixosModules.forVmTests` and enabling `services.komorebi` | 0.40 GB | +| the real suite (its node files, the library, the script) | 0.55 GB | + +The module costs most: its options file (`nix/linux/komorebi-options.nix`, 1351 lines of typed +submodules) is evaluated once per NixOS system, and its overlay (`overlays.linux`, `overlays.forVmTests` +which applies the first as well) ran `mkPkgs` (a new `import nixpkgs` with the rust overlay) and the whole +crane `mkKomorebiPackages` once per node, twice. Doing that once per system (a `genAttrs` of the systems, +shared by every overlay application; tried on a scratch copy, not committed) lowered the 4-suite +stand-alone expression from 2.15 to 1.50 GB, 8 real suites from 5.18 to 4.79 GB and +linux-nixos-module-eval from 10.3 to 8.5 GB. That is worth doing (D-checks-3) and does not change the +picture: 34 + 26 + 9 + 9 full NixOS evaluations cost tens of GB however they are written, because each is a +separate module-system evaluation whose thunks stay reachable from the check attribute. + +`--all-systems` multiplies by the systems that have the checks (three here, each with the Windows and +platform checks; the VM suites exist on the two Linux systems), so it is not the cheap way to see a +problem either. + +## The sets + +A check keeps its name (`linux-clippy`), so no command that names a check changes. A SET is a list of +members, and every set is also one derivation (`pkgs.linkFarm`) that builds its members: + +| set | members (x86_64-linux names) | evaluation | +| --- | --- | --- | +| `fmt` | `treefmt`, `pre-commit`, `windows-fmt`, `windows-toml-fmt`, `linux-fmt`, `linux-toml-fmt` (macOS systems: `macos-fmt`, `macos-toml-fmt`) | cheap | +| `deny` | `windows-deny`, `linux-deny` (`macos-deny`) | cheap | +| `windows-lint` | `windows-clippy` | cheap | +| `windows-test` | `windows-e2e-tests` (the Rust tests of the Windows crates cannot run on a Linux builder; the old `windows-nextest` ran none, see below) | cheap | +| `linux-lint` | `linux-clippy`, `linux-doc` | cheap | +| `linux-test` | `linux-nextest`, `linux-bridge-js`, `linux-keybindings-fixtures`, `vm-lib`, `linux-wsl2-probe`, `linux-wsl2-probe-dry-run`, the two effect install checks | cheap | +| `linux-demo` | `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run` | 1.5 GB | +| `linux-options` | `linux-nix-options-validation`, `linux-schema-fresh` | cheap | +| `linux-nixos-eval` | `linux-nixos-module-eval`, `linux-plasma-guests-eval` | 19 GB | +| `macos-lint`, `macos-test`, `macos-options` | `macos-clippy`; `macos-nextest`; `macos-nix-options-validation`, `macos-darwin-module-validation` | cheap | +| `vm-fast` | the suites whose `tier` is `fast` (16), computed from `suite.tier` | 9 GB | +| `vm-slow` | the suites of tier `slow` (18) | 10 GB | + +The Windows end-to-end run is an app (`nix run .#e2e-windows`), not a check: it needs a golden image. The +sets that exist on a system are the ones with members on it (a Mac has no `windows-*` build host except +Apple silicon's cross build, which the member lists already say). + +Two more sets are made of the others: `cheap` is the sets of `defaultCheckSets` (what `checks` has, as one +derivation), and `all` is every check. A check that is in no set, or in two, fails the evaluation of `checks` +(`nix/checks.nix`), so a new check has to be placed. + +## How they are exposed + +`nix/checks.nix` is the one place that knows the sets. Each platform module only says which derivations it +has, as `allChecks.` (a per-system flake-parts option, an attrset of derivations that merges across the +platform modules; `nix/common.nix` adds treefmt and pre-commit), and `nix/checks.nix` does the rest with plain +functions: + +```nix +# nix/checks.nix (shape) +sets = { fmt = [ "treefmt" ... ]; deny = [ ... ]; ... }; # the names of each set; vm-fast and vm-slow from the tiers +defaultCheckSets = [ "fmt" "deny" "windows-lint" "windows-test" "linux-lint" "linux-test" + "linux-options" "macos-lint" "macos-test" "macos-options" ]; # the one list to edit +checks = members of the sets of defaultCheckSets; +legacyPackages.checkMembers = members; # nix build .#checkMembers.vm-fast.vm-kwin +legacyPackages.checkSets = mapAttrs (set: ms: pkgs.linkFarm "checks-${set}" ...) (members // { cheap; all; }); +``` + +* One set alone: `nix build -L .#checkSets.linux-lint`. Only that set's members are evaluated: sets are + attributes of `legacyPackages`, which `nix flake check` does not walk, and a set that is not named is a + thunk nothing forces. +* One check alone, as now: `nix build -L .#checks.x86_64-linux.linux-clippy` for a check of the default + sets and `.#checkMembers..` for any member (a heavy one is not in `checks`). +* The default composition, `nix flake check` and CI's pull request run: `defaultCheckSets`. Choosing what + `nix flake check` runs is an edit of that list. +* Everything: `nix build -L .#checkSets.all` (cheap sets and heavy ones; the VM suites need KVM and hours). + One set of sets: `nix build .#checkSets.cheap`. +* The existing `legacyPackages..vm`, `vmTiers`, `vmGuests`, `vmTcg` stay (the Mac and CI use them). + +`nixosConfigurations` (the nine Plasma guests) is the other thing a flake check could evaluate: forced alone, +their `toplevel` costs 7.0 GB and 76 s, but `nix flake check` of the whole flake peaks at 4.5 GB, so it does not +force them that deeply and they stay where they are. + +## Alternatives considered + +* A second flake in `nix/checks/flake.nix` with this flake as an input: the evaluation of the checks + would be separate by construction, but `self` becomes an input with a lock entry that has to be updated + (`--override-input` for the local tree) and every check would be a different attribute path than the + one the CI and the docs use. More machinery for what one `defaultCheckSets` list does. +* `--override-input`: it replaces an input, not the set of outputs; it cannot choose checks. +* A `checkSets` function of flake-parts options (the one chosen) versus a templating generator: rejected + by the brief; the sets are attrsets and `linkFarm`. +* Keeping flake-one's `legacyPackages..heavyChecks.`: a single bucket with no way to build + "the fast VM suites" or "the options" alone; the sets replace it and keep its property (nothing under it is + evaluated by `nix flake check`). +* Lowering the cost of one NixOS evaluation instead of not evaluating it: done where cheap (the shared + `mkPkgs`/crane per system), but the cost is the module system's, so it is not a substitute. + +## Measured with the sets (x86_64-linux unless said, one evaluation at a time, 12 GB cap, 31 GB host) + +`nix flake check --no-build` before the sets (the tree of `wip-cutover-flake`, where the heavy checks were +already out of `checks` as `legacyPackages..heavyChecks`) and after: + +| | peak | time | +| --- | --- | --- | +| before, this system | 4.54 GB | 32 s | +| after, this system | 4.54 GB | 33 s | +| before, `--all-systems` | 6.56 GB | 45 s | +| after, `--all-systems` | 6.55 GB | 40 s | + +The sets do not make `nix flake check` cheaper than flake-one's split did (they are the same checks); what +they add is that every heavy group is a set that can be built alone, and the default is one edited list. +The first measurement before the split, killed at 28 GB, was not repeated. + +Evaluating one set alone (`nix eval .#checkSets..drvPath`, peak / time): + +| set | peak | time | +| --- | --- | --- | +| fmt, deny, windows-lint, windows-test, linux-lint, linux-options | 0.19-0.40 GB | 2-4 s | +| linux-test | 0.54 GB | 6 s | +| cheap (the ten default sets; same on aarch64-linux and aarch64-darwin) | 0.55 GB | 10 s | +| linux-demo | 1.46 GB | 20 s | +| vm-fast (16 suites) | 8.6 GB | 109 s | +| vm-slow (18 suites) | 9.7 GB | 127 s | +| linux-nixos-eval (35 NixOS systems) | 11.6 GB | 191 s | +| macos-lint, macos-test, macos-options (aarch64-darwin); `all` on x86_64-darwin | 0.26-0.53 GB | 2-6 s | + +With the per-system sharing of nixpkgs and the crane build (D-checks-3) `linux-nixos-eval` fits in 12 GB +where its two checks took 10.3 + 9.0 GB alone before. Every set of one platform is far below the cap, and no +set was killed. + +Nothing is lost: the check names of every system (`checks` plus what `heavyChecks` had) before, and the union +of `checkMembers` after, are the same sets except `windows-nextest` (below): 61 names on x86_64-linux and +aarch64-linux, 27 on aarch64-darwin, 9 on x86_64-darwin. The `drvPath` of every one of them is the same before and +after, except `treefmt` and `pre-commit`, which hash the whole source tree and so change with any edit of it. + +## `windows-nextest` and `macos-nextest` ran no test + +`cargoNextest` of crane runs the tests in its check phase, and the Windows and macOS `individualCrateArgs` +have `doCheck = false` (Linux passes `doCheck = true`), so the derivation ran `cargo nextest --version` and +nothing else. A Windows program's tests cannot run on a Linux builder, so `windows-nextest` is gone (a check +that runs nothing under the name of one that runs tests is worse than no check) and `windows-clippy +--all-targets` is what compiles them; the Windows CI runs `cargo test`. `macos-nextest` is the same shape and is +kept, unchanged and still named for what it does not do, only until a Mac run decides whether `doCheck = true` +is right there (the tests of `komorebi-macos` run natively on an Apple machine). Open: the tests of the +platform-neutral crates `komorebi-layouts` and `komorebi-scrolling` run in the Windows CI and nowhere in nix +(`linux-nextest` names `-p komorebi -p komorebi-linux -p komorebic -p komorebi-client -p komorebi-hotkeys`). diff --git a/docs/dev/flake.md b/docs/dev/flake.md index 306d7667..bcc4905e 100644 --- a/docs/dev/flake.md +++ b/docs/dev/flake.md @@ -62,7 +62,7 @@ prefixes `kfde-` and `kfm-`) and of the nested flakes themselves. | `kfde-schema-fresh`, `kfde-nix-options-validation`, `kfde-keybindings-fixtures`, `kfde-nixos-module-eval`, `kfde-plasma-guests-eval` | `linux-schema-fresh`, `linux-nix-options-validation`, `linux-keybindings-fixtures`, `linux-nixos-module-eval`, `linux-plasma-guests-eval` | | `kfde-wsl2-probe`, `kfde-wsl2-probe-dry-run`, `kfde-run-kde`, `kfde-run-kde-doctor`, `kfde-run-kde-dry-run` | `linux-wsl2-probe`, `linux-wsl2-probe-dry-run`, `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run` | | `kfde-kde-rounded-corners-komorebi-install`, `kfde-kwin-effect-geometry-change-komorebi-install` | `linux-kde-rounded-corners-komorebi-install`, `linux-kwin-effect-geometry-change-komorebi-install` | -| `vm-NAME`, `vm-lib` (nested KDE flake only; the root flake left them out) | unchanged names, in `legacyPackages..heavyChecks` for now (see Checks); the suites are also `legacyPackages..vm.NAME` | +| `vm-NAME`, `vm-lib` (nested KDE flake only; the root flake left them out) | unchanged names, in `checks` (`vm-lib`) or the sets `vm-fast`, `vm-slow` (see Checks and flake-checks.md); the suites are also `legacyPackages..vm.NAME` | | `kfm-komorebi-workspace-clippy`, `-fmt`, `-toml-fmt`, `-deny`, `-nextest` | `macos-clippy`, `macos-fmt`, `macos-toml-fmt`, `macos-deny`, `macos-nextest` | | `kfm-nix-options-validation`, `kfm-darwin-module-validation` | `macos-nix-options-validation`, `macos-darwin-module-validation` | | `pre-commit`, `treefmt`, `kfde-pre-commit`, `kfde-treefmt`, `kfm-pre-commit`, `kfm-treefmt` | `pre-commit`, `treefmt` (one each, for the whole repository) | @@ -135,13 +135,13 @@ others' derivations. A change in a crate under `crates/` (or in `komorebic`, `ko `macos-darwin-module-validation` - `treefmt` and `pre-commit` for the whole tree -The checks that evaluate a NixOS system or a VM (`vm-lib`, one `vm-NAME` per suite, `linux-nixos-module-eval`, -`linux-plasma-guests-eval`, `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run`) are NOT in `checks`: -they are `legacyPackages..heavyChecks.` (`nix build -L .#legacyPackages.x86_64-linux.heavyChecks.vm-lib`). -Evaluating them takes minutes and more memory than `nix flake check --no-build --all-systems` can use (it was -killed at 28 GB). This is a placeholder until the structure of the check sets is decided. Name the check you want (`nix build -L -.#checks.x86_64-linux.windows-clippy`). `nix flake check --no-build --all-systems` evaluates every output of -every system (42 s) and is the evaluation test of this file. +The checks are in sets (docs/dev/flake-checks.md): `checks.` has the members of the sets of +`defaultCheckSets` in `nix/checks.nix`, the cheap ones. The ones that evaluate NixOS systems (the VM suites, +`linux-nixos-module-eval`, `linux-plasma-guests-eval`, `linux-run-kde*`) are the sets `vm-fast`, `vm-slow`, +`linux-nixos-eval` and `linux-demo`, which `nix flake check` leaves out (evaluating them took more memory than the host +has): `nix build -L .#checkSets.`, or one member, `.#checkMembers..`. A check is named as before +(`nix build -L .#checks.x86_64-linux.windows-clippy`). `nix flake check --no-build --all-systems` evaluates the +default sets and every other output of every system and is the evaluation test of this file. ## Dev shells diff --git a/docs/linux/CONTRIBUTING.md b/docs/linux/CONTRIBUTING.md index 999fae0f..984d1efd 100644 --- a/docs/linux/CONTRIBUTING.md +++ b/docs/linux/CONTRIBUTING.md @@ -68,7 +68,7 @@ with a scenario in [`nix/linux/tests/nixos-module.nix`](../../nix/linux/tests/ni the `nixos-module-eval` check: each scenario is a real NixOS system that is evaluated and never built, and its checks read the assertions, the unit and the options it evaluates to. It runs on a Mac too, with `nix build -.#legacyPackages.aarch64-darwin.heavyChecks.linux-nixos-module-eval`. A change to what the unit does in a +.#legacyPackages.aarch64-darwin.checkMembers.linux-nixos-eval.linux-nixos-module-eval`. A change to what the unit does in a session also changes the VM suite `module`, which logs in to Plasma on a machine with the module and checks the daemon the unit runs. Every other VM suite runs on machines built with the module too. diff --git a/docs/linux/dev/vm-tests.md b/docs/linux/dev/vm-tests.md index b4c09f6d..3ac02bed 100644 --- a/docs/linux/dev/vm-tests.md +++ b/docs/linux/dev/vm-tests.md @@ -49,7 +49,7 @@ Each suite is a directory under [`nix/linux/tests/vm/suites`](../../../nix/linux/tests/vm/suites). The generator in [`nix/linux/tests/vm/default.nix`](../../../nix/linux/tests/vm/default.nix) lists that directory, so a new suite is a new directory and nothing else: it appears as -`checks..vm-NAME` on Linux and as `legacyPackages..vm.NAME` +`legacyPackages..vm.NAME` (and, on Linux, as a member of the set `vm-fast` or `vm-slow` by its tier, `.#checkSets.vm-fast`) everywhere. `legacyPackages..vmTiers` lists the names of each tier as JSON, which is what CI reads. diff --git a/nix/linux/tests/vm/default.nix b/nix/linux/tests/vm/default.nix index 79a8b274..66d0f7a0 100644 --- a/nix/linux/tests/vm/default.nix +++ b/nix/linux/tests/vm/default.nix @@ -1,4 +1,4 @@ -# The NixOS VM tests. Every directory under `suites/` is one test, `checks..vm-` +# The NixOS VM tests. Every directory under `suites/` is one test, `legacyPackages..checkMembers.vm-.vm-` # (Linux) and `legacyPackages..vm.` (everywhere), so a new suite needs no edit to # the flake. See docs/dev/vm-tests.md. #