diff --git a/docs/dev/ci.md b/docs/dev/ci.md new file mode 100644 index 00000000..747be2fe --- /dev/null +++ b/docs/dev/ci.md @@ -0,0 +1,133 @@ +# Continuous integration + +Status: written together with the workflows, and none of them has run on GitHub: they were checked with +`actionlint` and `shellcheck`, and every nix attribute they name was evaluated (see "How to verify before merging"). +What a hosted runner does (KVM, memory, disk, the macOS image, the Windows ARM image, Cachix) is not known until +the first run. + +The workflows are in `.github/workflows/`; the paths that say which platform a change concerns are in +`.github/ci-paths.yaml`; the action that installs nix is `.github/actions/nix`. + +## The ladder (`ci.yaml`, named `ci`) + +Each stage is cheaper than the next and starts only when the one before it passed, so a typo costs two minutes +and not an hour. A stage that a change does not concern is skipped, and a skipped job does not stop the stages after +it (their conditions start with `!cancelled() && !failure()`). `ci / required` needs every job, and passes when +each is `success` or `skipped`: it is the only check a ruleset needs to require. + +| stage | jobs | runner | what it covers | when it runs | +|---|---|---|---|---| +| (gate) | `changes` | ubuntu-24.04 | `dorny/paths-filter` over `.github/ci-paths.yaml`: `linux`, `macos`, `windows` | always; a manual run concerns all platforms | +| 0 | `hygiene` | ubuntu-24.04, no nix | no CRLF; the macros `just lint` bans (`eyre!`, `dbg!`, `println!` in the Linux and macOS daemons); actionlint; shellcheck of `scripts/linux/wsl2-probe.sh` and `tools/*`; `tools/kde-closure-check`; `tools/test-partition check`; `dependencies.json` is what `just depgen` writes | always | +| 0 | `fmt` | ubuntu-24.04 | nightly rustfmt over every file (`tools/format-all --check`), all platforms | always | +| 0 | `deny` | ubuntu-24.04 | `cargo deny` over the workspace | always | +| 1 | `nix-eval` | ubuntu-24.04, nix, swap | `nix flake check --no-build --all-systems` (every output of every system evaluates: the Windows, Linux and macOS packages and checks); builds `checkSets.fmt`, `checkSets.linux-options`, `checkSets.macos-options` (the macOS options and the nix-darwin module, evaluated on Linux), and the Windows e2e Python | always | +| 2 | `lint` (4 legs) | ubuntu-24.04, nix | `checkSets.{windows-lint,linux-lint,recorder-lint,nixgen-lint}`: clippy and rustdoc, Windows by cross build | always | +| 2 | `mac-typecheck` | ubuntu-24.04, nix | `tools/mac-check.sh`: the macOS packages type-checked for `aarch64-apple-darwin`, no link, no run | macOS paths on a pull request; always on a push | +| 3 | `test` (4 legs) | ubuntu-24.04, nix | `checkSets.{linux-test,windows-test,recorder-test,nixgen-test}`: Linux tests, the Windows test executables cross-built (not run), the recorder's and nixgen's tests | always | +| 3 | `windows-native-test` | windows-latest | `cargo nextest` of the `windows` and `os_sensitive` classes of `tools/test-partition` | Windows or shared paths | +| 3 | `windows-native-test-arm` | windows-11-arm | the same on ARM | label `ci:win-arm`; a push to master with Windows paths; by hand | +| 4 | `discover`, `vm-fast` | ubuntu-24.04, KVM | the fast tier of the KDE VM suites, one machine each (`max-parallel: 8`, 45 minutes) | Linux paths; label `ci:vm`; every push to master | +| 5 | `macos-hosted` | macos-14 (`_macos.yaml`) | the macOS checks, the recorder's macOS tests, the binaries, the schemas | label `ci:macos` on a pull request; macOS paths on a push | + +Beside the ladder: + +| workflow | jobs | runner | what it covers | when it runs | +|---|---|---|---|---| +| `macos-selfhosted.yaml` | `macos` (`_macos.yaml`) | the Mac Mini | the same as `macos-hosted`, on the machine of the owner | variable `MACOS_SELF_HOSTED=true`, and: by hand, a push to master, or label `ci:macos-self` on a pull request from this repository | +| `sweeps.yaml` | `vm-slow` | ubuntu-24.04, KVM | the slow tier of the VM suites, 60 minutes each, logs kept 7 days | label `ci:vm-slow`; every push to master; by hand | +| | `nixos-eval`, `linux-demo` | ubuntu-24.04, swap | every NixOS system of the module and of the Plasma guests (about 11.6 GB to evaluate), and the demo guest | the same | +| | `canary` (3 legs) | ubuntu-24.04, KVM | every check against the tip of nixpkgs-unstable, one leg for the checks, one for each VM tier; it opens an issue when one fails | Saturday 05:20 UTC; by hand | +| `update-flake-lock.yaml` | `update` | ubuntu-24.04 | a pull request that moves `flake.lock` | Monday 03:00 UTC; by hand | +| `rerun-no-kvm.yaml` | `rerun` | ubuntu-24.04 | runs the failed jobs again when every one failed for want of `/dev/kvm` (three attempts at most) | after `ci` or `sweeps` | +| `windows-guest.yaml` | `guest` | ubuntu-24.04, KVM | EXPERIMENTAL: builds the Windows guest from the evaluation ISO and runs `nix run .#windows-nextest` in it | by hand only | + +`ci` runs on a pull request (opened, pushed to, reopened, labeled), in the merge queue, on a push to `master`, and by hand. A +pull request cancels its own earlier run; nothing else cancels. The token of every workflow is read-only (`contents: read`), and +no workflow uses `pull_request_target`: a pull request from a fork gets no secret and no self-hosted runner. + +## What is and is not covered + +| | Windows | Linux (KDE) | macOS | +|---|---|---|---| +| format | `fmt`, `checkSets.fmt` | the same | the same | +| clippy | `lint` (cross build) | `lint` | `macos-hosted` only (`macos-lint`); `mac-typecheck` type-checks on Linux | +| unit tests | `windows-native-test` on Windows, and `windows.yaml` on every event | `test` (`linux-test`) | `macos-hosted` only | +| integration | none (the guest is experimental: `windows-guest.yaml`) | `vm-fast`, `vm-slow` | none | +| options and schemas | `linux-options`, `macos-options` on Linux; the Windows schemas are checked by the Windows e2e Python | `linux-options` | `macos-options` on Linux; the schemas by `macos-hosted` | + +The gaps: + +- No Windows e2e: the guest needs the evaluation ISO and about 20 minutes of Windows Setup, and nothing says a hosted runner can do it. +- macOS is run only when asked for (the label, the paths on a push) or on the Mac Mini: hosted macOS minutes cost ten times those of Linux and the + macOS image may not have the nix store warm. Until then a macOS break is caught by `mac-typecheck` (types) and `nix-eval` (evaluation), not by a test. +- The nightly rustfmt of `fmt` is the nightly of the day; `checkSets.fmt` in stage 1 uses the one the flake pins. When only `fmt` fails, the nightly moved. +- `mac-typecheck` takes the std of `aarch64-apple-darwin` from the toolchain of the runner (`rustup target add`), because the dev shell of the + flake only has the Windows target. clippy could run the same way (`cargo clippy --target aarch64-apple-darwin`); `tools/mac-check.sh` does not. +- A change that touches no platform path (documentation, `README.md`) still runs stages 0 to 3, because they are cheap with the cache. + +## How this lives next to `windows.yaml` + +`windows.yaml` is the release workflow of the Windows build and is not touched: it keeps running `cargo test` for every crate of the workspace on +every pull request and push to master (about 40 to 50 Windows minutes per event) and builds and publishes the nightly, the MSI and winget. The new +workflows do not publish anything, use no `write-all`, and use their own concurrency groups and artifact names (`ci-*`). + +- `needs:` does not cross workflows, so `windows-native-test` waits for `lint` of `ci`, and not for the build of `windows.yaml`. The two run at the same time. +- `windows.yaml` has no `merge_group` trigger, so a ruleset cannot require it in a merge queue: it would never start. Require `ci / required` only. +- The partition makes `windows-native-test` run fewer tests than `windows.yaml` (`just test-windows-full` and `windows-nextest-full` run all of them by hand). When + it has been green for a while, the test step of `windows.yaml` can be removed by the owner of that file. +- The partition (`tools/test-partition.json`) puts `komorebi-windows` in the class `windows` and `komorebi-config`, `-protocol`, `-transport` and `komorebic-core` in `os_sensitive`: + those are what `windows-native-test` runs. `komorebi-client`, `komorebi-bar` and `komorebi-recorder` are `mixed`, so a platform job names them with `-p`. The nix set `windows-test` builds the + same subset; the whole suite is the set `windows-test-full` (not in the default sets, so not in `ci`) and `just test-windows-full`. +- Both run `cargo-deny`; `ci / deny` is the cheaper copy and stops the ladder first. +- `actionlint` in `hygiene` leaves `windows.yaml` out: it reads the PowerShell of that file as bash. `just lint` runs it on every workflow and fails on that file. + +## Repository settings that the owner has to change + +None of these can be set from a workflow. + +1. A ruleset on `master` that requires the status check `ci / required` (and nothing else), with the merge queue on if wanted. +2. Settings, Actions, General: the default token permission is read-only; approval to run workflows from outside collaborators (first-time contributors at least). +3. Already created, nothing to do: the public Cachix cache `komorebi` and the variable `CACHIX_CACHE=komorebi`, the secret `CACHIX_AUTH_TOKEN` (read and write), the secret + `CACHIX_AUTH_TOKEN_RO` and the variable `CACHIX_PUBLIC_KEY`. The workflows use only `CACHIX_CACHE` and `CACHIX_AUTH_TOKEN`: the cache is public, and `cachix/cachix-action` + configures the substituter and its key with no token at all, so a pull request, a fork and the temporary branch read the cache without one and + `CACHIX_AUTH_TOKEN_RO` and `CACHIX_PUBLIC_KEY` are not needed (they would only be for a hand-written `extra-substituters`). The write token is handed to the action + (as its `authToken` input, by `.github/actions/nix`, which cannot read secrets itself) only on a push to master, on `workflow_dispatch` and on `schedule`; on a + pull request it is an empty string. A job that has no `CACHIX_CACHE` builds with no cache. + Still the owner's: the variable `MACOS_SELF_HOSTED` (`true` once the Mac Mini has a runner with the labels `self-hosted`, `macOS`, `ARM64`). +4. The labels: `ci:macos`, `ci:macos-self`, `ci:vm`, `ci:vm-slow`, `ci:win-arm`. +5. Settings, Actions: allow the actions the workflows use (`cachix/*`, `dorny/paths-filter`, `taiki-e/install-action`, `Swatinem/rust-cache`, `DeterminateSystems/update-flake-lock`, + `EmbarkStudios/cargo-deny-action`) if the repository restricts them. + + +## The `komomono` removal checklist + +`ci.yaml` and `macos-selfhosted.yaml` run on a push to the temporary branch `komomono`, so that the stack can be tried before it is merged to `master`. Each +has a line `- komomono` under the comment `# TEMPORARY: remove when komomono is merged to master`. When it is merged: + +- [ ] delete the `- komomono` line and its comment in `.github/workflows/ci.yaml` +- [ ] delete the same two lines in `.github/workflows/macos-selfhosted.yaml` +- [ ] `sweeps.yaml` never runs on that branch (its push trigger is `master` only): nothing to delete there + +`grep -rn komomono .github` finds nothing afterwards. No expression in a workflow names the branch: the conditions say "a push" and "a push to master". + +## How to verify before merging + +Nothing can be run on GitHub from a checkout, so the first run is the test. + +1. `nix shell nixpkgs#actionlint nixpkgs#shellcheck -c actionlint` over the workflows other than `windows.yaml`. +2. `nix eval` of every attribute the workflows build, for example `nix eval .#legacyPackages.x86_64-linux.checkSets.linux-lint.name` and `nix eval --json .#legacyPackages.x86_64-linux.vmTiers`; + `nix flake check --no-build --all-systems`. +3. Push the stack to a scratch branch, add that branch to the `push` list of `ci.yaml` in the scratch branch, and run `ci` with `workflow_dispatch` (all platforms). Then: + a pull request with no label (stages 0 to 3), one with `ci:vm`, one with `ci:macos`, and one with `ci:win-arm`. +4. Break a check on purpose and see that `ci / required` fails; skip a stage with a path and see that it passes. +5. Run `sweeps` and `windows-guest` by hand once. + +## Open items + +- Cachix: whether the free tier of the cache `komorebi` is large enough for the store of this flake (several GB per system) is not verified. +- KVM on hosted runners: documented for public repositories; the action fails with a named step when a host has none, and `rerun-no-kvm` tries again. +- The macOS image: `macos-14` is Apple silicon with its own nix install each run; whether the cache makes the checks fit the 2 hours of the job is not known. +- The `dependencies.json` step of `hygiene` installs the latest `cargo-deny`; run on a Mac with cargo-deny 0.19.6 and no network it did not reproduce the checked-in file (the git dependency was missing and the workspace crates sat under another license), so the first run on GitHub may show a difference that is the version of `cargo-deny` and not a stale file. If so, pin the version in the `taiki-e/install-action` step to the one that wrote the file. +- The memory of `nix flake check --no-build` over the nixosConfigurations (about 7 GB) against a runner that has 7 GB (private repositories) or 16 GB (public): the swap of the action is the mitigation. +- `windows-native-test-arm` assumes the `windows-11-arm` runner exists for this repository and installs Rust itself when the image has none.