From 865ffcebda378f1f560b588f7ef548f69f467db7 Mon Sep 17 00:00:00 2001 From: LGUG2Z Date: Fri, 6 Nov 2026 09:17:56 -0800 Subject: [PATCH] docs(ci): what is guaranteed about KVM on hosted runners, what is drawn again, and what is not covered --- docs/dev/ci.md | 43 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/docs/dev/ci.md b/docs/dev/ci.md index 3bb2795b..14f43b2b 100644 --- a/docs/dev/ci.md +++ b/docs/dev/ci.md @@ -27,7 +27,7 @@ each is `success` or `skipped`: it is the only check a ruleset needs to require. | 3 | `test` (4 legs) | ubuntu-24.04, nix | `checkSets.{linux-test,windows-test,recorder-test,nixgen-test}`: Linux tests, the Windows test executables cross-built (not run), the recorder's and nixgen's tests | always | | 3 | `windows-native-test` | windows-latest | `cargo nextest` of the `windows` and `os_sensitive` classes of `tools/test-partition` | Windows or shared paths | | 3 | `windows-native-test-arm` | windows-11-arm | the same on ARM | label `ci:win-arm`; a push to master with Windows paths; by hand | -| 4 | `discover`, `vm-fast` | ubuntu-24.04, KVM | the fast tier of the KDE VM suites, one machine each (`max-parallel: 8`, 45 minutes) | Linux paths; label `ci:vm`; every push to master | +| 4 | `discover`, `vm-fast` | ubuntu-24.04, KVM | the fast tier of the KDE VM suites, one machine each (`max-parallel: 8`, 45 minutes), a machine with no KVM drawn again (`_vm.yaml`, see KVM) | Linux paths; label `ci:vm`; every push to master | | 5 | `macos-hosted` | macos-14 (`_macos.yaml`) | the macOS checks, the recorder's macOS tests, the binaries, the schemas | label `ci:macos` on a pull request; macOS paths on a push | Beside the ladder: @@ -35,11 +35,10 @@ Beside the ladder: | workflow | jobs | runner | what it covers | when it runs | |---|---|---|---|---| | `macos-selfhosted.yaml` | `macos` (`_macos.yaml`) | the Mac Mini | the same as `macos-hosted`, on the machine of the owner | variable `MACOS_SELF_HOSTED=true`, and: by hand, a push to master, or label `ci:macos-self` on a pull request from this repository | -| `sweeps.yaml` | `vm-slow` | ubuntu-24.04, KVM | the slow tier of the VM suites, 60 minutes each, logs kept 7 days | label `ci:vm-slow`; every push to master; by hand | +| `sweeps.yaml` | `vm-slow` | ubuntu-24.04, KVM | the slow tier of the VM suites, 60 minutes each, logs kept 7 days, a machine with no KVM drawn again (see KVM) | label `ci:vm-slow`; every push to master; by hand | | | `nixos-eval`, `linux-demo` | ubuntu-24.04, swap | every NixOS system of the module and of the Plasma guests (about 11.6 GB to evaluate), and the demo guest | the same | | | `canary` (3 legs) | ubuntu-24.04, KVM | every check against the tip of nixpkgs-unstable, one leg for the checks, one for each VM tier; it opens an issue when one fails | Saturday 05:20 UTC; by hand | | `update-flake-lock.yaml` | `update` | ubuntu-24.04 | a pull request that moves `flake.lock` | Monday 03:00 UTC; by hand | -| `rerun-no-kvm.yaml` | `rerun` | ubuntu-24.04 | runs the failed jobs again when every one failed for want of `/dev/kvm` (three attempts at most) | after `ci` or `sweeps` | | `windows-guest.yaml` | `guest` | ubuntu-24.04, KVM | EXPERIMENTAL: builds the Windows guest from the evaluation ISO and runs `nix run .#windows-nextest` in it | by hand only | `ci` runs on a pull request (opened, pushed to, reopened, labeled), in the merge queue, on a push to `master`, and by hand. A @@ -121,6 +120,42 @@ None of these can be set from a workflow. `EmbarkStudios/cargo-deny-action`) if the repository restricts them. +## KVM on hosted runners + +What GitHub documents: hardware acceleration (KVM) on the Linux runners, with the udev rule that `.github/actions/nix` writes +(changelog of 2024-04-02, and of 2023-02-23 for the first release on larger runners). What it does not document: that every +host of the `ubuntu-24.04` pool has it. The pages "GitHub-hosted runners" and "Larger runners" say nothing about nested +virtualization, and no service level covers it. Nothing in the repository can choose the host: a job is given whichever one the +pool has. In run 42182554717 (all 16 hosts the same image, `ubuntu24/20261101.360`, spread over six Azure regions) 3 of 16 failed +the check of `/dev/kvm`, 13 had it; what is known of those 3 is only that the check failed 42 to 46 ms after the udev trigger, which +cannot tell a missing device from a rule that was not applied yet. The step now logs which of the two it is. + +What the repository does: + +- `.github/actions/nix`, step "Enable KVM" (the only place that enables it: `enable_kvm` of `cachix/install-nix-action` writes the + same rule, ignores the outcome, and is turned off): writes the rule, loads `kvm_intel` or `kvm_amd`, reloads and triggers udev, + waits for `udevadm settle`, polls `/dev/kvm` for 30 seconds for read and write access, and as the last resort sets the mode by + hand. It prints `ls -l /dev/kvm`, `id`, `systemd-detect-virt`, the count of `vmx`/`svm` flags and the loaded `kvm` modules, before and after. +- `_vm.yaml` calls `_vm-attempt.yaml` up to three times. In the first two a host with no KVM does not fail its leg: the leg + uploads the artifact `kvm-missing---` and ends green without having run the suite, and the next + attempt (a job `plan` that lists the artifacts) runs those suites again, each on a new machine. In the third, a host with no KVM fails + the suite. So a suite is run, or the call fails: three draws without KVM are the only way a suite is not run and the check is red for it. +- `vm-fast` (`ci.yaml`) and `vm-slow` (`sweeps.yaml`) are calls of `_vm.yaml`; `required` reads the result of the call. It works on any + branch and any event, because it is inside the one run: it needs no workflow on the default branch and no rerun. (A rerun of the + failed jobs of a run cannot be started by the run itself while it is in progress, and a `workflow_run` workflow runs only from the file + on the default branch; that is why the earlier `rerun-no-kvm.yaml` is gone.) +- Not covered: the three legs of `canary` and `windows-guest.yaml` (one machine each, hours long): a host with no KVM fails the + step "Enable KVM", with the message `this runner has no usable /dev/kvm`; run them again by hand. + +When GitHub gives no KVM at all for a long stretch (every leg of three draws fails, the log of the step says the device is absent +and `systemd-detect-virt` or the `vmx` count says why): the check cannot pass on hosted runners. The ways out are a larger +runner (`runs-on` with a label of a larger-runner group: billed per minute also on a public repository, and nothing in the +documentation promises nested virtualization there either, only that the Android emulator is supported), or a self-hosted x86-64 +Linux machine with KVM, with `runs-on` changed in `_vm-attempt.yaml`. + +Not verified: none of this has run on GitHub. In particular the artifact listing of `plan`, the nesting of two reusable workflows +and the result of a call whose legs are skipped. + ## The `komomono` removal checklist `ci.yaml` and `macos-selfhosted.yaml` run on a push to the temporary branch `komomono`, so that the stack can be tried before it is merged to `master`. Each @@ -159,7 +194,7 @@ the post step runs when the job fails too). ## Open items - Cachix: whether the free tier of the cache `komorebi` is large enough for the store of this flake (several GB per system) is not verified. -- KVM on hosted runners: documented for public repositories; the action fails with a named step when a host has none, and `rerun-no-kvm` tries again. +- KVM on hosted runners: documented for public repositories; see KVM below: GitHub does not promise it of every host, so the workflows draw again, up to three times in all. - The macOS image: `macos-14` is Apple silicon with its own nix install each run; whether the cache makes the checks fit the 2 hours of the job is not known. - The first run of stage 0 sets up the `hygiene` shell, which is a download of about 1 GB until the Cachix cache has it (a push and a manual run push it). - The memory of `nix flake check --no-build` over the nixosConfigurations (about 7 GB) against a runner that has 7 GB (private repositories) or 16 GB (public): the swap of the action is the mitigation.