diff --git a/docs/dev/build-and-test.md b/docs/dev/build-and-test.md index 9847ccaa..bd1e7542 100644 --- a/docs/dev/build-and-test.md +++ b/docs/dev/build-and-test.md @@ -58,7 +58,7 @@ shared by all of them): ```sh nix build -L --no-link .#checks.x86_64-linux.windows-clippy -# (there is no windows-nextest: the Windows crates' tests run on Windows, `cargo test` of .github/workflows/windows.yaml) +nix build -L --no-link .#checks.x86_64-linux.windows-nextest-build # cross-compiles the Windows crates' test executables (48), runs none nix build -L --no-link .#checks.x86_64-linux.windows-fmt .#checks.x86_64-linux.windows-toml-fmt .#checks.x86_64-linux.windows-deny nix build -L --no-link .#checks.x86_64-linux.windows-e2e-tests # the Python of the e2e (pytest), no guest needed ``` @@ -186,6 +186,7 @@ Full text: `e2e/platforms/qemu_windows/README.md` (building the golden image, ev ```sh E2E_GOLDEN=golden nix run .#e2e-windows -- OUT SCENARIO... # e.g. OUT tiling focus commands E2E_GOLDEN=golden KOMOREBI_E2E_SCALE=1.5 nix run .#e2e-windows -- OUT bar # the komorebi-bar scenario +E2E_GOLDEN=golden KOMOREBI_E2E_STATE=state-nextest nix run .#windows-nextest -- OUT # the Windows crates' unit tests in the guest ``` The app uses this checkout's binaries, built with the constant stamp `e2e` (so a docs-only commit reuses them); diff --git a/docs/dev/flake-checks.md b/docs/dev/flake-checks.md index 1affc2e9..19d41968 100644 --- a/docs/dev/flake-checks.md +++ b/docs/dev/flake-checks.md @@ -75,7 +75,7 @@ members, and every set is also one derivation (`pkgs.linkFarm`) that builds its | `fmt` | `treefmt`, `pre-commit`, `windows-fmt`, `windows-toml-fmt`, `linux-fmt`, `linux-toml-fmt` (macOS systems: `macos-fmt`, `macos-toml-fmt`) | cheap | | `deny` | `windows-deny`, `linux-deny` (`macos-deny`) | cheap | | `windows-lint` | `windows-clippy` | cheap | -| `windows-test` | `windows-e2e-tests` (the Rust tests of the Windows crates cannot run on a Linux builder; the old `windows-nextest` ran none, see below) | cheap | +| `windows-test` | `windows-e2e-tests`, `windows-nextest-build` (the Windows test executables, cross-built and not run; they run in the guest with `nix run .#windows-nextest`, below) | cheap | | `linux-lint` | `linux-clippy`, `linux-doc` | cheap | | `linux-test` | `linux-nextest`, `linux-bridge-js`, `linux-keybindings-fixtures`, `vm-lib`, `linux-wsl2-probe`, `linux-wsl2-probe-dry-run`, the two effect install checks | cheap | | `linux-demo` | `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run` | 1.5 GB | @@ -85,7 +85,7 @@ members, and every set is also one derivation (`pkgs.linkFarm`) that builds its | `vm-fast` | the suites whose `tier` is `fast` (16), computed from `suite.tier` | 9 GB | | `vm-slow` | the suites of tier `slow` (18) | 10 GB | -The Windows end-to-end run is an app (`nix run .#e2e-windows`), not a check: it needs a golden image. The +The Windows end-to-end run is an app (`nix run .#e2e-windows`), and so is `windows-nextest`, not a check: it needs a golden image. The sets that exist on a system are the ones with members on it (a Mac has no `windows-*` build host except Apple silicon's cross build, which the member lists already say). @@ -174,18 +174,36 @@ where its two checks took 10.3 + 9.0 GB alone before. Every set of one platform set was killed. Nothing is lost: the check names of every system (`checks` plus what `heavyChecks` had) before, and the union -of `checkMembers` after, are the same sets except `windows-nextest` (below): 61 names on x86_64-linux and -aarch64-linux, 27 on aarch64-darwin, 9 on x86_64-darwin. The `drvPath` of every one of them is the same before and +of `checkMembers` after, are the same sets except `windows-nextest` (below), which is now an app, and the new check +`windows-nextest-build`: that was 61 names on x86_64-linux and aarch64-linux, 27 on aarch64-darwin, 9 on +x86_64-darwin, and each system that builds the Windows crates has one more now (not re-counted). The `drvPath` of every one of them is the same before and after, except `treefmt` and `pre-commit`, which hash the whole source tree and so change with any edit of it. ## `windows-nextest` and `macos-nextest` ran no test `cargoNextest` of crane runs the tests in its check phase, and the Windows and macOS `individualCrateArgs` have `doCheck = false` (Linux passes `doCheck = true`), so the derivation ran `cargo nextest --version` and -nothing else. A Windows program's tests cannot run on a Linux builder, so `windows-nextest` is gone (a check -that runs nothing under the name of one that runs tests is worse than no check) and `windows-clippy ---all-targets` is what compiles them; the Windows CI runs `cargo test`. `macos-nextest` is the same shape and is -kept, unchanged and still named for what it does not do, only until a Mac run decides whether `doCheck = true` -is right there (the tests of `komorebi-macos` run natively on an Apple machine). Open: the tests of the -platform-neutral crates `komorebi-layouts` and `komorebi-scrolling` run in the Windows CI and nowhere in nix -(`linux-nextest` names `-p komorebi -p komorebi-linux -p komorebic -p komorebi-client -p komorebi-hotkeys`). +nothing else. A Windows program's tests cannot run on a Linux builder, so the check `windows-nextest` was +removed (a check that runs nothing under the name of one that runs tests is worse than no check). The tests +now run in the Windows e2e guest, which is the answer to "can the Windows nextest run in a VM from NixOS": + +- `windows-nextest-build` (a check, set `windows-test`) cross-compiles every test executable of the Windows + workspace (`cargo test --workspace --release --no-run`, xwin, 48 executables) and keeps them, so that they + are proved to compile and link; about 90 s with warm dependencies. +- `windows-nextest` (an app, the name of the old check) boots the golden image like a scenario, uploads the + executables and the data files they read at the path the build baked in (the schemas, the fixtures), runs + every test in a process of its own (as nextest does) and prints a line per executable and the failures; + `OUT/cargo-tests.json` has every result. The guest needs KVM and the golden image, so it is not a check: + + ```sh + E2E_GOLDEN=golden KOMOREBI_E2E_SCALE=1.5 KOMOREBI_E2E_STATE=state-nextest nix run .#windows-nextest -- OUT + ``` + + 964 tests ran and passed (the first run also saves the guest state, 163 s; after that about 3.5 minutes, + of which the tests are 171 s). It is a release build: debug assertions are off, as in `linux-nextest`. + +The neutral crates (`komorebi-layouts`, `komorebi-scrolling`, `komorebi-themes`, the shared `crates/*`) now +also run in `linux-nextest` (1532 tests with the Linux packages), which needs the schemas of every platform +and `docs/macos/cli` in the Linux source (`nix/workspace-src.nix`). `macos-nextest` is the same shape as the +old Windows one and is kept, unchanged and still named for what it does not do, only until a Mac run decides +whether `doCheck = true` is right there (the tests of `komorebi-macos` run natively on an Apple machine). diff --git a/e2e/platforms/qemu_windows/cargo_tests.py b/e2e/platforms/qemu_windows/cargo_tests.py index a8e9ddfe..80d38903 100644 --- a/e2e/platforms/qemu_windows/cargo_tests.py +++ b/e2e/platforms/qemu_windows/cargo_tests.py @@ -1,10 +1,13 @@ """Runs the cargo tests of the workspace in the Windows guest. -usage: python -m platforms.qemu_windows.cargo_tests GOLDEN OUT STATE MESSAGES +usage: python -m platforms.qemu_windows.cargo_tests GOLDEN OUT STATE TESTS -MESSAGES is the `--message-format json` output of -`cargo test --workspace --no-run` for the Windows target, which names every -test executable. Each is uploaded to the guest, asked for its tests, and every +TESTS is a directory with a `tests.tsv` (name, tab, file under `tests/`: the +output of the flake check `windows-nextest-build`), or the +`--message-format json` output of `cargo test --workspace --no-run` for the +Windows target, which names every test executable. STATE is a directory to +restore the guest from (made if missing, as `KOMOREBI_E2E_STATE` does for the +scenarios), or `-` to boot. Each is uploaded to the guest, asked for its tests, and every test is run in a process of its own (as nextest does: komorebi's tests share global state), with its stdout kept. Prints a line per executable and the tests that failed; OUT/cargo-tests.json has every result. Exits 1 if any test @@ -17,11 +20,18 @@ import time from pathlib import Path from .guest import Guest +from .state import ensure GUEST_DIR = "C:\\e2e\\tests" def executables(messages: Path) -> list[tuple[str, Path]]: + if messages.is_dir(): + listing = (messages / "tests.tsv").read_text().splitlines() + return sorted( + (name, messages / "tests" / file) + for name, file in (line.split("\t") for line in listing) + ) found = {} for line in messages.read_text().splitlines(): try: @@ -43,12 +53,43 @@ def tests_of(guest: Guest, exe: str) -> list[str]: ] +def upload_data(guest: Guest, tests: Path) -> None: + """Puts the data files the tests read (the schemas, fixtures) in the guest + at the path the build baked into the executables (CARGO_MANIFEST_DIR, the + build directory of the check, as a path on the guest's C: drive).""" + if not (tests.is_dir() and (tests / "srcroot").exists()): + return + root = "C:" + (tests / "srcroot").read_text().strip().replace("/", "\\") + data = tests / "data" + files = sorted(p for p in data.rglob("*") if p.is_file()) + made: set[str] = set() + for path in files: + remote = root + "\\" + str(path.relative_to(data)).replace("/", "\\") + directory = remote.rsplit("\\", 1)[0] + if directory not in made: + guest.agent.exec("cmd", "/c", f"mkdir {directory} 2>nul") + made.add(directory) + guest.agent.put_file(path, remote) + print(f"data: {len(files)} files under {root}", flush=True) + + +def reason(output: str) -> str: + """The lines that say why a test failed: the panic message, else the end.""" + lines = [l for l in output.splitlines() if l.strip()] + for i, line in enumerate(lines): + if "panicked at" in line: + return " | ".join(lines[i : i + 2]) + return " | ".join(lines[-3:]) + + def main(argv: list[str]) -> int: golden, out, state, messages = (Path(a) for a in argv[:4]) out.mkdir(parents=True, exist_ok=True) + saved = ensure(golden, state) if str(state) != "-" else None results: dict[str, dict] = {} - with Guest(golden, out / "guest", state=state if str(state) != "-" else None) as guest: + with Guest(golden, out / "guest", state=saved) as guest: guest.agent.exec("cmd", "/c", f"mkdir {GUEST_DIR} 2>nul") + upload_data(guest, messages) for name, path in executables(messages): exe = f"{GUEST_DIR}\\{name}.exe" began = time.monotonic() @@ -62,7 +103,7 @@ def main(argv: list[str]) -> int: else: failed.append({ "test": test, "status": run["status"], - "output": (run["stdout"] + run["stderr"])[-1500:], + "output": (run["stdout"] + run["stderr"])[-4000:], }) results[name] = { "tests": len(names), "passed": passed, "failed": failed, @@ -72,8 +113,7 @@ def main(argv: list[str]) -> int: f"{'' if not failed else f', {len(failed)} FAILED'}" f" ({results[name]['seconds']} s)", flush=True) for f in failed: - first = [l for l in f["output"].splitlines() if l.strip()][-3:] - print(f" FAIL {f['test']} (status {f['status']}): {' | '.join(first)}", flush=True) + print(f" FAIL {f['test']} (status {f['status']}): {reason(f['output'])}", flush=True) (out / "cargo-tests.json").write_text(json.dumps(results, indent=2)) total = sum(r["tests"] for r in results.values()) failed = sum(len(r["failed"]) for r in results.values()) diff --git a/nix/windows/flake-module.nix b/nix/windows/flake-module.nix index 4aee5a63..8008dd50 100644 --- a/nix/windows/flake-module.nix +++ b/nix/windows/flake-module.nix @@ -62,6 +62,25 @@ in "$E2E_GOLDEN" "$bins" "$out" "$@" ''; }; + # the Windows unit tests in the guest: every test of every executable in a process of its own + windowsNextest = pkgs.writeShellApplication { + name = "windows-nextest"; + runtimeInputs = [ + pkgs.qemu + pkgs.swtpm + (pkgs.python3.withPackages (p: [ p.pillow ])) + ]; + text = '' + : "''${E2E_GOLDEN:?set E2E_GOLDEN to the golden image directory}" + out=''${1:?usage: windows-nextest OUT} + export OVMF_FD=${ovmfWithTpm} + export PYTHONPATH=${lib'.repoFile "e2e/lib"}:${lib'.repoFile "e2e"} + exec python -m platforms.qemu_windows.cargo_tests \ + "$E2E_GOLDEN" "$out" "''${KOMOREBI_E2E_STATE:--}" ${windowsNextestBuild} + ''; + }; + # the test executables of the Windows workspace, with a list of them (name, tab, file) for the + # guest runner; the check `windows-nextest-build` and the app `windows-nextest` use it # the members that are stubs here (the Linux and macOS packages) have no test and do not build excluded = pkgs.lib.concatMapStringsSep " " (p: "--exclude ${p}") build.stubbedPackages; windowsNextestBuild = build.craneLib.mkCargoDerivation ( @@ -69,7 +88,10 @@ in // { pnameSuffix = "-nextest-build"; doInstallCargoArtifacts = false; - nativeBuildInputs = build.individualCrateArgs.nativeBuildInputs ++ [ pkgs.jq ]; + nativeBuildInputs = build.individualCrateArgs.nativeBuildInputs ++ [ + pkgs.jq + pkgs.cpio + ]; buildPhaseCargoCommand = '' cargo test --locked --workspace ${excluded} --release --no-run --message-format json > messages.json ''; @@ -81,6 +103,13 @@ in cp "$exe" "$out/tests/$name.exe" printf '%s\t%s.exe\n' "$name" "$name" >> $out/tests.tsv done + # the data files the tests read by the absolute path the build baked into them + # (CARGO_MANIFEST_DIR): the guest gets them at the same path under C: + pwd > $out/srcroot + mkdir -p $out/data + find . -path ./target -prune -o -type f \ + \( -path './schemas/*' -o -path '*/tests/*' -o -path '*/fixtures/*' \) \ + ! -name '*.rs' -print | cpio -pdm $out/data 2>/dev/null ''; } ); @@ -223,6 +252,12 @@ in type = "app"; program = pkgs.lib.getExe e2eWindows; }; + # the unit tests of the Windows crates in the guest (E2E_GOLDEN as above): + # E2E_GOLDEN= KOMOREBI_E2E_SCALE=1.5 nix run .#windows-nextest -- OUT + windows-nextest = { + type = "app"; + program = pkgs.lib.getExe windowsNextest; + }; }; }; }