mirror of
https://github.com/LGUG2Z/komorebi.git
synced 2026-11-18 06:38:38 +01:00
feat(e2e): windows-nextest runs the Windows unit tests in the guest
The Windows crates' unit tests could only run in the (future) hosted Windows CI. They run in the existing Windows e2e guest: nix run .#windows-nextest boots the golden image like a scenario (restoring a saved state when KOMOREBI_E2E_STATE is set), uploads the test executables that the check windows-nextest-build cross-compiled, runs every test in a process of its own (nextest.exe is not needed: komorebi's tests share global state, so one process per test is what nextest does too) and prints a line per executable and the failures, with OUT/cargo-tests.json for every result. The driver e2e/platforms/qemu_windows/cargo_tests.py already existed and was wired to nothing; it now takes the check output (tests.tsv) as well as cargo messages, creates the saved state itself, prints the panic message of a failure, and uploads the data files the tests read. Those are found by the CARGO_MANIFEST_DIR baked into the executables, a path of the Nix build, so the check also keeps schemas/ and the tests' fixtures and the driver puts them at that path under C:. Without them 20 tests failed for the missing files. docs/dev/build-and-test.md and docs/dev/flake-checks.md describe the check, the app and the neutral crates in linux-nextest. Verified: nix run .#windows-nextest on the golden image g8 (scale 1.5, state ~/e2e-w0/state-nextest): 964 tests in 37 executables, 964 passed, 0 failed, 202 s with the state restored (first run 163 s more to save the state). One run only; not soaked. No e2e scenario, the golden image or the existing e2e-windows behaviour changed.
This commit is contained in:
1 parent
d872c9ec43
commit
3e21ebd3a4
4 files changed
+115
-21
No files matched your search
@@ -58,7 +58,7 @@ shared by all of them):
|
||||
|
||||
```sh
|
||||
nix build -L --no-link .#checks.x86_64-linux.windows-clippy
|
||||
# (there is no windows-nextest: the Windows crates' tests run on Windows, `cargo test` of .github/workflows/windows.yaml)
|
||||
nix build -L --no-link .#checks.x86_64-linux.windows-nextest-build # cross-compiles the Windows crates' test executables (48), runs none
|
||||
nix build -L --no-link .#checks.x86_64-linux.windows-fmt .#checks.x86_64-linux.windows-toml-fmt .#checks.x86_64-linux.windows-deny
|
||||
nix build -L --no-link .#checks.x86_64-linux.windows-e2e-tests # the Python of the e2e (pytest), no guest needed
|
||||
```
|
||||
@@ -186,6 +186,7 @@ Full text: `e2e/platforms/qemu_windows/README.md` (building the golden image, ev
|
||||
```sh
|
||||
E2E_GOLDEN=golden nix run .#e2e-windows -- OUT SCENARIO... # e.g. OUT tiling focus commands
|
||||
E2E_GOLDEN=golden KOMOREBI_E2E_SCALE=1.5 nix run .#e2e-windows -- OUT bar # the komorebi-bar scenario
|
||||
E2E_GOLDEN=golden KOMOREBI_E2E_STATE=state-nextest nix run .#windows-nextest -- OUT # the Windows crates' unit tests in the guest
|
||||
```
|
||||
|
||||
The app uses this checkout's binaries, built with the constant stamp `e2e` (so a docs-only commit reuses them);
|
||||
|
||||
+29
-11
@@ -75,7 +75,7 @@ members, and every set is also one derivation (`pkgs.linkFarm`) that builds its
|
||||
| `fmt` | `treefmt`, `pre-commit`, `windows-fmt`, `windows-toml-fmt`, `linux-fmt`, `linux-toml-fmt` (macOS systems: `macos-fmt`, `macos-toml-fmt`) | cheap |
|
||||
| `deny` | `windows-deny`, `linux-deny` (`macos-deny`) | cheap |
|
||||
| `windows-lint` | `windows-clippy` | cheap |
|
||||
| `windows-test` | `windows-e2e-tests` (the Rust tests of the Windows crates cannot run on a Linux builder; the old `windows-nextest` ran none, see below) | cheap |
|
||||
| `windows-test` | `windows-e2e-tests`, `windows-nextest-build` (the Windows test executables, cross-built and not run; they run in the guest with `nix run .#windows-nextest`, below) | cheap |
|
||||
| `linux-lint` | `linux-clippy`, `linux-doc` | cheap |
|
||||
| `linux-test` | `linux-nextest`, `linux-bridge-js`, `linux-keybindings-fixtures`, `vm-lib`, `linux-wsl2-probe`, `linux-wsl2-probe-dry-run`, the two effect install checks | cheap |
|
||||
| `linux-demo` | `linux-run-kde`, `linux-run-kde-doctor`, `linux-run-kde-dry-run` | 1.5 GB |
|
||||
@@ -85,7 +85,7 @@ members, and every set is also one derivation (`pkgs.linkFarm`) that builds its
|
||||
| `vm-fast` | the suites whose `tier` is `fast` (16), computed from `suite.tier` | 9 GB |
|
||||
| `vm-slow` | the suites of tier `slow` (18) | 10 GB |
|
||||
|
||||
The Windows end-to-end run is an app (`nix run .#e2e-windows`), not a check: it needs a golden image. The
|
||||
The Windows end-to-end run is an app (`nix run .#e2e-windows`), and so is `windows-nextest`, not a check: it needs a golden image. The
|
||||
sets that exist on a system are the ones with members on it (a Mac has no `windows-*` build host except
|
||||
Apple silicon's cross build, which the member lists already say).
|
||||
|
||||
@@ -174,18 +174,36 @@ where its two checks took 10.3 + 9.0 GB alone before. Every set of one platform
|
||||
set was killed.
|
||||
|
||||
Nothing is lost: the check names of every system (`checks` plus what `heavyChecks` had) before, and the union
|
||||
of `checkMembers` after, are the same sets except `windows-nextest` (below): 61 names on x86_64-linux and
|
||||
aarch64-linux, 27 on aarch64-darwin, 9 on x86_64-darwin. The `drvPath` of every one of them is the same before and
|
||||
of `checkMembers` after, are the same sets except `windows-nextest` (below), which is now an app, and the new check
|
||||
`windows-nextest-build`: that was 61 names on x86_64-linux and aarch64-linux, 27 on aarch64-darwin, 9 on
|
||||
x86_64-darwin, and each system that builds the Windows crates has one more now (not re-counted). The `drvPath` of every one of them is the same before and
|
||||
after, except `treefmt` and `pre-commit`, which hash the whole source tree and so change with any edit of it.
|
||||
|
||||
## `windows-nextest` and `macos-nextest` ran no test
|
||||
|
||||
`cargoNextest` of crane runs the tests in its check phase, and the Windows and macOS `individualCrateArgs`
|
||||
have `doCheck = false` (Linux passes `doCheck = true`), so the derivation ran `cargo nextest --version` and
|
||||
nothing else. A Windows program's tests cannot run on a Linux builder, so `windows-nextest` is gone (a check
|
||||
that runs nothing under the name of one that runs tests is worse than no check) and `windows-clippy
|
||||
--all-targets` is what compiles them; the Windows CI runs `cargo test`. `macos-nextest` is the same shape and is
|
||||
kept, unchanged and still named for what it does not do, only until a Mac run decides whether `doCheck = true`
|
||||
is right there (the tests of `komorebi-macos` run natively on an Apple machine). Open: the tests of the
|
||||
platform-neutral crates `komorebi-layouts` and `komorebi-scrolling` run in the Windows CI and nowhere in nix
|
||||
(`linux-nextest` names `-p komorebi -p komorebi-linux -p komorebic -p komorebi-client -p komorebi-hotkeys`).
|
||||
nothing else. A Windows program's tests cannot run on a Linux builder, so the check `windows-nextest` was
|
||||
removed (a check that runs nothing under the name of one that runs tests is worse than no check). The tests
|
||||
now run in the Windows e2e guest, which is the answer to "can the Windows nextest run in a VM from NixOS":
|
||||
|
||||
- `windows-nextest-build` (a check, set `windows-test`) cross-compiles every test executable of the Windows
|
||||
workspace (`cargo test --workspace --release --no-run`, xwin, 48 executables) and keeps them, so that they
|
||||
are proved to compile and link; about 90 s with warm dependencies.
|
||||
- `windows-nextest` (an app, the name of the old check) boots the golden image like a scenario, uploads the
|
||||
executables and the data files they read at the path the build baked in (the schemas, the fixtures), runs
|
||||
every test in a process of its own (as nextest does) and prints a line per executable and the failures;
|
||||
`OUT/cargo-tests.json` has every result. The guest needs KVM and the golden image, so it is not a check:
|
||||
|
||||
```sh
|
||||
E2E_GOLDEN=golden KOMOREBI_E2E_SCALE=1.5 KOMOREBI_E2E_STATE=state-nextest nix run .#windows-nextest -- OUT
|
||||
```
|
||||
|
||||
964 tests ran and passed (the first run also saves the guest state, 163 s; after that about 3.5 minutes,
|
||||
of which the tests are 171 s). It is a release build: debug assertions are off, as in `linux-nextest`.
|
||||
|
||||
The neutral crates (`komorebi-layouts`, `komorebi-scrolling`, `komorebi-themes`, the shared `crates/*`) now
|
||||
also run in `linux-nextest` (1532 tests with the Linux packages), which needs the schemas of every platform
|
||||
and `docs/macos/cli` in the Linux source (`nix/workspace-src.nix`). `macos-nextest` is the same shape as the
|
||||
old Windows one and is kept, unchanged and still named for what it does not do, only until a Mac run decides
|
||||
whether `doCheck = true` is right there (the tests of `komorebi-macos` run natively on an Apple machine).
|
||||
@@ -1,10 +1,13 @@
|
||||
"""Runs the cargo tests of the workspace in the Windows guest.
|
||||
|
||||
usage: python -m platforms.qemu_windows.cargo_tests GOLDEN OUT STATE MESSAGES
|
||||
usage: python -m platforms.qemu_windows.cargo_tests GOLDEN OUT STATE TESTS
|
||||
|
||||
MESSAGES is the `--message-format json` output of
|
||||
`cargo test --workspace --no-run` for the Windows target, which names every
|
||||
test executable. Each is uploaded to the guest, asked for its tests, and every
|
||||
TESTS is a directory with a `tests.tsv` (name, tab, file under `tests/`: the
|
||||
output of the flake check `windows-nextest-build`), or the
|
||||
`--message-format json` output of `cargo test --workspace --no-run` for the
|
||||
Windows target, which names every test executable. STATE is a directory to
|
||||
restore the guest from (made if missing, as `KOMOREBI_E2E_STATE` does for the
|
||||
scenarios), or `-` to boot. Each is uploaded to the guest, asked for its tests, and every
|
||||
test is run in a process of its own (as nextest does: komorebi's tests share
|
||||
global state), with its stdout kept. Prints a line per executable and the
|
||||
tests that failed; OUT/cargo-tests.json has every result. Exits 1 if any test
|
||||
@@ -17,11 +20,18 @@ import time
|
||||
from pathlib import Path
|
||||
|
||||
from .guest import Guest
|
||||
from .state import ensure
|
||||
|
||||
GUEST_DIR = "C:\\e2e\\tests"
|
||||
|
||||
|
||||
def executables(messages: Path) -> list[tuple[str, Path]]:
|
||||
if messages.is_dir():
|
||||
listing = (messages / "tests.tsv").read_text().splitlines()
|
||||
return sorted(
|
||||
(name, messages / "tests" / file)
|
||||
for name, file in (line.split("\t") for line in listing)
|
||||
)
|
||||
found = {}
|
||||
for line in messages.read_text().splitlines():
|
||||
try:
|
||||
@@ -43,12 +53,43 @@ def tests_of(guest: Guest, exe: str) -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def upload_data(guest: Guest, tests: Path) -> None:
|
||||
"""Puts the data files the tests read (the schemas, fixtures) in the guest
|
||||
at the path the build baked into the executables (CARGO_MANIFEST_DIR, the
|
||||
build directory of the check, as a path on the guest's C: drive)."""
|
||||
if not (tests.is_dir() and (tests / "srcroot").exists()):
|
||||
return
|
||||
root = "C:" + (tests / "srcroot").read_text().strip().replace("/", "\\")
|
||||
data = tests / "data"
|
||||
files = sorted(p for p in data.rglob("*") if p.is_file())
|
||||
made: set[str] = set()
|
||||
for path in files:
|
||||
remote = root + "\\" + str(path.relative_to(data)).replace("/", "\\")
|
||||
directory = remote.rsplit("\\", 1)[0]
|
||||
if directory not in made:
|
||||
guest.agent.exec("cmd", "/c", f"mkdir {directory} 2>nul")
|
||||
made.add(directory)
|
||||
guest.agent.put_file(path, remote)
|
||||
print(f"data: {len(files)} files under {root}", flush=True)
|
||||
|
||||
|
||||
def reason(output: str) -> str:
|
||||
"""The lines that say why a test failed: the panic message, else the end."""
|
||||
lines = [l for l in output.splitlines() if l.strip()]
|
||||
for i, line in enumerate(lines):
|
||||
if "panicked at" in line:
|
||||
return " | ".join(lines[i : i + 2])
|
||||
return " | ".join(lines[-3:])
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
golden, out, state, messages = (Path(a) for a in argv[:4])
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
saved = ensure(golden, state) if str(state) != "-" else None
|
||||
results: dict[str, dict] = {}
|
||||
with Guest(golden, out / "guest", state=state if str(state) != "-" else None) as guest:
|
||||
with Guest(golden, out / "guest", state=saved) as guest:
|
||||
guest.agent.exec("cmd", "/c", f"mkdir {GUEST_DIR} 2>nul")
|
||||
upload_data(guest, messages)
|
||||
for name, path in executables(messages):
|
||||
exe = f"{GUEST_DIR}\\{name}.exe"
|
||||
began = time.monotonic()
|
||||
@@ -62,7 +103,7 @@ def main(argv: list[str]) -> int:
|
||||
else:
|
||||
failed.append({
|
||||
"test": test, "status": run["status"],
|
||||
"output": (run["stdout"] + run["stderr"])[-1500:],
|
||||
"output": (run["stdout"] + run["stderr"])[-4000:],
|
||||
})
|
||||
results[name] = {
|
||||
"tests": len(names), "passed": passed, "failed": failed,
|
||||
@@ -72,8 +113,7 @@ def main(argv: list[str]) -> int:
|
||||
f"{'' if not failed else f', {len(failed)} FAILED'}"
|
||||
f" ({results[name]['seconds']} s)", flush=True)
|
||||
for f in failed:
|
||||
first = [l for l in f["output"].splitlines() if l.strip()][-3:]
|
||||
print(f" FAIL {f['test']} (status {f['status']}): {' | '.join(first)}", flush=True)
|
||||
print(f" FAIL {f['test']} (status {f['status']}): {reason(f['output'])}", flush=True)
|
||||
(out / "cargo-tests.json").write_text(json.dumps(results, indent=2))
|
||||
total = sum(r["tests"] for r in results.values())
|
||||
failed = sum(len(r["failed"]) for r in results.values())
|
||||
|
||||
@@ -62,6 +62,25 @@ in
|
||||
"$E2E_GOLDEN" "$bins" "$out" "$@"
|
||||
'';
|
||||
};
|
||||
# the Windows unit tests in the guest: every test of every executable in a process of its own
|
||||
windowsNextest = pkgs.writeShellApplication {
|
||||
name = "windows-nextest";
|
||||
runtimeInputs = [
|
||||
pkgs.qemu
|
||||
pkgs.swtpm
|
||||
(pkgs.python3.withPackages (p: [ p.pillow ]))
|
||||
];
|
||||
text = ''
|
||||
: "''${E2E_GOLDEN:?set E2E_GOLDEN to the golden image directory}"
|
||||
out=''${1:?usage: windows-nextest OUT}
|
||||
export OVMF_FD=${ovmfWithTpm}
|
||||
export PYTHONPATH=${lib'.repoFile "e2e/lib"}:${lib'.repoFile "e2e"}
|
||||
exec python -m platforms.qemu_windows.cargo_tests \
|
||||
"$E2E_GOLDEN" "$out" "''${KOMOREBI_E2E_STATE:--}" ${windowsNextestBuild}
|
||||
'';
|
||||
};
|
||||
# the test executables of the Windows workspace, with a list of them (name, tab, file) for the
|
||||
# guest runner; the check `windows-nextest-build` and the app `windows-nextest` use it
|
||||
# the members that are stubs here (the Linux and macOS packages) have no test and do not build
|
||||
excluded = pkgs.lib.concatMapStringsSep " " (p: "--exclude ${p}") build.stubbedPackages;
|
||||
windowsNextestBuild = build.craneLib.mkCargoDerivation (
|
||||
@@ -69,7 +88,10 @@ in
|
||||
// {
|
||||
pnameSuffix = "-nextest-build";
|
||||
doInstallCargoArtifacts = false;
|
||||
nativeBuildInputs = build.individualCrateArgs.nativeBuildInputs ++ [ pkgs.jq ];
|
||||
nativeBuildInputs = build.individualCrateArgs.nativeBuildInputs ++ [
|
||||
pkgs.jq
|
||||
pkgs.cpio
|
||||
];
|
||||
buildPhaseCargoCommand = ''
|
||||
cargo test --locked --workspace ${excluded} --release --no-run --message-format json > messages.json
|
||||
'';
|
||||
@@ -81,6 +103,13 @@ in
|
||||
cp "$exe" "$out/tests/$name.exe"
|
||||
printf '%s\t%s.exe\n' "$name" "$name" >> $out/tests.tsv
|
||||
done
|
||||
# the data files the tests read by the absolute path the build baked into them
|
||||
# (CARGO_MANIFEST_DIR): the guest gets them at the same path under C:
|
||||
pwd > $out/srcroot
|
||||
mkdir -p $out/data
|
||||
find . -path ./target -prune -o -type f \
|
||||
\( -path './schemas/*' -o -path '*/tests/*' -o -path '*/fixtures/*' \) \
|
||||
! -name '*.rs' -print | cpio -pdm $out/data 2>/dev/null
|
||||
'';
|
||||
}
|
||||
);
|
||||
@@ -223,6 +252,12 @@ in
|
||||
type = "app";
|
||||
program = pkgs.lib.getExe e2eWindows;
|
||||
};
|
||||
# the unit tests of the Windows crates in the guest (E2E_GOLDEN as above):
|
||||
# E2E_GOLDEN=<golden> KOMOREBI_E2E_SCALE=1.5 nix run .#windows-nextest -- OUT
|
||||
windows-nextest = {
|
||||
type = "app";
|
||||
program = pkgs.lib.getExe windowsNextest;
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user