docs: drop the stale "just test exit code" caveat

`just test` has enforced the suite result since #350: the recipe raises unless
`evalTests` is `true`, so a zero exit already means pass. Update the places that
still described the old raw-`nix eval` behaviour: AGENTS.md, README.md,
SECURITY.md, outputs/README.md, and the nix-config-debug / nix-config-new-host /
nix-config-update skills.

The exit-0 caveat is kept only where the raw `nix eval .#evalTests` command is
documented, because that command genuinely exits 0 even on `false`.
This commit is contained in:
Ryan Yin committed 2026-10-07 19:54:32 +08:00
1 parent 6f136b50e4
commit 124bcd8490
8 files changed
+17 -11

No files matched your search

+3 -2
View File
@@ -26,7 +26,7 @@ bump, and the commands that destroy rollback points are in the `nix-config-updat
| Layer | Symptom | Look with |
| ------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| eval | `just test` prints `false`, an eval error | `just eval-host <host>` (already passes `--show-trace`) |
| eval | `just test` fails, an eval error | `just eval-host <host>` (already passes `--show-trace`) |
| build | build error, hash mismatch, "marked as broken" | `just build-host <host>`, then `nix log <drv>` |
| activation | the deploy fails after building | the deploy output; `journalctl -u home-manager-$USER -b` for Home Manager |
| runtime | a unit is failed or restarting | `just list-failed`, `systemctl status <unit>`, `journalctl -u <unit> -b` |
@@ -39,7 +39,8 @@ bump, and the commands that destroy rollback points are in the `nix-config-updat
## 2. Read the evaluated value
When `just test` prints `false`, re-run the suite directly to get the trace:
When `just test` fails, re-run the suite directly to get the trace (`nix eval` exits 0 even when the
suite returns `false`, so read its output):
```bash
nix eval .#evalTests --show-trace
+1 -1
View File
@@ -79,7 +79,7 @@ check every configuration and which list hosts by name:
- `security-*`, `kernel`, `nix-system-features`: apply to every configuration automatically.
- `home-manager`, `btrbk`, and the other host-listing tests: add the host if it should be covered.
`just test` must print `true`; an exit code of `0` with `false` is a failure.
`just test` fails (non-zero) unless the suite returns `true`.
## 4. Install and deploy
+1 -1
View File
@@ -75,7 +75,7 @@ Treat an update as a supply-chain event.
Run these before any host is touched:
- `just test` — must print `true`. An exit code of `0` with `false` is a failed suite.
- `just test` — fails (non-zero) unless the suite returns `true`.
- `just eval-host <host>` — fast, evaluation only.
- `just build-host <host>` — builds the full system closure and catches broken packages or build
failures that eval misses.
+3 -2
View File
@@ -38,8 +38,9 @@ deployments. Keep repository guidance here; reusable global rules live in `agent
`nixfmt` with width 100.
- For supported non-Nix files, use `prettier --write <file>` and `prettier --check <file>`;
configuration lives in `.prettierrc.yaml`. Spelling checks use `typos` and `.typos.toml`.
- Run `just test` for configuration changes. It evaluates `.#evalTests` across Linux and Darwin; the
output must be `true`. Exit code zero with `false` is a failed suite.
- Run `just test` for configuration changes; it evaluates `.#evalTests` across Linux and Darwin and
exits non-zero unless the result is `true`. The bare `nix eval .#evalTests` exits 0 even on
`false`, so run `just test` or compare its `--json` output yourself.
- Eval tests are `expr.nix` / `expected.nix` pairs under `outputs/<system>/tests/`. Update focused
cases when changing behavior covered by those tests.
- Use `nix flake check` for broader flake checks. A host build can validate changes beyond eval:
+1 -1
View File
@@ -159,7 +159,7 @@ of the same flake, e.g. `just col <tag>`, `just k3s-test`, `just lab`.
`nix develop` provides the formatters and linters used by the repository. The most useful commands:
```bash
just test # eval tests across Linux & Darwin; the output must be `true`
just test # eval tests across Linux & Darwin (fails unless the result is `true`)
just fmt # format all Nix files with nixfmt
just --list # all recipes
+2 -2
View File
@@ -109,8 +109,8 @@ See [application hardening](hardening/README.md) for wrapper details.
1. Check the advisory and affected package/kernel; prioritize active exploitation and reachable
attack paths. Review changed inputs and cache trust instead of treating updates as inherently
safe.
2. Run `just test` (must return `true`), build affected systems/runners and review closure
differences. Evaluate tests cannot prove runtime enforcement or workload compatibility.
2. Run `just test` (must pass), build affected systems/runners and review closure differences.
Evaluate tests cannot prove runtime enforcement or workload compatibility.
3. Record the old system generation/runner and ensure backups and recovery access exist. Use `boot`
mode for bridge/network-stack changes or broad VM-host updates.
4. Deploy MicroVM closures to the **physical host**, not the guest's read-only store. Install the
@@ -106,7 +106,7 @@ document is the agreed design and the rationale.
## Validation
- `just fmt`; `just test` -> `true`; `nix build` the youko toplevel.
- `just fmt`; `just test` passes; `nix build` the youko toplevel.
- Postgres: `\dx` lists `vectorchord`/`pgvector`; `SHOW shared_preload_libraries` includes
`vchord.so`; the `immich` DB exists.
- Units: `immich-server`, `immich-machine-learning`, `redis-shared`, `postgresql` are active.
+5 -1
View File
@@ -31,9 +31,13 @@ set for each NixOS and nix-darwin host.
How to run all the eval tests:
```bash
nix eval .#evalTests --show-trace --print-build-logs --verbose
just test
```
`just test` exits non-zero unless the suite returns `true`. The underlying
`nix eval .#evalTests --show-trace --print-build-logs --verbose` prints the result but exits 0 even
on `false`, so read its output when you need the failing trace.
Each test is a directory `outputs/<system>/tests/<name>/` holding `expr.nix` and `expected.nix`; the
suite passes when every `expr` equals its `expected`.