feat(assert): scripted pass/fail checks over a recording's findings, log, motion and notes

assert <session> --check "<kind> <args> <metric><op><number>" evaluates checks
with the existing analyses (findings, logq, motion, blank, annotations), one
Check trait and one registry line per kind. Exit 6 (ExitKind::Assertion) when a
check fails; checks that cannot be measured are errors (2/3/4), never a pass.
JSON envelope assert/1.
This commit is contained in:
LGUG2Z committed 2026-10-28 16:24:29 -07:00
1 parent e7283ae45d
commit 13713d65e8
11 files changed
+1768 -6

No files matched your search

+4 -1
View File
@@ -63,7 +63,10 @@ komorebi-recorder trace ./repro event:7 # one event, from the handler that ran
```
A script that drives the repro can name its steps on the timeline while recording, with
`komorebi-recorder mark "step 3: move window right"`; they show up as notes.
`komorebi-recorder mark "step 3: move window right"`; they show up as notes. And the same script can end with a
pass or fail: `komorebi-recorder assert ./repro --check "findings kind=layout-mismatch count<=0" --check "log \"level=ERROR\" count==0"`
exits 0 when every check holds and 6 when one does not (checks over findings, the log, motion, blank regions and
notes; see the guide).
For example, a four second recording of a Mac with komorebi running:
+2 -1
View File
@@ -14,7 +14,7 @@ komorebi-recorder guide # the playbook, for an agent
Every command takes `--json`, which prints an envelope `{ok, schema, data, warnings, truncated, next_cursor, error}`;
`komorebi-recorder schema` lists the schemas and `komorebi-recorder schema brief/1` prints one. Exit codes: 0 ok, 1
error, 2 usage, 3 not found, 4 the session cannot answer (still recording, or the recorder died: `reindex` it), 5 over
budget.
budget, 6 an `assert` check did not hold.
## The viewer
@@ -127,6 +127,7 @@ and `event:12.end` when it finished.
| `framediff A B [--region ..] [--png FILE] [--against SESSION]` | How two frames differ: changed share, mean and peak brightness delta, the box of change, and a PNG with the changed pixels in red |
| `blank --region x,y,w,h` | When a region was blank (one colour, named: `white`, `black`, a hex code) or frozen (identical to the frame before): "the window went white for 200 ms" |
| `compare A B` | Two recordings side by side: duration, frames, effective fps, worst frame gap, motion, findings by kind, annotations, and the difference |
| `assert --check "KIND ARGS METRIC<=N"` | Pass/fail checks for a scripted scenario, repeatable. Kinds: `findings kind=K [severity=S] count<=N`, `log "<logq query>" count==N`, `motion [region=x,y,w,h] [from=T] [to=T] segments\|active_ms\|peak_share<=N`, `blank region=x,y,w,h [from=T] [to=T] [color=white] ms<=N`, `annotations kind=note\|bug\|finding count>=N`. Operators `== != <= >= < >`. Prints `PASS`/`FAIL` per check; exit 0 if all pass, 6 if one fails, 4 if one cannot be measured (no video), 2 if malformed. `--json` gives `assert/1` |
| `annotate`, `annotations` | Add, change and list bug regions, notes and findings |
| `mark TEXT` | While recording: drop a named point note (author `script`) onto the timeline from a script; it then appears in `annotations`, `brief` and the viewer like any note |
+25
View File
@@ -144,6 +144,31 @@ It marks the session named by `--session DIR`, else by the environment variable
still running. A recording that has finished is refused: it has no "now" to mark; use `annotate <session> add` with a
time instead. `--json` prints the `mark/1` envelope.
### Checking the result from a script
`assert` turns the recording into a pass or fail, so the same script that reproduces the bug can end by saying whether
it happened. The exit status is the result: 0 when every check holds, 6 when one does not, 4 when one could not be
measured (for example `motion` on a recording without video) and 2 when a check is malformed.
```
komorebi-recorder record --until-exit --out ./repro -- ./repro.sh # repro.sh calls `komorebi-recorder mark` between steps
komorebi-recorder assert ./repro --check "annotations kind=note count>=3" --check "findings kind=layout-mismatch count<=0" --check "log \"level=ERROR\" count==0" --check "blank region=0,0,800,600 color=white ms<=100"
```
```
PASS annotations kind=note count>=3 (observed 4)
FAIL findings kind=layout-mismatch count<=0 (observed 2)
f-3a9c1e invariant:layout-mismatch at 00:03.280: ...
PASS log "level=ERROR" count==0 (observed 0)
PASS blank region=0,0,800,600 color=white ms<=100 (observed 0)
1 of 4 check(s): 1 failed
```
The first check confirms the script's `mark`s were recorded. A check is a kind, its arguments and a comparison
(`== != <= >= < >`); the kinds are `findings`, `log` (a `logq` query), `motion`, `blank` and `annotations`, and
times in `from=` and `to=` are any time the other commands take, including `event:N` and `bug:<id>`. `--json` prints the
`assert/1` envelope. `assert` only reads the session.
### Without video
`komorebi-recorder record --no-video` records everything but the screen: komorebi's events and log, the OS trace and
File diff suppressed because it is too large. Load diff
+52 -3
View File
@@ -21,6 +21,7 @@
//! | 3 | [`NotFound`](ExitKind::NotFound) | the session, or something asked for in it, does not exist |
//! | 4 | [`Unusable`](ExitKind::Unusable) | the session exists but cannot answer: still recording, or it recorded nothing |
//! | 5 | [`Budget`](ExitKind::Budget) | the answer would have exceeded the time or size budget |
//! | 6 | [`Assertion`](ExitKind::Assertion) | `assert` measured the recording and a check did not hold |
//!
//! The codes are stable. [`ENVELOPE_VERSION`] is bumped when the envelope's own fields change;
//! the shape of each command's `data` is versioned by its `schema` (`brief/1`), and the
@@ -66,7 +67,8 @@ pub const ENVELOPE_VERSION: u32 = 1;
/// How the process exits: the kind of a failure, and the exit code that goes with it.
///
/// Serialized in kebab case (`ok`, `error`, `usage`, `not-found`, `unusable`, `budget`) in an
/// Serialized in kebab case (`ok`, `error`, `usage`, `not-found`, `unusable`, `budget`,
/// `assertion`) in an
/// envelope's `error.kind`. [`ExitKind::code`] gives the exit code.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, JsonSchema)]
#[serde(rename_all = "kebab-case")]
@@ -86,10 +88,13 @@ pub enum ExitKind {
/// The answer would have exceeded the time or size budget: exit code 5. Narrowing the request
/// is the way forward.
Budget,
/// `assert` measured the recording and a check did not hold: exit code 6. Not a failure of the
/// command: the answer is "no".
Assertion,
}
impl ExitKind {
/// Returns the process exit code of this kind: 0 for [`ExitKind::Ok`], then 1 to 5 in the order
/// Returns the process exit code of this kind: 0 for [`ExitKind::Ok`], then 1 to 6 in the order
/// the variants are declared.
///
/// The mapping is stable, so a script may rely on the numbers (`exit_codes_are_stable` pins
@@ -104,6 +109,7 @@ impl ExitKind {
/// assert_eq!(ExitKind::Usage.code(), 2);
/// assert_eq!(ExitKind::NotFound.code(), 3);
/// assert_eq!(ExitKind::Budget.code(), 5);
/// assert_eq!(ExitKind::Assertion.code(), 6);
/// ```
#[must_use]
pub fn code(self) -> u8 {
@@ -114,6 +120,7 @@ impl ExitKind {
Self::NotFound => 3,
Self::Unusable => 4,
Self::Budget => 5,
Self::Assertion => 6,
}
}
}
@@ -248,6 +255,32 @@ pub fn exit_kind(report: &eyre::Report) -> ExitKind {
.map_or(ExitKind::Error, |error| error.kind)
}
/// Marks a failure whose answer the command has already printed, so that
/// `crate::cli::execute` only sets the exit code and prints nothing more (no second envelope).
///
/// Made by [`reported`]; the kind is still read with [`exit_kind`].
#[derive(Debug)]
pub struct Reported;
impl fmt::Display for Reported {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str("the answer was already printed")
}
}
impl std::error::Error for Reported {}
/// Returns an error of this kind whose answer the command has already printed (see [`Reported`]).
pub fn reported(kind: ExitKind, message: impl Into<String>) -> eyre::Report {
failure(kind, message).wrap_err(Reported)
}
/// Returns whether `report` was made by [`reported`].
#[must_use]
pub fn is_reported(report: &eyre::Report) -> bool {
report.downcast_ref::<Reported>().is_some()
}
/// The `error` of a failed [`Envelope`]: the kind of failure and what to do about it.
#[derive(Debug, Serialize, JsonSchema)]
pub struct ErrorBody {
@@ -332,6 +365,21 @@ impl<T> Envelope<T> {
}
}
/// Returns the envelope as a failed one that still carries its `data`: `ok` is `false` and
/// `error` has `kind` and its exit code. For a command whose answer is the point even when it
/// is "no" (`assert`).
#[must_use]
pub fn with_error(mut self, kind: ExitKind, message: impl Into<String>) -> Self {
self.ok = false;
self.error = Some(ErrorBody {
kind,
code: kind.code(),
message: message.into(),
hint: None,
});
self
}
/// Returns the envelope with `warnings` as its warnings. They replace any that were there.
#[must_use]
pub fn warnings(mut self, warnings: Vec<String>) -> Self {
@@ -415,11 +463,12 @@ mod tests {
ExitKind::NotFound,
ExitKind::Unusable,
ExitKind::Budget,
ExitKind::Assertion,
]
.iter()
.map(|kind| kind.code())
.collect();
assert_eq!(codes, [0, 1, 2, 3, 4, 5]);
assert_eq!(codes, [0, 1, 2, 3, 4, 5, 6]);
}
#[test]
+1
View File
@@ -8,6 +8,7 @@
//! bound ([`budget`]).
pub mod answers;
pub mod assertions;
pub mod blank;
pub mod blocks;
pub mod brief;
@@ -174,6 +174,11 @@ pub const ALL: &[Entry] = &[
about: "`findings`: detector findings (with stable ids) and an agent's findings, together",
schema: of::<super::findings::FindingsData>,
},
Entry {
name: "assert/1",
about: "`assert`: pass, fail or error for each scripted check, with what was observed",
schema: of::<super::assertions::AssertData>,
},
Entry {
name: "record/1",
about: "`record --json`: the finished session, its displays and frames, warnings and the --until-exit command's status",
+63
View File
@@ -0,0 +1,63 @@
//! `assert`: scripted pass/fail checks over a recording, for an agent's own scenario tests.
//!
//! Each `--check` is a kind, its arguments and a comparison (see
//! [`crate::analysis::assertions`] for the grammar and the kinds). The answer is a line per check
//! and a summary; with `--json` it is the `assert/1` envelope. The exit code is the result: 0 when
//! every check passes, 6 when one does not hold, and 2, 3 or 4 when the checks are malformed or
//! cannot be measured. Read-only: it never writes to the session.
//!
//! Covered by `tests/assert.rs`, which runs the command through the real binary.
use super::print_json;
use crate::analysis::Analysis;
use crate::analysis::assertions;
use crate::analysis::assertions::SCHEMA_ASSERT;
use crate::analysis::envelope::Envelope;
use crate::analysis::envelope::ExitKind;
use crate::analysis::envelope::reported;
use clap::Args;
use color_eyre::eyre;
use std::path::PathBuf;
/// The arguments of `assert`.
#[derive(Args, Debug)]
pub struct AssertArgs {
/// Session directory
pub session: PathBuf,
/// A check, `<kind> <args...> <metric><op><number>` (repeatable): `findings kind=layout-mismatch
/// count<=0`, `log "level=ERROR" count==0`, `motion region=0,0,800,600 segments<=2`,
/// `blank region=0,0,800,600 color=white ms<=100`, `annotations kind=note count>=1`. Operators:
/// == != <= >= < >. Exits 0 if all pass, 6 if one fails
#[arg(long = "check", value_name = "CHECK", required = true)]
pub checks: Vec<String>,
/// Machine-readable JSON output
#[arg(long)]
pub json: bool,
}
pub(super) fn assert(args: &AssertArgs) -> eyre::Result<()> {
// read every check before opening the session: a typo is a usage error whatever the session is
let parsed = assertions::parse_all(&args.checks)?;
let analysis = Analysis::open(&args.session)?;
let data = assertions::run_parsed(&analysis, &parsed);
let kind = data.exit_kind();
let summary = data.summary();
if args.json {
let mut envelope = Envelope::ok(SCHEMA_ASSERT, data).warnings(analysis.warnings());
if kind != ExitKind::Ok {
envelope = envelope.with_error(kind, summary.clone());
}
print_json(&envelope)?;
} else {
for warning in analysis.warnings() {
eprintln!("note: {warning}");
}
print!("{}", data.to_text());
}
match kind {
ExitKind::Ok => Ok(()),
kind => Err(reported(kind, summary)),
}
}
+10
View File
@@ -7,6 +7,7 @@
mod agent;
mod annotate;
mod assert;
mod doctor;
mod findings;
mod investigate;
@@ -38,6 +39,7 @@ pub use agent::SchemaArgs;
pub use agent::SearchArgs;
pub use annotate::AnnotateArgs;
pub use annotate::AnnotationsArgs;
pub use assert::AssertArgs;
pub use doctor::DoctorArgs;
pub use findings::FindingsArgs;
pub use investigate::CodeArgs;
@@ -148,6 +150,8 @@ pub enum Command {
Annotate(AnnotateArgs),
/// List a session's bug regions, notes and findings
Annotations(AnnotationsArgs),
/// Pass/fail checks over a recording (findings, log, motion, blank, notes); exit 6 if one fails
Assert(AssertArgs),
/// Drop a named marker onto the timeline of a recording that is still going
Mark(mark::MarkArgs),
/// Rebuild a session's timeline from its raw files, recovering it first if the recorder died
@@ -226,6 +230,7 @@ impl Command {
Self::Annotate(args) => args.json(),
Self::Annotations(args) => args.json,
Self::Mark(args) => args.json,
Self::Assert(args) => args.json,
Self::Doctor(args) => args.json,
Self::Record(args) => args.json,
Self::Displays(args) => args.json,
@@ -248,6 +253,10 @@ pub fn execute(command: Command) -> std::process::ExitCode {
Ok(()) => std::process::ExitCode::SUCCESS,
Err(error) => {
let kind = crate::analysis::envelope::exit_kind(&error);
if crate::analysis::envelope::is_reported(&error) {
// the command printed its own answer; only the exit status is left to set
return std::process::ExitCode::from(kind.code());
}
if json {
let envelope = crate::analysis::envelope::Envelope::failed(&error);
if let Ok(text) = serde_json::to_string_pretty(&envelope) {
@@ -299,6 +308,7 @@ pub fn run(command: Command) -> eyre::Result<()> {
Command::Annotate(args) => annotate::annotate(&args),
Command::Annotations(args) => annotate::list(&args),
Command::Mark(args) => mark::mark(&args),
Command::Assert(args) => assert::assert(&args),
Command::Reindex(args) => reindex(&args),
Command::Selftest(args) => crate::selftest::run(&args),
Command::SelftestClock => crate::selftest::run_clock(),
+13 -1
View File
@@ -10,7 +10,7 @@ what *produces* the bug the user described, from the recording, and then propose
what it shows. Every command takes `--json`; its output is an envelope
`{ok, schema, data, warnings, truncated, next_cursor, error}` whose `data` is described by
`komorebi-recorder schema <schema>`. Exit codes: 0 ok, 1 error, 2 usage, 3 not found, 4 unusable
session, 5 over budget.
session, 5 over budget, 6 an `assert` check failed.
## 1. Start with the brief
@@ -130,6 +130,18 @@ the command prints the line it resolved to, so check it is the statement you mea
text lists the candidates and is refused. Never claim a cause without a reference,
and say what you could not verify.
### Scripted checks
To turn a scenario into pass or fail, `komorebi-recorder assert <session> --check "<check>" ...` evaluates checks with
the same analyses and answers with its exit code (0 all pass, 6 one failed, 4 one could not be measured, 2 malformed).
A check is `<kind> <args> <metric><op><number>`, with `== != <= >= < >`:
findings kind=layout-mismatch count<=0
log "level=ERROR and message~\"ghost\"" count==0 (the logq language)
motion region=0,0,800,600 from=bug:a-1.start segments<=2 (also active_ms, peak_share)
blank region=0,0,800,600 color=white ms<=100
annotations kind=note count>=3 (did the script's `mark`s land?)
## 5. Propose the fix
Name the file and function from `code`. Explain the chain from what the user saw to the state to the
+349
View File
@@ -0,0 +1,349 @@
//! `assert` through the real binary: checks over findings, the log, annotations, motion and blank
//! regions, and what the exit code says.
use komorebi_recorder::testkit;
use komorebi_recorder::testkit::SessionBuilder;
use serde_json::Value;
use std::process::Command;
use std::process::Output;
fn recorder(args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_komorebi-recorder"))
.args(args)
.output()
.expect("run komorebi-recorder")
}
fn stdout(output: &Output) -> String {
String::from_utf8_lossy(&output.stdout).into_owned()
}
fn stderr(output: &Output) -> String {
String::from_utf8_lossy(&output.stderr).into_owned()
}
/// A session with an error and a warning in komorebi's log, and one note.
fn session(name: &str) -> String {
let (session, _) = SessionBuilder::new(name)
.log(2.0, "INFO", "komorebi::process_event: processing event")
.log(
3.0,
"ERROR",
"komorebi::window: could not place window 4711",
)
.log(4.0, "WARN", "komorebi::reaper: window 12 is gone")
.build();
let dir = session.dir.display().to_string();
let added = recorder(&[
"annotate", &dir, "add", "--kind", "note", "--start", "2.5", "--text", "step one",
]);
assert!(added.status.success(), "{}", stderr(&added));
dir
}
#[test]
fn passing_checks_exit_zero_and_print_a_line_each() {
let dir = session("assert-pass");
let output = recorder(&[
"assert",
&dir,
"--check",
"log \"level=ERROR\" count==1",
"--check",
"log message~\"window 12\" count>=1",
"--check",
"findings kind=layout-mismatch count<=0",
"--check",
"annotations kind=note count>=1",
"--check",
"annotations kind=bug count==0",
]);
assert_eq!(output.status.code(), Some(0), "{}", stderr(&output));
let text = stdout(&output);
let lines: Vec<&str> = text.lines().collect();
assert_eq!(lines.len(), 6, "{text}");
assert_eq!(lines[0], "PASS log \"level=ERROR\" count==1 (observed 1)");
assert_eq!(
lines[2],
"PASS findings kind=layout-mismatch count<=0 (observed 0)"
);
assert_eq!(lines[5], "all 5 check(s) passed");
}
#[test]
fn a_failing_check_exits_six_and_is_named() {
let dir = session("assert-fail");
let output = recorder(&[
"assert",
&dir,
"--check",
"log \"level=ERROR\" count==0",
"--check",
"findings kind=log-error count==1",
"--check",
"annotations kind=note count>=3",
]);
assert_eq!(output.status.code(), Some(6), "{}", stderr(&output));
let text = stdout(&output);
assert!(
text.contains("FAIL log \"level=ERROR\" count==0 (observed 1)"),
"{text}"
);
// the line that broke it is shown under the failure
assert!(text.contains("could not place window 4711"), "{text}");
assert!(
text.contains("PASS findings kind=log-error count==1 (observed 1)"),
"{text}"
);
assert!(
text.contains("FAIL annotations kind=note count>=3 (observed 1)"),
"{text}"
);
assert!(text.contains("2 of 3 check(s): 2 failed"), "{text}");
}
#[test]
fn json_is_an_assert_envelope_and_the_exit_code_still_says_the_result() {
let dir = session("assert-json");
let passing = recorder(&[
"assert",
&dir,
"--check",
"log \"level=WARN\" count==1",
"--json",
]);
assert_eq!(passing.status.code(), Some(0));
let envelope: Value = serde_json::from_slice(&passing.stdout).expect("JSON");
assert_eq!(envelope["ok"], true);
assert_eq!(envelope["schema"], "assert/1");
assert_eq!(envelope["data"]["passed"], true);
assert_eq!(envelope["data"]["checks"][0]["passed"], true);
assert_eq!(envelope["data"]["checks"][0]["observed"], 1);
assert_eq!(
envelope["data"]["checks"][0]["check"],
"log \"level=WARN\" count==1"
);
let failing = recorder(&[
"assert",
&dir,
"--check",
"log \"level=ERROR\" count==0",
"--check",
"log \"level=INFO\" count==1",
"--json",
]);
assert_eq!(failing.status.code(), Some(6));
// one document, not an answer followed by an error envelope
let envelope: Value = serde_json::from_slice(&failing.stdout).expect("one JSON document");
assert_eq!(envelope["ok"], false);
assert_eq!(envelope["schema"], "assert/1");
assert_eq!(envelope["error"]["kind"], "assertion");
assert_eq!(envelope["error"]["code"], 6);
assert_eq!(envelope["data"]["passed"], false);
assert_eq!(envelope["data"]["failed"], 1);
let checks = envelope["data"]["checks"].as_array().expect("checks");
assert_eq!(checks[0]["passed"], false);
assert_eq!(checks[0]["status"], "fail");
assert_eq!(checks[0]["observed"], 1);
assert!(
checks[0]["detail"]
.as_str()
.expect("detail")
.contains("4711")
);
assert_eq!(checks[1]["passed"], true);
let schema = recorder(&["schema", "assert/1"]);
assert!(schema.status.success(), "{}", stderr(&schema));
}
#[test]
fn malformed_checks_are_usage_errors_that_list_the_kinds() {
let dir = session("assert-usage");
for bad in [
"nonsense count==1",
"findings",
"findings count=1",
"findings count<=lots",
"log count==0",
"motion bogus=1 segments==0",
"blank segments==0",
] {
let output = recorder(&["assert", &dir, "--check", bad]);
assert_eq!(output.status.code(), Some(2), "{bad}: {}", stderr(&output));
}
let output = recorder(&["assert", &dir, "--check", "nonsense count==1"]);
let message = stderr(&output);
for kind in ["findings", "log", "motion", "blank", "annotations"] {
assert!(message.contains(kind), "{message}");
}
// a bad logq query is a usage error of that check, not a pass; so is no check at all
let output = recorder(&["assert", &dir, "--check", "log \"level=\" count==0"]);
assert_eq!(output.status.code(), Some(2), "{}", stderr(&output));
assert_eq!(recorder(&["assert", &dir]).status.code(), Some(2));
// a typo in one check measures nothing, whatever the others would say
let output = recorder(&[
"assert",
&dir,
"--check",
"log \"level=ERROR\" count==1",
"--check",
"oops count==0",
]);
assert_eq!(output.status.code(), Some(2));
assert_eq!(stdout(&output), "");
}
#[test]
fn a_session_that_does_not_exist_is_not_found() {
let output = recorder(&[
"assert",
"no-such-session-dir",
"--check",
"findings count==0",
]);
assert_eq!(output.status.code(), Some(3), "{}", stderr(&output));
}
#[test]
fn a_check_that_cannot_be_measured_is_an_error_not_a_pass() {
// no video was recorded, so a motion check has nothing to look at
let dir = session("assert-no-video");
let output = recorder(&[
"assert",
&dir,
"--check",
"annotations count>=1",
"--check",
"motion segments<=100",
"--json",
]);
assert_eq!(output.status.code(), Some(4), "{}", stderr(&output));
let envelope: Value = serde_json::from_slice(&output.stdout).expect("JSON");
assert_eq!(envelope["ok"], false);
assert_eq!(envelope["error"]["kind"], "unusable");
assert_eq!(envelope["data"]["passed"], false);
assert_eq!(envelope["data"]["errors"], 1);
assert_eq!(envelope["data"]["checks"][0]["status"], "pass");
let motion = &envelope["data"]["checks"][1];
assert_eq!(motion["status"], "error");
assert_eq!(motion["passed"], false);
assert!(
motion["detail"]
.as_str()
.expect("detail")
.contains("no video")
);
let text = recorder(&["assert", &dir, "--check", "motion segments<=100"]);
assert_eq!(text.status.code(), Some(4));
assert!(
stdout(&text).starts_with("ERROR motion segments<=100"),
"{}",
stdout(&text)
);
}
fn walk(dir: &std::path::Path) -> Vec<(String, u64)> {
let mut out = Vec::new();
for entry in std::fs::read_dir(dir).expect("read dir").flatten() {
let path = entry.path();
if path.is_dir() {
out.extend(walk(&path));
} else {
out.push((
path.display().to_string(),
entry.metadata().map_or(0, |meta| meta.len()),
));
}
}
out.sort();
out
}
#[test]
fn assert_leaves_the_session_as_it_found_it() {
let dir = session("assert-readonly");
let before = walk(std::path::Path::new(&dir));
let _ = recorder(&[
"assert",
&dir,
"--check",
"findings count>=0",
"--check",
"log \"level=ERROR\" count==1",
]);
assert_eq!(walk(std::path::Path::new(&dir)), before);
}
#[test]
fn motion_and_blank_checks_measure_the_video() {
if !testkit::ffmpeg_available() {
eprintln!("skipped: ffmpeg is not available to make a test video");
return;
}
// 3 s at 30 fps: counting for a second, still for half a second, white for half a second,
// counting again
let (session, _) = SessionBuilder::new("assert-video")
.video_showing(1, 0.0, 90, 30, |frame| match frame {
0..=29 => frame,
30..=44 => 29,
45..=59 => 0xFFFF,
_ => frame,
})
.build();
let dir = session.dir.display().to_string();
let output = recorder(&[
"assert",
&dir,
"--check",
"motion region=500,0,500,800 segments>=2",
"--check",
"motion region=0,0,500,800 to=1.4 segments==0",
"--check",
"motion from=1.1 to=1.4 active_ms<=100",
"--check",
"blank region=0,0,1000,800 color=white ms>=300",
"--check",
"blank region=0,0,1000,800 color=white ms<=100",
"--check",
"blank region=0,0,1000,800 from=0.5 color=black stretches==0",
]);
let text = stdout(&output);
assert_eq!(output.status.code(), Some(6), "{text}{}", stderr(&output));
let lines: Vec<&str> = text.lines().collect();
assert!(
lines[0].starts_with("PASS motion region=500,0,500,800 segments>=2"),
"{text}"
);
assert!(
lines[1].starts_with("PASS motion region=0,0,500,800 to=1.4 segments==0"),
"{text}"
);
assert!(lines[2].starts_with("PASS motion from=1.1"), "{text}");
assert!(lines[3].starts_with("PASS blank"), "{text}");
assert!(lines[4].starts_with("FAIL blank"), "{text}");
assert!(lines[6].starts_with("PASS blank"), "{text}");
let answer: Value = serde_json::from_slice(
&recorder(&["assert", &dir, "--check", "motion peak_share>0", "--json"]).stdout,
)
.expect("JSON");
assert_eq!(answer["data"]["passed"], true);
assert!(answer["data"]["checks"][0]["observed"].is_number());
// times are TimeSpecs; one that does not exist is that check's error
let output = recorder(&["assert", &dir, "--check", "motion from=event:9 segments==0"]);
assert_eq!(output.status.code(), Some(3), "{}", stdout(&output));
}