mirror of
https://github.com/LGUG2Z/komorebi.git
synced 2026-11-18 22:58:34 +01:00
feat(assert): scripted pass/fail checks over a recording's findings, log, motion and notes
assert <session> --check "<kind> <args> <metric><op><number>" evaluates checks with the existing analyses (findings, logq, motion, blank, annotations), one Check trait and one registry line per kind. Exit 6 (ExitKind::Assertion) when a check fails; checks that cannot be measured are errors (2/3/4), never a pass. JSON envelope assert/1.
This commit is contained in:
1 parent
e7283ae45d
commit
13713d65e8
11 files changed
+1768
-6
No files matched your search
@@ -63,7 +63,10 @@ komorebi-recorder trace ./repro event:7 # one event, from the handler that ran
|
||||
```
|
||||
|
||||
A script that drives the repro can name its steps on the timeline while recording, with
|
||||
`komorebi-recorder mark "step 3: move window right"`; they show up as notes.
|
||||
`komorebi-recorder mark "step 3: move window right"`; they show up as notes. And the same script can end with a
|
||||
pass or fail: `komorebi-recorder assert ./repro --check "findings kind=layout-mismatch count<=0" --check "log \"level=ERROR\" count==0"`
|
||||
exits 0 when every check holds and 6 when one does not (checks over findings, the log, motion, blank regions and
|
||||
notes; see the guide).
|
||||
|
||||
For example, a four second recording of a Mac with komorebi running:
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ komorebi-recorder guide # the playbook, for an agent
|
||||
Every command takes `--json`, which prints an envelope `{ok, schema, data, warnings, truncated, next_cursor, error}`;
|
||||
`komorebi-recorder schema` lists the schemas and `komorebi-recorder schema brief/1` prints one. Exit codes: 0 ok, 1
|
||||
error, 2 usage, 3 not found, 4 the session cannot answer (still recording, or the recorder died: `reindex` it), 5 over
|
||||
budget.
|
||||
budget, 6 an `assert` check did not hold.
|
||||
|
||||
## The viewer
|
||||
|
||||
@@ -127,6 +127,7 @@ and `event:12.end` when it finished.
|
||||
| `framediff A B [--region ..] [--png FILE] [--against SESSION]` | How two frames differ: changed share, mean and peak brightness delta, the box of change, and a PNG with the changed pixels in red |
|
||||
| `blank --region x,y,w,h` | When a region was blank (one colour, named: `white`, `black`, a hex code) or frozen (identical to the frame before): "the window went white for 200 ms" |
|
||||
| `compare A B` | Two recordings side by side: duration, frames, effective fps, worst frame gap, motion, findings by kind, annotations, and the difference |
|
||||
| `assert --check "KIND ARGS METRIC<=N"` | Pass/fail checks for a scripted scenario, repeatable. Kinds: `findings kind=K [severity=S] count<=N`, `log "<logq query>" count==N`, `motion [region=x,y,w,h] [from=T] [to=T] segments\|active_ms\|peak_share<=N`, `blank region=x,y,w,h [from=T] [to=T] [color=white] ms<=N`, `annotations kind=note\|bug\|finding count>=N`. Operators `== != <= >= < >`. Prints `PASS`/`FAIL` per check; exit 0 if all pass, 6 if one fails, 4 if one cannot be measured (no video), 2 if malformed. `--json` gives `assert/1` |
|
||||
| `annotate`, `annotations` | Add, change and list bug regions, notes and findings |
|
||||
| `mark TEXT` | While recording: drop a named point note (author `script`) onto the timeline from a script; it then appears in `annotations`, `brief` and the viewer like any note |
|
||||
|
||||
|
||||
@@ -144,6 +144,31 @@ It marks the session named by `--session DIR`, else by the environment variable
|
||||
still running. A recording that has finished is refused: it has no "now" to mark; use `annotate <session> add` with a
|
||||
time instead. `--json` prints the `mark/1` envelope.
|
||||
|
||||
### Checking the result from a script
|
||||
|
||||
`assert` turns the recording into a pass or fail, so the same script that reproduces the bug can end by saying whether
|
||||
it happened. The exit status is the result: 0 when every check holds, 6 when one does not, 4 when one could not be
|
||||
measured (for example `motion` on a recording without video) and 2 when a check is malformed.
|
||||
|
||||
```
|
||||
komorebi-recorder record --until-exit --out ./repro -- ./repro.sh # repro.sh calls `komorebi-recorder mark` between steps
|
||||
komorebi-recorder assert ./repro --check "annotations kind=note count>=3" --check "findings kind=layout-mismatch count<=0" --check "log \"level=ERROR\" count==0" --check "blank region=0,0,800,600 color=white ms<=100"
|
||||
```
|
||||
|
||||
```
|
||||
PASS annotations kind=note count>=3 (observed 4)
|
||||
FAIL findings kind=layout-mismatch count<=0 (observed 2)
|
||||
f-3a9c1e invariant:layout-mismatch at 00:03.280: ...
|
||||
PASS log "level=ERROR" count==0 (observed 0)
|
||||
PASS blank region=0,0,800,600 color=white ms<=100 (observed 0)
|
||||
1 of 4 check(s): 1 failed
|
||||
```
|
||||
|
||||
The first check confirms the script's `mark`s were recorded. A check is a kind, its arguments and a comparison
|
||||
(`== != <= >= < >`); the kinds are `findings`, `log` (a `logq` query), `motion`, `blank` and `annotations`, and
|
||||
times in `from=` and `to=` are any time the other commands take, including `event:N` and `bug:<id>`. `--json` prints the
|
||||
`assert/1` envelope. `assert` only reads the session.
|
||||
|
||||
### Without video
|
||||
|
||||
`komorebi-recorder record --no-video` records everything but the screen: komorebi's events and log, the OS trace and
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -21,6 +21,7 @@
|
||||
//! | 3 | [`NotFound`](ExitKind::NotFound) | the session, or something asked for in it, does not exist |
|
||||
//! | 4 | [`Unusable`](ExitKind::Unusable) | the session exists but cannot answer: still recording, or it recorded nothing |
|
||||
//! | 5 | [`Budget`](ExitKind::Budget) | the answer would have exceeded the time or size budget |
|
||||
//! | 6 | [`Assertion`](ExitKind::Assertion) | `assert` measured the recording and a check did not hold |
|
||||
//!
|
||||
//! The codes are stable. [`ENVELOPE_VERSION`] is bumped when the envelope's own fields change;
|
||||
//! the shape of each command's `data` is versioned by its `schema` (`brief/1`), and the
|
||||
@@ -66,7 +67,8 @@ pub const ENVELOPE_VERSION: u32 = 1;
|
||||
|
||||
/// How the process exits: the kind of a failure, and the exit code that goes with it.
|
||||
///
|
||||
/// Serialized in kebab case (`ok`, `error`, `usage`, `not-found`, `unusable`, `budget`) in an
|
||||
/// Serialized in kebab case (`ok`, `error`, `usage`, `not-found`, `unusable`, `budget`,
|
||||
/// `assertion`) in an
|
||||
/// envelope's `error.kind`. [`ExitKind::code`] gives the exit code.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, JsonSchema)]
|
||||
#[serde(rename_all = "kebab-case")]
|
||||
@@ -86,10 +88,13 @@ pub enum ExitKind {
|
||||
/// The answer would have exceeded the time or size budget: exit code 5. Narrowing the request
|
||||
/// is the way forward.
|
||||
Budget,
|
||||
/// `assert` measured the recording and a check did not hold: exit code 6. Not a failure of the
|
||||
/// command: the answer is "no".
|
||||
Assertion,
|
||||
}
|
||||
|
||||
impl ExitKind {
|
||||
/// Returns the process exit code of this kind: 0 for [`ExitKind::Ok`], then 1 to 5 in the order
|
||||
/// Returns the process exit code of this kind: 0 for [`ExitKind::Ok`], then 1 to 6 in the order
|
||||
/// the variants are declared.
|
||||
///
|
||||
/// The mapping is stable, so a script may rely on the numbers (`exit_codes_are_stable` pins
|
||||
@@ -104,6 +109,7 @@ impl ExitKind {
|
||||
/// assert_eq!(ExitKind::Usage.code(), 2);
|
||||
/// assert_eq!(ExitKind::NotFound.code(), 3);
|
||||
/// assert_eq!(ExitKind::Budget.code(), 5);
|
||||
/// assert_eq!(ExitKind::Assertion.code(), 6);
|
||||
/// ```
|
||||
#[must_use]
|
||||
pub fn code(self) -> u8 {
|
||||
@@ -114,6 +120,7 @@ impl ExitKind {
|
||||
Self::NotFound => 3,
|
||||
Self::Unusable => 4,
|
||||
Self::Budget => 5,
|
||||
Self::Assertion => 6,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -248,6 +255,32 @@ pub fn exit_kind(report: &eyre::Report) -> ExitKind {
|
||||
.map_or(ExitKind::Error, |error| error.kind)
|
||||
}
|
||||
|
||||
/// Marks a failure whose answer the command has already printed, so that
|
||||
/// `crate::cli::execute` only sets the exit code and prints nothing more (no second envelope).
|
||||
///
|
||||
/// Made by [`reported`]; the kind is still read with [`exit_kind`].
|
||||
#[derive(Debug)]
|
||||
pub struct Reported;
|
||||
|
||||
impl fmt::Display for Reported {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
f.write_str("the answer was already printed")
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for Reported {}
|
||||
|
||||
/// Returns an error of this kind whose answer the command has already printed (see [`Reported`]).
|
||||
pub fn reported(kind: ExitKind, message: impl Into<String>) -> eyre::Report {
|
||||
failure(kind, message).wrap_err(Reported)
|
||||
}
|
||||
|
||||
/// Returns whether `report` was made by [`reported`].
|
||||
#[must_use]
|
||||
pub fn is_reported(report: &eyre::Report) -> bool {
|
||||
report.downcast_ref::<Reported>().is_some()
|
||||
}
|
||||
|
||||
/// The `error` of a failed [`Envelope`]: the kind of failure and what to do about it.
|
||||
#[derive(Debug, Serialize, JsonSchema)]
|
||||
pub struct ErrorBody {
|
||||
@@ -332,6 +365,21 @@ impl<T> Envelope<T> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the envelope as a failed one that still carries its `data`: `ok` is `false` and
|
||||
/// `error` has `kind` and its exit code. For a command whose answer is the point even when it
|
||||
/// is "no" (`assert`).
|
||||
#[must_use]
|
||||
pub fn with_error(mut self, kind: ExitKind, message: impl Into<String>) -> Self {
|
||||
self.ok = false;
|
||||
self.error = Some(ErrorBody {
|
||||
kind,
|
||||
code: kind.code(),
|
||||
message: message.into(),
|
||||
hint: None,
|
||||
});
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the envelope with `warnings` as its warnings. They replace any that were there.
|
||||
#[must_use]
|
||||
pub fn warnings(mut self, warnings: Vec<String>) -> Self {
|
||||
@@ -415,11 +463,12 @@ mod tests {
|
||||
ExitKind::NotFound,
|
||||
ExitKind::Unusable,
|
||||
ExitKind::Budget,
|
||||
ExitKind::Assertion,
|
||||
]
|
||||
.iter()
|
||||
.map(|kind| kind.code())
|
||||
.collect();
|
||||
assert_eq!(codes, [0, 1, 2, 3, 4, 5]);
|
||||
assert_eq!(codes, [0, 1, 2, 3, 4, 5, 6]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
//! bound ([`budget`]).
|
||||
|
||||
pub mod answers;
|
||||
pub mod assertions;
|
||||
pub mod blank;
|
||||
pub mod blocks;
|
||||
pub mod brief;
|
||||
|
||||
@@ -174,6 +174,11 @@ pub const ALL: &[Entry] = &[
|
||||
about: "`findings`: detector findings (with stable ids) and an agent's findings, together",
|
||||
schema: of::<super::findings::FindingsData>,
|
||||
},
|
||||
Entry {
|
||||
name: "assert/1",
|
||||
about: "`assert`: pass, fail or error for each scripted check, with what was observed",
|
||||
schema: of::<super::assertions::AssertData>,
|
||||
},
|
||||
Entry {
|
||||
name: "record/1",
|
||||
about: "`record --json`: the finished session, its displays and frames, warnings and the --until-exit command's status",
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
//! `assert`: scripted pass/fail checks over a recording, for an agent's own scenario tests.
|
||||
//!
|
||||
//! Each `--check` is a kind, its arguments and a comparison (see
|
||||
//! [`crate::analysis::assertions`] for the grammar and the kinds). The answer is a line per check
|
||||
//! and a summary; with `--json` it is the `assert/1` envelope. The exit code is the result: 0 when
|
||||
//! every check passes, 6 when one does not hold, and 2, 3 or 4 when the checks are malformed or
|
||||
//! cannot be measured. Read-only: it never writes to the session.
|
||||
//!
|
||||
//! Covered by `tests/assert.rs`, which runs the command through the real binary.
|
||||
|
||||
use super::print_json;
|
||||
use crate::analysis::Analysis;
|
||||
use crate::analysis::assertions;
|
||||
use crate::analysis::assertions::SCHEMA_ASSERT;
|
||||
use crate::analysis::envelope::Envelope;
|
||||
use crate::analysis::envelope::ExitKind;
|
||||
use crate::analysis::envelope::reported;
|
||||
use clap::Args;
|
||||
use color_eyre::eyre;
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// The arguments of `assert`.
|
||||
#[derive(Args, Debug)]
|
||||
pub struct AssertArgs {
|
||||
/// Session directory
|
||||
pub session: PathBuf,
|
||||
/// A check, `<kind> <args...> <metric><op><number>` (repeatable): `findings kind=layout-mismatch
|
||||
/// count<=0`, `log "level=ERROR" count==0`, `motion region=0,0,800,600 segments<=2`,
|
||||
/// `blank region=0,0,800,600 color=white ms<=100`, `annotations kind=note count>=1`. Operators:
|
||||
/// == != <= >= < >. Exits 0 if all pass, 6 if one fails
|
||||
#[arg(long = "check", value_name = "CHECK", required = true)]
|
||||
pub checks: Vec<String>,
|
||||
/// Machine-readable JSON output
|
||||
#[arg(long)]
|
||||
pub json: bool,
|
||||
}
|
||||
|
||||
pub(super) fn assert(args: &AssertArgs) -> eyre::Result<()> {
|
||||
// read every check before opening the session: a typo is a usage error whatever the session is
|
||||
let parsed = assertions::parse_all(&args.checks)?;
|
||||
let analysis = Analysis::open(&args.session)?;
|
||||
let data = assertions::run_parsed(&analysis, &parsed);
|
||||
|
||||
let kind = data.exit_kind();
|
||||
let summary = data.summary();
|
||||
if args.json {
|
||||
let mut envelope = Envelope::ok(SCHEMA_ASSERT, data).warnings(analysis.warnings());
|
||||
if kind != ExitKind::Ok {
|
||||
envelope = envelope.with_error(kind, summary.clone());
|
||||
}
|
||||
print_json(&envelope)?;
|
||||
} else {
|
||||
for warning in analysis.warnings() {
|
||||
eprintln!("note: {warning}");
|
||||
}
|
||||
print!("{}", data.to_text());
|
||||
}
|
||||
|
||||
match kind {
|
||||
ExitKind::Ok => Ok(()),
|
||||
kind => Err(reported(kind, summary)),
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@
|
||||
|
||||
mod agent;
|
||||
mod annotate;
|
||||
mod assert;
|
||||
mod doctor;
|
||||
mod findings;
|
||||
mod investigate;
|
||||
@@ -38,6 +39,7 @@ pub use agent::SchemaArgs;
|
||||
pub use agent::SearchArgs;
|
||||
pub use annotate::AnnotateArgs;
|
||||
pub use annotate::AnnotationsArgs;
|
||||
pub use assert::AssertArgs;
|
||||
pub use doctor::DoctorArgs;
|
||||
pub use findings::FindingsArgs;
|
||||
pub use investigate::CodeArgs;
|
||||
@@ -148,6 +150,8 @@ pub enum Command {
|
||||
Annotate(AnnotateArgs),
|
||||
/// List a session's bug regions, notes and findings
|
||||
Annotations(AnnotationsArgs),
|
||||
/// Pass/fail checks over a recording (findings, log, motion, blank, notes); exit 6 if one fails
|
||||
Assert(AssertArgs),
|
||||
/// Drop a named marker onto the timeline of a recording that is still going
|
||||
Mark(mark::MarkArgs),
|
||||
/// Rebuild a session's timeline from its raw files, recovering it first if the recorder died
|
||||
@@ -226,6 +230,7 @@ impl Command {
|
||||
Self::Annotate(args) => args.json(),
|
||||
Self::Annotations(args) => args.json,
|
||||
Self::Mark(args) => args.json,
|
||||
Self::Assert(args) => args.json,
|
||||
Self::Doctor(args) => args.json,
|
||||
Self::Record(args) => args.json,
|
||||
Self::Displays(args) => args.json,
|
||||
@@ -248,6 +253,10 @@ pub fn execute(command: Command) -> std::process::ExitCode {
|
||||
Ok(()) => std::process::ExitCode::SUCCESS,
|
||||
Err(error) => {
|
||||
let kind = crate::analysis::envelope::exit_kind(&error);
|
||||
if crate::analysis::envelope::is_reported(&error) {
|
||||
// the command printed its own answer; only the exit status is left to set
|
||||
return std::process::ExitCode::from(kind.code());
|
||||
}
|
||||
if json {
|
||||
let envelope = crate::analysis::envelope::Envelope::failed(&error);
|
||||
if let Ok(text) = serde_json::to_string_pretty(&envelope) {
|
||||
@@ -299,6 +308,7 @@ pub fn run(command: Command) -> eyre::Result<()> {
|
||||
Command::Annotate(args) => annotate::annotate(&args),
|
||||
Command::Annotations(args) => annotate::list(&args),
|
||||
Command::Mark(args) => mark::mark(&args),
|
||||
Command::Assert(args) => assert::assert(&args),
|
||||
Command::Reindex(args) => reindex(&args),
|
||||
Command::Selftest(args) => crate::selftest::run(&args),
|
||||
Command::SelftestClock => crate::selftest::run_clock(),
|
||||
|
||||
@@ -10,7 +10,7 @@ what *produces* the bug the user described, from the recording, and then propose
|
||||
what it shows. Every command takes `--json`; its output is an envelope
|
||||
`{ok, schema, data, warnings, truncated, next_cursor, error}` whose `data` is described by
|
||||
`komorebi-recorder schema <schema>`. Exit codes: 0 ok, 1 error, 2 usage, 3 not found, 4 unusable
|
||||
session, 5 over budget.
|
||||
session, 5 over budget, 6 an `assert` check failed.
|
||||
|
||||
## 1. Start with the brief
|
||||
|
||||
@@ -130,6 +130,18 @@ the command prints the line it resolved to, so check it is the statement you mea
|
||||
text lists the candidates and is refused. Never claim a cause without a reference,
|
||||
and say what you could not verify.
|
||||
|
||||
### Scripted checks
|
||||
|
||||
To turn a scenario into pass or fail, `komorebi-recorder assert <session> --check "<check>" ...` evaluates checks with
|
||||
the same analyses and answers with its exit code (0 all pass, 6 one failed, 4 one could not be measured, 2 malformed).
|
||||
A check is `<kind> <args> <metric><op><number>`, with `== != <= >= < >`:
|
||||
|
||||
findings kind=layout-mismatch count<=0
|
||||
log "level=ERROR and message~\"ghost\"" count==0 (the logq language)
|
||||
motion region=0,0,800,600 from=bug:a-1.start segments<=2 (also active_ms, peak_share)
|
||||
blank region=0,0,800,600 color=white ms<=100
|
||||
annotations kind=note count>=3 (did the script's `mark`s land?)
|
||||
|
||||
## 5. Propose the fix
|
||||
|
||||
Name the file and function from `code`. Explain the chain from what the user saw to the state to the
|
||||
|
||||
@@ -0,0 +1,349 @@
|
||||
//! `assert` through the real binary: checks over findings, the log, annotations, motion and blank
|
||||
//! regions, and what the exit code says.
|
||||
|
||||
use komorebi_recorder::testkit;
|
||||
use komorebi_recorder::testkit::SessionBuilder;
|
||||
use serde_json::Value;
|
||||
use std::process::Command;
|
||||
use std::process::Output;
|
||||
|
||||
fn recorder(args: &[&str]) -> Output {
|
||||
Command::new(env!("CARGO_BIN_EXE_komorebi-recorder"))
|
||||
.args(args)
|
||||
.output()
|
||||
.expect("run komorebi-recorder")
|
||||
}
|
||||
|
||||
fn stdout(output: &Output) -> String {
|
||||
String::from_utf8_lossy(&output.stdout).into_owned()
|
||||
}
|
||||
|
||||
fn stderr(output: &Output) -> String {
|
||||
String::from_utf8_lossy(&output.stderr).into_owned()
|
||||
}
|
||||
|
||||
/// A session with an error and a warning in komorebi's log, and one note.
|
||||
fn session(name: &str) -> String {
|
||||
let (session, _) = SessionBuilder::new(name)
|
||||
.log(2.0, "INFO", "komorebi::process_event: processing event")
|
||||
.log(
|
||||
3.0,
|
||||
"ERROR",
|
||||
"komorebi::window: could not place window 4711",
|
||||
)
|
||||
.log(4.0, "WARN", "komorebi::reaper: window 12 is gone")
|
||||
.build();
|
||||
let dir = session.dir.display().to_string();
|
||||
|
||||
let added = recorder(&[
|
||||
"annotate", &dir, "add", "--kind", "note", "--start", "2.5", "--text", "step one",
|
||||
]);
|
||||
assert!(added.status.success(), "{}", stderr(&added));
|
||||
dir
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn passing_checks_exit_zero_and_print_a_line_each() {
|
||||
let dir = session("assert-pass");
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"log \"level=ERROR\" count==1",
|
||||
"--check",
|
||||
"log message~\"window 12\" count>=1",
|
||||
"--check",
|
||||
"findings kind=layout-mismatch count<=0",
|
||||
"--check",
|
||||
"annotations kind=note count>=1",
|
||||
"--check",
|
||||
"annotations kind=bug count==0",
|
||||
]);
|
||||
assert_eq!(output.status.code(), Some(0), "{}", stderr(&output));
|
||||
|
||||
let text = stdout(&output);
|
||||
let lines: Vec<&str> = text.lines().collect();
|
||||
assert_eq!(lines.len(), 6, "{text}");
|
||||
assert_eq!(lines[0], "PASS log \"level=ERROR\" count==1 (observed 1)");
|
||||
assert_eq!(
|
||||
lines[2],
|
||||
"PASS findings kind=layout-mismatch count<=0 (observed 0)"
|
||||
);
|
||||
assert_eq!(lines[5], "all 5 check(s) passed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_failing_check_exits_six_and_is_named() {
|
||||
let dir = session("assert-fail");
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"log \"level=ERROR\" count==0",
|
||||
"--check",
|
||||
"findings kind=log-error count==1",
|
||||
"--check",
|
||||
"annotations kind=note count>=3",
|
||||
]);
|
||||
assert_eq!(output.status.code(), Some(6), "{}", stderr(&output));
|
||||
|
||||
let text = stdout(&output);
|
||||
assert!(
|
||||
text.contains("FAIL log \"level=ERROR\" count==0 (observed 1)"),
|
||||
"{text}"
|
||||
);
|
||||
// the line that broke it is shown under the failure
|
||||
assert!(text.contains("could not place window 4711"), "{text}");
|
||||
assert!(
|
||||
text.contains("PASS findings kind=log-error count==1 (observed 1)"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("FAIL annotations kind=note count>=3 (observed 1)"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(text.contains("2 of 3 check(s): 2 failed"), "{text}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn json_is_an_assert_envelope_and_the_exit_code_still_says_the_result() {
|
||||
let dir = session("assert-json");
|
||||
|
||||
let passing = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"log \"level=WARN\" count==1",
|
||||
"--json",
|
||||
]);
|
||||
assert_eq!(passing.status.code(), Some(0));
|
||||
let envelope: Value = serde_json::from_slice(&passing.stdout).expect("JSON");
|
||||
assert_eq!(envelope["ok"], true);
|
||||
assert_eq!(envelope["schema"], "assert/1");
|
||||
assert_eq!(envelope["data"]["passed"], true);
|
||||
assert_eq!(envelope["data"]["checks"][0]["passed"], true);
|
||||
assert_eq!(envelope["data"]["checks"][0]["observed"], 1);
|
||||
assert_eq!(
|
||||
envelope["data"]["checks"][0]["check"],
|
||||
"log \"level=WARN\" count==1"
|
||||
);
|
||||
|
||||
let failing = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"log \"level=ERROR\" count==0",
|
||||
"--check",
|
||||
"log \"level=INFO\" count==1",
|
||||
"--json",
|
||||
]);
|
||||
assert_eq!(failing.status.code(), Some(6));
|
||||
// one document, not an answer followed by an error envelope
|
||||
let envelope: Value = serde_json::from_slice(&failing.stdout).expect("one JSON document");
|
||||
assert_eq!(envelope["ok"], false);
|
||||
assert_eq!(envelope["schema"], "assert/1");
|
||||
assert_eq!(envelope["error"]["kind"], "assertion");
|
||||
assert_eq!(envelope["error"]["code"], 6);
|
||||
assert_eq!(envelope["data"]["passed"], false);
|
||||
assert_eq!(envelope["data"]["failed"], 1);
|
||||
let checks = envelope["data"]["checks"].as_array().expect("checks");
|
||||
assert_eq!(checks[0]["passed"], false);
|
||||
assert_eq!(checks[0]["status"], "fail");
|
||||
assert_eq!(checks[0]["observed"], 1);
|
||||
assert!(
|
||||
checks[0]["detail"]
|
||||
.as_str()
|
||||
.expect("detail")
|
||||
.contains("4711")
|
||||
);
|
||||
assert_eq!(checks[1]["passed"], true);
|
||||
|
||||
let schema = recorder(&["schema", "assert/1"]);
|
||||
assert!(schema.status.success(), "{}", stderr(&schema));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_checks_are_usage_errors_that_list_the_kinds() {
|
||||
let dir = session("assert-usage");
|
||||
|
||||
for bad in [
|
||||
"nonsense count==1",
|
||||
"findings",
|
||||
"findings count=1",
|
||||
"findings count<=lots",
|
||||
"log count==0",
|
||||
"motion bogus=1 segments==0",
|
||||
"blank segments==0",
|
||||
] {
|
||||
let output = recorder(&["assert", &dir, "--check", bad]);
|
||||
assert_eq!(output.status.code(), Some(2), "{bad}: {}", stderr(&output));
|
||||
}
|
||||
|
||||
let output = recorder(&["assert", &dir, "--check", "nonsense count==1"]);
|
||||
let message = stderr(&output);
|
||||
for kind in ["findings", "log", "motion", "blank", "annotations"] {
|
||||
assert!(message.contains(kind), "{message}");
|
||||
}
|
||||
|
||||
// a bad logq query is a usage error of that check, not a pass; so is no check at all
|
||||
let output = recorder(&["assert", &dir, "--check", "log \"level=\" count==0"]);
|
||||
assert_eq!(output.status.code(), Some(2), "{}", stderr(&output));
|
||||
assert_eq!(recorder(&["assert", &dir]).status.code(), Some(2));
|
||||
|
||||
// a typo in one check measures nothing, whatever the others would say
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"log \"level=ERROR\" count==1",
|
||||
"--check",
|
||||
"oops count==0",
|
||||
]);
|
||||
assert_eq!(output.status.code(), Some(2));
|
||||
assert_eq!(stdout(&output), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_session_that_does_not_exist_is_not_found() {
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
"no-such-session-dir",
|
||||
"--check",
|
||||
"findings count==0",
|
||||
]);
|
||||
assert_eq!(output.status.code(), Some(3), "{}", stderr(&output));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_check_that_cannot_be_measured_is_an_error_not_a_pass() {
|
||||
// no video was recorded, so a motion check has nothing to look at
|
||||
let dir = session("assert-no-video");
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"annotations count>=1",
|
||||
"--check",
|
||||
"motion segments<=100",
|
||||
"--json",
|
||||
]);
|
||||
assert_eq!(output.status.code(), Some(4), "{}", stderr(&output));
|
||||
let envelope: Value = serde_json::from_slice(&output.stdout).expect("JSON");
|
||||
assert_eq!(envelope["ok"], false);
|
||||
assert_eq!(envelope["error"]["kind"], "unusable");
|
||||
assert_eq!(envelope["data"]["passed"], false);
|
||||
assert_eq!(envelope["data"]["errors"], 1);
|
||||
assert_eq!(envelope["data"]["checks"][0]["status"], "pass");
|
||||
let motion = &envelope["data"]["checks"][1];
|
||||
assert_eq!(motion["status"], "error");
|
||||
assert_eq!(motion["passed"], false);
|
||||
assert!(
|
||||
motion["detail"]
|
||||
.as_str()
|
||||
.expect("detail")
|
||||
.contains("no video")
|
||||
);
|
||||
|
||||
let text = recorder(&["assert", &dir, "--check", "motion segments<=100"]);
|
||||
assert_eq!(text.status.code(), Some(4));
|
||||
assert!(
|
||||
stdout(&text).starts_with("ERROR motion segments<=100"),
|
||||
"{}",
|
||||
stdout(&text)
|
||||
);
|
||||
}
|
||||
|
||||
fn walk(dir: &std::path::Path) -> Vec<(String, u64)> {
|
||||
let mut out = Vec::new();
|
||||
for entry in std::fs::read_dir(dir).expect("read dir").flatten() {
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
out.extend(walk(&path));
|
||||
} else {
|
||||
out.push((
|
||||
path.display().to_string(),
|
||||
entry.metadata().map_or(0, |meta| meta.len()),
|
||||
));
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assert_leaves_the_session_as_it_found_it() {
|
||||
let dir = session("assert-readonly");
|
||||
let before = walk(std::path::Path::new(&dir));
|
||||
let _ = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"findings count>=0",
|
||||
"--check",
|
||||
"log \"level=ERROR\" count==1",
|
||||
]);
|
||||
assert_eq!(walk(std::path::Path::new(&dir)), before);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn motion_and_blank_checks_measure_the_video() {
|
||||
if !testkit::ffmpeg_available() {
|
||||
eprintln!("skipped: ffmpeg is not available to make a test video");
|
||||
return;
|
||||
}
|
||||
|
||||
// 3 s at 30 fps: counting for a second, still for half a second, white for half a second,
|
||||
// counting again
|
||||
let (session, _) = SessionBuilder::new("assert-video")
|
||||
.video_showing(1, 0.0, 90, 30, |frame| match frame {
|
||||
0..=29 => frame,
|
||||
30..=44 => 29,
|
||||
45..=59 => 0xFFFF,
|
||||
_ => frame,
|
||||
})
|
||||
.build();
|
||||
let dir = session.dir.display().to_string();
|
||||
|
||||
let output = recorder(&[
|
||||
"assert",
|
||||
&dir,
|
||||
"--check",
|
||||
"motion region=500,0,500,800 segments>=2",
|
||||
"--check",
|
||||
"motion region=0,0,500,800 to=1.4 segments==0",
|
||||
"--check",
|
||||
"motion from=1.1 to=1.4 active_ms<=100",
|
||||
"--check",
|
||||
"blank region=0,0,1000,800 color=white ms>=300",
|
||||
"--check",
|
||||
"blank region=0,0,1000,800 color=white ms<=100",
|
||||
"--check",
|
||||
"blank region=0,0,1000,800 from=0.5 color=black stretches==0",
|
||||
]);
|
||||
let text = stdout(&output);
|
||||
assert_eq!(output.status.code(), Some(6), "{text}{}", stderr(&output));
|
||||
let lines: Vec<&str> = text.lines().collect();
|
||||
assert!(
|
||||
lines[0].starts_with("PASS motion region=500,0,500,800 segments>=2"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
lines[1].starts_with("PASS motion region=0,0,500,800 to=1.4 segments==0"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(lines[2].starts_with("PASS motion from=1.1"), "{text}");
|
||||
assert!(lines[3].starts_with("PASS blank"), "{text}");
|
||||
assert!(lines[4].starts_with("FAIL blank"), "{text}");
|
||||
assert!(lines[6].starts_with("PASS blank"), "{text}");
|
||||
|
||||
let answer: Value = serde_json::from_slice(
|
||||
&recorder(&["assert", &dir, "--check", "motion peak_share>0", "--json"]).stdout,
|
||||
)
|
||||
.expect("JSON");
|
||||
assert_eq!(answer["data"]["passed"], true);
|
||||
assert!(answer["data"]["checks"][0]["observed"].is_number());
|
||||
|
||||
// times are TimeSpecs; one that does not exist is that check's error
|
||||
let output = recorder(&["assert", &dir, "--check", "motion from=event:9 segments==0"]);
|
||||
assert_eq!(output.status.code(), Some(3), "{}", stdout(&output));
|
||||
}
|
||||
Reference in new issue
Block a user