diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 14e76069..043dea8b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -55,6 +55,8 @@ the shell. The Windows code is type-checked from a Mac with `nix develop .#windows`, whose target is `x86_64-pc-windows-msvc`. Nothing there runs it: a change to a `cfg(windows)` line is first run by the `windows` job in CI. +On a Windows machine, the steps in [docs/dev/windows-checkpoints.md](docs/dev/windows-checkpoints.md) +say what to run and what to look for. ## Checks diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 00000000..b3158ace --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,115 @@ +# Design + +This document records why the recorder is built the way it is. It is organized as questions, because each answer is a +decision that could have gone another way. The facts in it were measured or read from the code, and where a reason is a +judgment it says so. It is written in the first person only where the decision was the owner's. + +## Why record the screen, the log, the events and the OS trace together? + +Most of the komorebi bugs I see show up when a laptop is docked or undocked, or wakes when the screen is opened. To +reproduce one, I had to watch the screen, hand-grep komorebi's log and export the system log from Console, and there was +no way to say: at 00:12.5 in the video, what was komorebi doing, and what was the OS doing? The recorder puts the four on +one clock so that the question has an answer, and so that the answer also works the other way, from a log line to the +moment in the video. + +OS-level tracing is in scope because the interesting seam is between komorebi and the system, and between the system and +the application being tiled; komorebi's own log only shows one side of each. + +## Why one clock, and why that one? + +Every record is stamped on the host's monotonic clock (`mach_absolute_time` on macOS, `QueryPerformanceCounter` on +Windows) because the two sources that cannot be re-stamped are already on it: ScreenCaptureKit frame timestamps and the +unified log's `machTimestamp` on a Mac, and Windows.Graphics.Capture frame times on Windows. komorebi's log lines carry +UTC wall-clock time, so they are mapped onto the master clock with paired readings of both clocks (anchors) sampled while +recording, interpolated piecewise so that a step or a sleep between two anchors does not smear the mapping. + +The alignment is as good as that allows: video to OS trace within about one frame, and video to komorebi's log within +about one frame plus the error of the wall-clock anchor, tens of microseconds. + +## Why are fields called `*_mach_ns` on Windows? + +They hold nanoseconds on the master clock, whichever clock that was. The spelling is historical and was kept: renaming +them would make a session written by one build unreadable by another. The manifest's `host.clock` says which clock a +session was on. + +## Why does the recorder never change komorebi's log level or lifecycle? + +That was decided at the start: the recorder only observes. It warns when komorebi is logging at a level that leaves out +most of what the window manager decided, and says how to restart it, but it never restarts komorebi or changes its +settings. (The reason for the decision is not written down anywhere the design could be checked against; this entry +needs the owner's words.) + +## Why does the accept loop never block, and why is an event stamped when its handler finished? + +komorebi connects to each subscriber synchronously and sends a notification while it holds its own locks, so a recorder +that was slow to read would stall komorebi. The accept thread stamps the arrival, reads, and hands the event to a channel; +a writer thread does everything else. For the same reason the per-event window probe (`--probe-windows`) runs on the +writer thread and never on the accept thread. + +komorebi sends the notification after it has handled the event, so the stamp is when the handler finished. The start of +the handler is the `processing event` line in komorebi's log, which is why `trace` pairs the two. + +## Why ScreenCaptureKit through `objc2` and not the `screencapturekit` crate? + +The `screencapturekit` crate needs a Swift bridge at build time, which complicates the nix build. `objc2-screen-capture-kit` +and its siblings exist at the versions the rest of komorebi already uses, so there is no Swift and no Xcode dependency. + +## Why fragmented mp4, and why re-submit the last frame? + +A recorder that is killed, or a machine that loses power, must not lose the recording, and a plain mp4 is unreadable +until it is finalized. Fragmented mp4 plays up to the last fragment. Two measured facts decided the rest: ScreenCaptureKit +only emits frames when the screen changes, and fragments are cut on media time, so a static display never flushed a +fragment; the capture therefore re-submits the last frame to keep it moving. And the duration `ffprobe` reports for a +truncated fragmented file under-reads, so recovery asks AVFoundation instead. + +## Why a new file for every segment? + +Docking and undocking, sleep and lock change what can be recorded, and I want the recording to carry on through all of +them without losing what was already captured. A display that disappears has its video finalized on the spot, so the file +is complete and playable whatever happens next, and a display that returns starts a new segment in a new file. Every +display's segments are listed in the manifest with their time spans, so a gap is a fact the session states rather than a +hole. + +## Why raw files plus an index, and not one file or a database? + +The raw files are written as the data arrives and are never duplicated or rewritten; `timeline.jsonl` only references byte +ranges in them, so the streams stay separate and can be reconciled or re-indexed later. The index can be rebuilt from the +raw files at any time (`reindex`), including after a crash and after a hand edit. + +## Why does reading a session never ask which host it is on? + +A session recorded on Windows is investigated on a Mac, and the other way round. The host and the komorebi flavor are +recorded in the manifest, and everything that reads a session asks the manifest. + +## Why can a bug only be marked in the viewer? + +The recorder window has no button for marking a bug, and the viewer has one. I found it confusing to have two places to do +the same thing, so a bug is marked in the viewer only. `komorebi-recorder annotate` writes the same annotations for a +script or an agent that has no window. + +## Why is the recorder window untitled? + +komorebi does not manage windows with an empty title, so a borderless untitled window is never tiled: opening the recorder +does not move the layout of the machine whose bug is being recorded. + +## Why is ffmpeg not part of the program? + +ffmpeg is used only to extract frames (`frames`) and to make test videos, and never for capture. It is in the dev shell and +not in the packaged binary. + +## Why is a komorebi source from a different commit "skewed" by files and not by commit? + +The recorder's own commits advance the repository's head without touching komorebi, so comparing commits calls almost +everything skewed. The files that differ between the commit a session recorded and the commit indexed are what say whether +a line of the log can still be traced to the statement that wrote it. + +## Why is the hidden-window check skipped on Windows? + +komorebi for Windows hides a window by cloaking it where it is, so a window's rectangle says nothing about whether it is +hidden. The check that flags a window parked off screen on the showing workspace holds only where komorebi hides windows +by moving them, so it is skipped on Windows, and `--probe-windows` reads the real state from Windows instead. + +## Why do the Windows rules say they are unverified? + +They were written from komorebi for Windows' source. Until a recording of it has been read, an analysis that uses them says +so; `Rules::verified` is the flag, and it is flipped once the first checks on a Windows machine confirm the rules. diff --git a/docs/PLATFORM.md b/docs/PLATFORM.md index cb6a7331..78efe185 100644 --- a/docs/PLATFORM.md +++ b/docs/PLATFORM.md @@ -19,6 +19,7 @@ that only read a session directory work on a Linux agent box; recording there sa Nothing that touches a real Windows desktop has been seen running: the Windows host was written from the Windows API documentation and komorebi for Windows' source, and every claim below about Windows says what the code does, not what was observed. +[windows-checkpoints.md](dev/windows-checkpoints.md) lists the checks that turn each into an observation. ## Vocabulary diff --git a/docs/dev/windows-checkpoints.md b/docs/dev/windows-checkpoints.md new file mode 100644 index 00000000..be29b7dd --- /dev/null +++ b/docs/dev/windows-checkpoints.md @@ -0,0 +1,214 @@ +# Windows checkpoints + +`komorebi-recorder` is built for Windows from a Mac (and, in CI, on Windows), so nothing that touches a real +Windows desktop has been seen running. Each checkpoint below is a short list of commands to run on a Windows PC or +VM, what to expect, and what to send back. The point of each is to replace something read in +komorebi's source with something observed. + +Send back, for any step that does not behave: the exact command, what it printed, the output of +`komorebi-recorder.exe doctor --json`, and the session directory if there is one. + +## Getting the binaries + +On the Mac, in this repository: + +```sh +nix build .#komorebi-recorder-windows --out-link result-windows +ls result-windows/bin +# komorebi-recorder.exe console build: use this one to run commands and read their output +# komorebi-recorder-gui.exe the same program without a console window (shortcuts, the viewer) +``` + +Copy `komorebi-recorder.exe` to the Windows machine. The `windows` job of CI builds the same executables natively and +uploads them, with the installer, as artifacts of the run. To check that the Windows code still type-checks +without building it: `nix flake check` (the `komorebi-recorder-windows-clippy` check) or +`nix develop .#windows --command cargo clippy --all-targets`. + +## W0: what komorebi for Windows really sends + +Status: **can be run now, and is the most useful one.** Reading komorebi for Windows' events, state +and logs was written from komorebi's source and has not been checked against a recording of it (the +analysis says so in a warning until it has); this is what checks it. + +1. Start komorebi for Windows with debug logging. The default of `RUST_LOG=debug` also prints every + dependency, so name the crate: + + ```powershell + komorebi.exe --log-level debug + ``` + + (or set `RUST_LOG=komorebi=debug` first). + +2. Describe the machine: + + ```powershell + komorebi-recorder.exe doctor + komorebi-recorder.exe doctor --json > doctor.json + ``` + + Expect: `komorebi for Windows`, `running: yes`, a binary path and version, `log level: debug`, + `answers a state query: yes`, the log directory under `%TEMP%` with a `komorebi_plaintext.log.` + in it, your displays with their scale, and `clock: Qpc`. `can record: yes` and `can decode: yes` + are what this build claims; whether they are true is W1 and W3. Anything under "things to know" + is worth reading. If the state query is + reported as answered with nothing, say what komorebi was doing (paused, another virtual desktop). + +3. Record 60 seconds without video, and during them: + + - focus a few windows of different programs (click, and with `komorebic.exe focus left|right`); + - drag a window to a new place, and drag-resize one; + - move windows with komorebic (`move`, `stack`, `unstack`, `focus-workspace 1` and back); + - open and close a program, and minimize and restore one; + - if you can: change the display scaling or resolution, connect or disconnect a monitor, switch to + another virtual desktop and back, lock (Win+L) and unlock once. + + ```powershell + komorebi-recorder.exe record --no-video --duration 60 --out C:\temp\w0 + ``` + + Expect a line per second with `events` counting up as you act, `komorebi log` growing and `OS + trace` counting up (WinEvents for every window, so it is busy while you work), then `session: + C:\temp\w0`. If the OS trace is too much or too little, say so: it is folded per window but its + volume on a real machine has not been seen. `--no-oslog` leaves it out. + +4. Look at it, to see what the current (macOS-vocabulary) analysis makes of it. It is expected to be + wrong in places, and what is wrong is the point: + + ```powershell + komorebi-recorder.exe brief C:\temp\w0 + komorebi-recorder.exe events C:\temp\w0 --json > events.json + ``` + +5. Send back `doctor.json` and the whole `C:\temp\w0` folder, zipped. It contains the titles and + program names of your windows (`events.jsonl`, `states\*.json`, `komorebi.log`); redact them first + if that matters (they only need to still be valid JSON, and the Windows fixtures are scrubbed by + hand anyway). + +Also worth one run, for the checks komorebi's state cannot make (it hides windows by cloaking them +in place): record with the probe on, and switch workspaces a few times (`komorebic.exe +focus-workspace 1`, `2`, `1`) while focusing windows. + +```powershell +komorebi-recorder.exe record --probe-windows --no-video --duration 60 --out C:\temp\w0-probe +komorebi-recorder.exe overview C:\temp\w0-probe +``` + +`overview` should report nothing about hidden windows if komorebi and Windows agree. If it reports +`visible-window-hidden` or `hidden-window-visible`, `komorebi-recorder.exe trace C:\temp\w0-probe +event:` shows what Windows said; say whether it is a real fault or a false alarm (a window mid +transition, say), and how many there were. The probe asks komorebi about a window by `DebugWindow`; +if komorebi stalls while it runs, say so. + +Optional, for the part of the plan that stops the recorder window being tiled: for one window komorebi +manages and one it does not (get handles from `komorebic.exe state`), run + +```powershell +komorebi-recorder.exe doctor --probe window --hwnd +``` + +and send what komorebi says. + +## W1: capture and the clock across a sleep + +Status: **can be run now.** Windows capture, the power watcher and the OS trace are built but have +never run on Windows; these are the first runs. + +1. The clock across a sleep (see above): `doctor --probe clock --seconds 120`, sleeping the machine + for about a minute while it runs. + +2. Capture on a static screen. Leave the screen completely still for most of it: + + ```powershell + komorebi-recorder.exe doctor --probe capture --seconds 20 + ``` + + Expect, per display, a segment with a handful of frames, `idle callbacks 0` (Windows says nothing + about a still screen), a file that `plays` for about the length recorded, and `latency` of a few + milliseconds to well under a second, `(NOT on the host clock)` absent. Check it with ffmpeg if you + have it: `ffprobe \video-.mp4`. + +3. Two monitors, if you have them: the same again, and say what it printed for each. + +4. Kill it and play the file. Start a longer one, then end it without letting it stop: + + ```powershell + komorebi-recorder.exe doctor --probe capture --seconds 60 + # after 15 s, from another window: + taskkill /F /IM komorebi-recorder.exe + ``` + + Then try to play `video-*.mp4` in the session folder (any player), and run + `komorebi-recorder.exe reindex `. This answers whether Media Foundation's fragmented MP4 + survives a kill (the part before the last fragment, about a second, should play) and whether + `reindex` can recover the session. Say what the player did and what `reindex` printed. + +## W2: a real recording, and the machine doing the things + +Status: can be run once W1 shows capture works. + +```powershell +komorebi-recorder.exe selftest --seconds 8 +komorebi-recorder.exe record --duration 60 --out C:\temp\w2 +komorebi-recorder.exe reindex C:\temp\w2 +``` + +`selftest` needs `ffmpeg` on PATH; expect `PASS` with the worst disagreement under 100 ms. During the +60 s recording: dock or undock a display, put the machine to sleep and wake it, lock and unlock +(Win+L), and close the lid if it is a laptop. Afterwards `komorebi-recorder.exe summary C:\temp\w2` +and `komorebi-recorder.exe at C:\temp\w2