diff --git a/docs/PLATFORM.md b/docs/PLATFORM.md index 9777476c..f8c4b861 100644 --- a/docs/PLATFORM.md +++ b/docs/PLATFORM.md @@ -4,8 +4,8 @@ This document gathers, in one place, what differs by operating system. Most of t that implement them; here the same questions are answered once per host, so that someone who has to run the recorder on a machine, or change it for one, does not have to read the code to find out. -The recorder is built for macOS and Windows. A third host, `unsupported`, builds on anything else so that the commands -that only read a session directory work on a Linux agent box; recording there says that it cannot. +The recorder is built for macOS, Windows and Linux (KDE Plasma). A fourth host, `unsupported`, builds on anything else so +that the commands that only read a session directory work there; recording says that it cannot. ## Status @@ -14,6 +14,7 @@ that only read a session directory work on a Linux agent box; recording there sa | macOS (Apple silicon) | yes | yes: written and run on macOS 26 | | Windows x86_64 | yes, natively on `windows-latest` and cross-compiled from a Mac | no: type-checked and cross-built only | | Windows ARM64 | yes, cross-built on an x86_64 runner | no | +| Linux (KDE Plasma, x86_64) | yes, natively | decoding, the komorebi side of recording and the OS trace were run, with komorebi for KDE's in-memory desktop; the screencast was not: see [kde-checkpoints.md](dev/kde-checkpoints.md) | | anything else | the `unsupported` host builds | nothing records | Nothing that touches a real Windows desktop has been seen running: the Windows host was written from the Windows API @@ -139,6 +140,43 @@ so there is never a second window to exclude. | icon | a slice of watermelon, which the Dock shows while a window is open (eframe sets it as the application's icon image). The program is a bare executable and not an app bundle, so Finder gives the file the generic executable icon | the same picture as the window and taskbar icon while a window is open, and compiled into both executables as a resource, which is what Explorer shows for the files; the installer shows it beside the program in Apps & features | | executables | `komorebi-recorder` | `komorebi-recorder.exe` (console: run commands with this one) and `komorebi-recorder-gui.exe` (no console window) | +## Linux (KDE Plasma) + +The Linux host records komorebi for KDE (`komorebi-for-kde`), on Plasma 6.8 or later, on Wayland. It is the +only Linux desktop the recorder knows: the screen comes from KWin's own screencast, and the windows from komorebi for +KDE. What is written from the sources and has not been seen to work on a real Plasma session is listed, one check each, in +[kde-checkpoints.md](dev/kde-checkpoints.md). + +| | Linux (KDE Plasma) | +|---|---| +| master clock | `CLOCK_MONOTONIC`, `host.clock` is `monotonic`; it does not count the time the machine is suspended, and `CLOCK_BOOTTIME`, which does, is the reference clock | +| sleep, wake and lock | heard on D-Bus: logind's `PrepareForSleep` on the system bus and the screen saver's `ActiveChanged` on the session bus; whether the screen is locked is `org.freedesktop.ScreenSaver.GetActive`. The lid and the displays' power are not read; the displays are polled | +| the displays | the compositor's outputs, over Wayland (`wl_output` and `xdg_output`), kept up to date by a thread that reads the compositor's events. A display's `id` is the FNV-1a hash of the output's name, which is komorebi for KDE's id for a monitor with no serial number | +| capturing the screen | KDE's `zkde_screencast_unstable_v1`, which gives a PipeWire node and asks the user nothing; `pipewiresrc` reads it, and the frames are placed in the file's time as on Windows | +| permission | KWin offers the screencast only to a program whose desktop file lists `zkde_screencast_unstable_v1` in `X-KDE-Wayland-Interfaces`. `doctor` prints the file to write, with the path of the executable in it | +| the video | H.264 only, written by GStreamer (`x264enc` and `mp4mux`) as fragmented MP4 in segments, as on the other hosts. HEVC is refused. A display of an odd size, or one H.264 cannot take, is scaled by the pipeline | +| decoding | GStreamer: `decodebin` to BGRA at the size asked for, seeking to the key frame before the time and decoding forward. A frame's time is its stream time, because an MP4's edit list puts the media time of the first frame after zero | +| the OS trace | the journal: `oslog/system.ndjson` (`journalctl --user`, of `kwin_wayland`, `plasmashell` and `komorebi`, stream `journal`) and `oslog/eventlog.ndjson` (every warning and worse, stream `journal-warnings`). An entry's time is `__MONOTONIC_TIMESTAMP`, which is the master clock | +| the window track | komorebi for KDE's own state: a Wayland client cannot list other programs' windows, so only windows komorebi manages are listed, each shown when its workspace is showing and it is in front of its stack | +| komorebi's sockets | `$XDG_RUNTIME_DIR/komorebi/` (the command socket `komorebi.sock` and the subscribers'), and the data directory where there is no runtime directory | +| komorebi's log | `~/.local/share/komorebi/komorebi_plaintext.log.`; the span of an event's handler is `handle_event` and of a command's `process_command` | +| the program's own windows | not left out of the video: the compositor's screencast of an output has every window on it. komorebi is told to ignore them by their application id, `komorebic ignore-rule class komorebi-recorder`, run before the window exists | +| the window's application id | the window's application id is `komorebi-recorder`, which is the `class` komorebi for KDE sees | +| `--probe-windows` | Windows only | +| ghost tiles | not found: they need the Windows probe or a picture of the screen read for them, and neither is written for KDE | + +komorebi for KDE's rules are checked against the log and the events of a running komorebi (`komorebi --fake-backend`, an +in-memory desktop with no windows), and against its source. They are not checked against a recording of a real +desktop, so an analysis of a KDE session says so, as it does for Windows. + +### What is needed on the machine + +- the GStreamer plugins that read PipeWire (`pipewire`), decode and encode H.264 (`gst-libav`, `gst-plugins-ugly` for + `x264enc`), and mux MP4 (`gst-plugins-good`); the Nix package puts them on the plugin path; +- `libwayland-client`, which is opened when it is needed, and what the window draws through (`libxkbcommon`, `libGL` or + Vulkan): the Nix package and the dev shell put them on `LD_LIBRARY_PATH`; +- `ffmpeg` for `frames` and `selftest`, and nothing else. + ## Degraded modes - `record --no-video` records everything but the screen, on the same clock. It asks the system nothing about displays and @@ -151,6 +189,8 @@ so there is never a second window to exclude. ## Advice by audience - **Recording on macOS.** Grant Screen Recording to the terminal, and start komorebi with `--log-level trace`. +- **Recording on Linux (KDE Plasma).** Install the desktop file `doctor` prints, so that KWin offers the screencast, and start + komorebi for KDE with `--log-level debug`. Use `record --no-video` where there is no screencast to be had. - **Recording on Windows.** There is nothing to grant. Use `komorebi-recorder.exe` for commands, since the GUI build prints nowhere, and start komorebi with `--log-level debug` and `RUST_LOG=komorebi=debug`. - **Investigating on any host.** A session is read the same everywhere. `ffmpeg` is needed only for `frames` and diff --git a/docs/dev/kde-checkpoints.md b/docs/dev/kde-checkpoints.md new file mode 100644 index 00000000..376fdea6 --- /dev/null +++ b/docs/dev/kde-checkpoints.md @@ -0,0 +1,97 @@ +# KDE checkpoints + +`komorebi-recorder` is built and tested on Linux, but with no KWin to talk to: what touches a real Plasma session was +written from KWin's and komorebi for KDE's sources and has not been seen running. Each checkpoint below is a short list +of commands to run on a Plasma 6.8 session that runs komorebi for KDE, what to expect, and what to send back. The point +of each is to replace something read in a source with something observed. + +Send back, for any step that does not behave: the exact command, what it printed, the output of +`komorebi-recorder doctor --json`, and the session directory if there is one. + +## What has been seen + +Against komorebi for KDE's in-memory desktop (`komorebi --fake-backend`, which has no windows) and the Wayland compositor +of WSLg (which has outputs and no screencast): + +- `doctor` names komorebi for KDE and finds its socket in `$XDG_RUNTIME_DIR/komorebi`; +- `record --no-video` records its log, its commands as events with the handler of each, and the journal in two streams, + and `events`, `summary` and `findings` read the session with the KDE rules; +- the displays are listed from the compositor, with a size and a position; +- a made-up source of frames is written to an H.264 file, which the decoder reads back, at the size of the display and + at an odd size. + +## Getting the binary + +```sh +nix build .#komorebi-recorder --out-link result +result/bin/komorebi-recorder doctor +``` + +The package wraps the program so that GStreamer finds its plugins (`gst-libav`, `x264enc`, the PipeWire source) and the +program finds `libwayland-client`. Outside Nix, install the same. + +## K0: what komorebi for KDE really sends + +Status: **can be run now, and is the most useful one.** The rules that read komorebi for KDE's events, `State` and log +were written from its source and checked against its log and commands, not against windows (the analysis says so in a +warning until they have been). + +1. Start komorebi for KDE with debug logging: `komorebi --log-level debug` (or `RUST_LOG=komorebi=debug`). +2. `komorebi-recorder doctor`. Expect `komorebi for KDE`, `running: yes`, the socket under `$XDG_RUNTIME_DIR/komorebi` + answering a state query, the log directory `~/.local/share/komorebi` with a `komorebi_plaintext.log.` in it, your + outputs, and `clock: Monotonic`. +3. Record 60 seconds with `--no-video`, and during them: open and close a few windows of different programs, focus + between them, drag one to a new place and drag-resize one, stack and unstack, switch workspaces + (`komorebic focus-workspace 1` and back), and lock and unlock the screen. +4. `komorebi-recorder events `. Expect each window event with its window id, program and notification + (`WindowAdded`, `InteractiveMoveFinished`, ...), each with a handler. Send back any event named `unreadable` or + classified `other`, any window event with no handler, and any `MoveEnd` that is not classified `drag`. + +## K1: the screencast is offered, and what it asks + +Status: not run. This is the one the Linux host depends on. + +1. `komorebi-recorder doctor`: under `capture`, `permitted: no` and the note with the desktop file. +2. Write `~/.local/share/applications/komorebi-recorder.desktop` as the note says (`Exec=` is the path of the executable + that runs), and start the recorder again. Expect `permitted: yes`. In the Nix package the program + is a wrapper that starts `.komorebi-recorder-wrapped`, and that is the executable the compositor sees, so it is the path + `doctor` prints (it is `current_exe`). +3. If it is still `no`: say where KWin looks for desktop files and how it matches the program to one. Whether a + desktop file in the user's `applications` directory is enough, whether `Exec=` has to be the executable's real path, + and whether a program started from a terminal is matched at all, are what this finds out. If it cannot be made to + work, the screencast portal (`xdg-desktop-portal`) is the way around it, behind the same `CaptureBackend`. + +## K2: recording the screen + +Status: not run. + +1. `komorebi-recorder displays`. Expect each output with its size and position in the layout (logical pixels). Compare the + ids with komorebi's: `komorebic monitor-information` (they agree for a monitor with no serial number). +2. `komorebi-recorder record --duration 20`, and move windows during it. Expect a `video-.mp4` for each output and + `frames-.csv`, and in the session summary no error. +3. Play it with `komorebi-recorder view `. Expect the picture to move with the windows, and the cursor in it. +4. `komorebi-recorder doctor --json`: the clock check (`latency`) of the segments should be near zero. A large number + means the frames' times are not on the master clock, which would be the PipeWire timestamps: send the numbers back. +5. Leave the screen still for ten seconds: the video should still advance (the last frame is repeated). +6. Unplug a monitor, or turn one off, if there is one to spare: the segment ends and a new one starts when it returns. +7. Suspend and resume: every output starts a new segment. + +## K3: the program's own window + +Status: not run. + +1. `komorebi-recorder` with no command opens the window. Expect it to be titled `komorebi-recorder` and not tiled. +2. If it was tiled, `komorebic ignore-rule class komorebi-recorder` was not applied in time, or the window's class is not + what is expected: `komorebic state` says what the window's class is. +3. The window is in the video of the screen. Whether that matters, and whether a screencast of the other windows only is + wanted, is open (`stream_window` of the protocol takes one window). + +## K4: the OS trace and the window track + +Status: the journal was followed on a machine with no Plasma. + +1. `record --duration 30`, and cause some noise: open a program that is not in the session, plug something in. + Expect `oslog/system.ndjson` to have `kwin_wayland` and `plasmashell` entries, and `oslog/eventlog.ndjson` the warnings. + A user who is not in the `systemd-journal` group may see none of the system's: the problems say so. +2. `record --window-track --duration 30`. Expect `windows.jsonl` with the windows komorebi manages, a window behind a + stack not shown, and a window of another workspace not shown.