On a host whose GPU can encode H.264 (a Windows driver through d3d12 on WSL 2 does), KRDP in
the demo guest encoded on the guest's CPUs. The host side was fine (vainfo on the host lists
H.264 and HEVC EncSlice), and virgl can pass a guest's VA-API video to it, but a chain of seven
problems stood between the two. Each is a patch, and the guest takes them all:
QEMU (nix/dev/qemu-virgl-video.patch)
QEMU never asks virglrenderer for video: virglrenderer turns it on only for an embedder that
passes VIRGL_RENDERER_USE_VIDEO and a get_drm_fd callback to open the host's video device on.
The flag is defined only with VIRGL_RENDERER_UNSTABLE_APIS, which the build sets.
virglrenderer 1.3.0, four patches in this order
virglrenderer-chroma-format.patch
the guest's Mesa and virglrenderer number the chroma format of a codec differently: 4:2:0
was read as 4:2:2, and the host's encoder refused every codec it was asked for
virglrenderer-video-memory-upload.patch
the d3d12 driver has no dma-bufs: vaExportSurfaceHandle fails, nothing was uploaded and
the encoder returned an empty bitstream. The planes of the guest's picture are read back
(glReadPixels) and written to the surface through a derived image (vaDeriveImage, with
vaCreateImage and vaPutImage as a fallback)
virglrenderer-h264-packed-sps.patch
Mesa's VA-API takes the level of the stream from a packed SPS, which virglrenderer never
sent: level 0 reached UNREACHABLE in the d3d12 encoder and crashed QEMU
virglrenderer-ubo0-constants.patch
a constant buffer 0 bound as a buffer was never read by the shader, so the colour matrix
of an RGB to YUV conversion was zero and the picture was a solid green
Mesa 26.2.5 in the guest, two patches (hardware.graphics.package)
mesa-virgl-rgb-yuv-csc.patch
the graphics compositor never loaded an RGB to YUV matrix, and multiplied by the YUV to
RGB one of its initial state
mesa-virgl-fence-leak.patch
a surface kept the out-fence (a sync_file) of its last flush and the next flush stored a
new one over it: one file descriptor leaked per converted frame, 44 a second under motion,
and KRDP froze when it reached the limit of 1024, about 28 seconds in
KRDP (krdp.nix) asks for H264Baseline and the guest's driver lists Main and High, so ffmpeg's
h264_vaapi found no profile it could use: it is built with H264Main.
What was checked: a luma ramp and a colour-bar pattern encoded through the guest to the 4090
decode to the right values; the real desktop shows over RDP in Windows' own client with the
right colours; under es2gears motion the descriptors of krdpserver stay flat (3 sync_file, 78
of 1024) where they were 949 of 1024 after 28 seconds.
run-kde sets what the host needs for Mesa and for virglrenderer's libva to find the Windows
driver, on WSL 2 with the GPU on and unless it is set already: GALLIUM_DRIVER=d3d12,
LD_LIBRARY_PATH with /usr/lib/wsl/lib first, LIBVA_DRIVER_NAME=d3d12 and LIBVA_DRIVERS_PATH
(/run/opengl-driver/lib/dri, for the d3d12 video driver, which it looks for). --doctor checks
for the libraries and for that driver, and the tests have a scenario for each.
Known: the software decoder warns "illegal short term buffer state detected" on the stream; the
d3d12 encoder writes Constrained Baseline whatever profile is asked for. There is no binary
cache: the first run builds QEMU and Mesa, which takes about 20 minutes. QEMU, virglrenderer and
Mesa are the runner's and the guest's only, and the VM suites build the Mesa as well.
Not measured yet: whether it is smoother than the CPU encoder under the same load.
The guide for the one command: docs/trying-it.md says what a machine needs to
run the demo guest, with the lines of .wslconfig and of the NixOS-WSL
configuration for Windows 11, how to check it with `run-kde --doctor`, what the
command does and shows, the key bindings to try, how the GPU path is chosen, how
to open Windows' own RDP client, how long the first run takes (measured on a
machine with 32 cores and an empty store, by number of cores, with an estimate
for a WSL 2 machine and its caveats), how to start again, how to work on
komorebi in the guest, and what to do when something is wrong, by symptom. It
ends with what was run and what was not: nothing has been run on WSL 2 yet. The
README has a section and a link, and the user guide index a line.
DESIGN.md has four entries. Two record what I decided and want: the QEMU window
first and RDP second, and the GPU wherever the host has one. Two say that a
reason was not written down, for QEMU with KVM and not Hyper-V and for sharing
the password by a directory, and what is known that bears on each; the password
entry also says why a password is made for each user. PLATFORM.md has a section
for WSL 2 with what the command expects, a table of the probe results still to
be recorded (KVM, display, GPU, RDP, resources, virtiofs), and what is known of
virtiofsd without WSL 2.
Starting the demo guest by hand takes a build, a state directory, a password, a
key, the sizes of the machine and the right arguments for its display, and a
mistake shows up as an error from QEMU a minute later. `run-kde` does all of it:
`nix run .#run-kde` (or `just run-kde`) checks the machine, builds the guest
when its runner is not in the store, with the build log shown, makes the state
directory with a sparse disk, a random RDP password and an ssh key (all mode
0600, none of it in the Nix store), and starts the runner with KVM, the memory
and cores the machine has, the repository at /repo, and the state directory at
/run/run-kde. It prints a banner with the RDP address and password, the ssh
command, how to quit, and which graphics mode it chose and why.
`--doctor` prints every prerequisite as a tick, a cross or a warning with the
exact fix: WSL 2 or not, KVM and its group, nested virtualization, virtiofs (a
virtiofsd started as the user with the options of the guest's runner must serve
a directory, since the guest mounts its /nix/store through one; /dev/fuse is
only a warning), memory, disk, WSLg, Windows interop, the GPU and the OpenGL of
the host, and the lines of .wslconfig. It exits with 1 when something is a
cross, which includes a missing KVM unless `--tcg` allows software emulation.
The GPU is used when the display is a window, the host has a GPU device and its
OpenGL is not llvmpipe, and `--gpu on|off` overrides it; on WSL 2, `--rdp` opens
mstsc with a credential that lasts as long as the run.
The script reads the machine through variables, so `--dry-run` and `--doctor`
can be tested against machines that a check makes up: `run-kde-dry-run` and
`run-kde-doctor` compare their output with the texts in nix/tests/run-kde, and
the package, which shellchecks the script when it is built, is a check too.
LGUG2Z