diff --git a/.agents/skills/nix-config-update/SKILL.md b/.agents/skills/nix-config-update/SKILL.md index bfd9f07c..04331a99 100644 --- a/.agents/skills/nix-config-update/SKILL.md +++ b/.agents/skills/nix-config-update/SKILL.md @@ -133,9 +133,10 @@ ssh root@ hostname # the host that actually answers - Remote servers: `just shoryu [mode]`, `just shushou`, `just youko`, `just ruby`, `just kana` - All VM hosts at once: `just lab [mode]`; any Colmena tag: `just col [mode]` - k3s test nodes: `just k3s-test [mode]` -- MicroVM guest: `just microvm-deploy` currently fails at its activation step on the guest's - read-only `/nix/store`; WA-026 in [WORKAROUNDS.md](../../../WORKAROUNDS.md) carries the manual - path. Deploy guests serially and check each one before moving on. +- MicroVM guest: `just microvm-deploy ` installs the runner on the physical + host and restarts the guest unit there (WA-026); switch that host afterwards, or the next + activation reverts the guest to its own runner. Deploy guests serially and check each one before + moving on. Use `boot` plus a deliberate reboot for anything that can drop networking mid-flight: the VM hosts with the `br0` bridge, and broad nixpkgs bumps. See diff --git a/AGENTS.md b/AGENTS.md index e32532dd..9d8fed3a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,10 +84,10 @@ introduces one, and re-evaluate entries when their `Revisit` trigger comes up. - `just shoryu`, `just shushou`, `just youko`, `just ruby`, `just kana`, `just lab`, `just k3s-test`, and `just col ` activate systems through Colmena. Use the narrower recipe that matches the intended host scope. -- `just microvm-deploy ` installs and activates one MicroVM guest. - WA-026 currently replaces the activation step; check [WORKAROUNDS.md](./WORKAROUNDS.md) before - deploying. Deploy guests serially and check the guest Node and host services after each - activation. +- `just microvm-deploy ` installs one MicroVM guest's runner on the physical + host and restarts its unit there. A later host activation re-points the guest at the runner baked + into the host's system; see WA-026 in [WORKAROUNDS.md](./WORKAROUNDS.md) before deploying. Deploy + guests serially and check the guest Node and host services after each activation. - VM hosts (`shoryu`, `shushou`, `youko`) carry the `br0` bridge for their guests. Use the `boot`-based host deployment procedure for network stack or broad nixpkgs changes; see [hosts/README.md](./hosts/README.md#deploying-vm-hosts). diff --git a/Justfile b/Justfile index 590f898c..518d6b9f 100644 --- a/Justfile +++ b/Justfile @@ -39,7 +39,7 @@ eval-host host: build-host host: nix build $".#nixosConfigurations.{{host}}.config.system.build.toplevel" --no-link --print-build-logs -# Build a MicroVM runner locally. WA-026 in WORKAROUNDS.md covers how to deploy it. +# Build a MicroVM runner locally. microvm-deploy copies the closure to the physical host. [group('nix')] build-microvm guest: nix build $".#nixosConfigurations.{{guest}}.config.microvm.declaredRunner" --no-link --print-build-logs @@ -210,14 +210,15 @@ reset-launchpad: col tag mode="switch": colmena apply {{mode}} --on '@{{tag}}' --verbose --show-trace -# Deploy one microVM guest through microvm.nix's SSH deployment workflow (WA-026: the -# activation step currently fails; see WORKAROUNDS.md). -# The host is the physical machine running the guest; the guest address is used for activation. +# Deploy one microVM guest: copy its runner to the physical host, then restart the +# guest unit there. microvm.nix's sshSwitch step is not used: it switches the guest over +# SSH and dies on the read-only virtiofs /nix/store (WA-026 in WORKAROUNDS.md). +# The host is the physical machine running the guest. [linux] [group('homelab')] -microvm-deploy guest host guest_ip: +microvm-deploy guest host: nix run $".#nixosConfigurations.{{guest}}.config.microvm.deploy.installOnHost" -- root@{{host}} - nix run $".#nixosConfigurations.{{guest}}.config.microvm.deploy.sshSwitch" -- root@{{guest_ip}} --use-remote-sudo + ssh root@{{host}} systemctl restart microvm@{{guest}} # Deploy all the VM hosts (physical machines running the VMs) [linux] diff --git a/WORKAROUNDS.md b/WORKAROUNDS.md index 5a7b83d0..1096893f 100644 --- a/WORKAROUNDS.md +++ b/WORKAROUNDS.md @@ -33,18 +33,18 @@ A version or revision pinned for a reason that can go away. A setting that works around a bug or a hardware quirk. -| ID | What & where | Why | Removal condition | Added | Revisit | Status | -| ------ | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | --------------------------------------------------------------------------------------------------- | ------ | -| WA-008 | Relaxed NVMe APST (`nvme_core.default_ps_max_latency_us=100000`) — `hosts/idols-ai/hardware-configuration.nix` | the KINGBANK KP260 (DRAM-less QLC) froze on slow APST wake-ups | Drop the override once kernel/firmware fixes or replacement hardware make default APST reliable | 2026-10-03 | watch NVMe timeout/reset/AER in the journal | active | -| WA-009 | USB-SATA bridge mitigations (`usbcore.autosuspend=-1`, `usb-storage.delay_use=10`) — `hosts/12kingdoms-youko/default.nix` | flaky JMicron JMS567 resets with UDMA-CRC and I/O errors | Drop once the bridge/controller is replaced; the signal is smartctl CRC/media errors | 2026-09-21 | on hardware change or CRC errors | active | -| WA-010 | Insta360 Link USB reset on S3 resume — `hosts/idols-ai/default.nix` | the camera stays enumerated with a stalled UVC endpoint after resume | Drop after the Insta360 firmware or the kernel `uvcvideo` fixes it | 2026-08-01 | on kernel or firmware change | active | -| WA-011 | Kernel module blacklist (`esp4`, `esp6`, `rxrpc`) — `hardening/kernel-hardening.nix` | attack surface for the Dirty Frag LPE (CVE-2026-43284 / CVE-2026-43500) | Kept as defence in depth; drop only if something needs those modules | 2026-10-03 | rarely | active | -| WA-017 | keyd right-hand modifier passthrough — `modules/nixos/desktop/peripherals.nix` | keyd folds right Ctrl/Shift/Meta into their left keys | Drop once keyd or fcitx5 keeps right-hand modifiers distinct | 2026-10-05 | on keyd bump | active | -| WA-018 | user-owned `~/.ssh/config` + disabled systemd ssh-proxy include — `home/base/tui/ssh.nix`, `modules/nixos/base/ssh.nix` | bwrap exposes root-owned store files as `nobody` | Drop once sandboxes preserve store ownership | 2026-10-06 | when sandbox layout changes | active | -| WA-019 | programs.steam.extraCompatPackages = [ pkgs.dwproton-bin pkgs.proton-ge-bin ] — modules/nixos/desktop/gaming.nix | ACE games fail on GE/mainline Proton, work with DW-Proton | Drop once mainline or GE-Proton runs these games (GE-Proton#433) | 2026-10-07 | on Proton/GE bump | active | -| WA-024 | Wi-Fi power saving disabled (`networking.networkmanager.wifi.powersave = false`) — `hosts/12kingdoms-shoukei/apple-silicon.nix` | On this BCM4378/brcmfmac laptop, powersave is a common source of intermittent ASSOC-REJECT / authentication-timeout failures; NetworkManager then mistakes the failure for a bad key, re-prompts, and the client (Noctalia, nmtui, nmcli) saves the answer as a duplicate ` 1` profile | Drop once the kernel carries the brcmfmac `port_authorized` fix (WA-025) and no association failures remain | 2026-10-07 | watch `journalctl -u wpa_supplicant` for ASSOC-REJECT / authentication timeout | active | -| WA-026 | guest deploys run `installOnHost` then `systemctl restart microvm@`, not `just microvm-deploy` — `lib/genMicrovmGuestModule.nix` | the sshSwitch step runs `nix-store --load-db` against the guest's read-only virtiofs `/nix/store` and dies on `cannot remount "/nix/store" writable` (nix 2.35) | Drop the manual path once microvm.nix or nix handles a read-only store switch | 2026-10-09 | on microvm.nix or nix bump | active | -| WA-029 | mihomo `fake-ip-range6` is `2001:2::/64`, not the shipped ULA default — `modules/nixos/desktop/networking/mihomo/generate.nu` | the default `fdfe:dcba:9876::/64` sits in `fc00::/7`, which Local Network Access calls local, so a public page loaded over the v4 fake pool and pulling a dual-stack subresource prompts for local-network access (Chrome 154, Firefox). The same range also stalled ssh: a session to a fake AAAA authenticated and then never closed, on both `gvisor` and `mips`. `2001:2::/48` is RFC 5180 benchmarking -- the v6 twin of the `198.18.0.0/15` v4 default, absent from that table -- and both symptoms went with the move | Go back to the upstream default once mihomo ships a pool outside the LNA non-public table; the ULA default brings the ssh stall back | 2026-10-09 | on a browser or mihomo bump -- that table already grew to cover `2001:db8::/32` and `198.18.0.0/15` | active | +| ID | What & where | Why | Removal condition | Added | Revisit | Status | +| ------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------- | ------ | +| WA-008 | Relaxed NVMe APST (`nvme_core.default_ps_max_latency_us=100000`) — `hosts/idols-ai/hardware-configuration.nix` | the KINGBANK KP260 (DRAM-less QLC) froze on slow APST wake-ups | Drop the override once kernel/firmware fixes or replacement hardware make default APST reliable | 2026-10-03 | watch NVMe timeout/reset/AER in the journal | active | +| WA-009 | USB-SATA bridge mitigations (`usbcore.autosuspend=-1`, `usb-storage.delay_use=10`) — `hosts/12kingdoms-youko/default.nix` | flaky JMicron JMS567 resets with UDMA-CRC and I/O errors | Drop once the bridge/controller is replaced; the signal is smartctl CRC/media errors | 2026-09-21 | on hardware change or CRC errors | active | +| WA-010 | Insta360 Link USB reset on S3 resume — `hosts/idols-ai/default.nix` | the camera stays enumerated with a stalled UVC endpoint after resume | Drop after the Insta360 firmware or the kernel `uvcvideo` fixes it | 2026-08-01 | on kernel or firmware change | active | +| WA-011 | Kernel module blacklist (`esp4`, `esp6`, `rxrpc`) — `hardening/kernel-hardening.nix` | attack surface for the Dirty Frag LPE (CVE-2026-43284 / CVE-2026-43500) | Kept as defence in depth; drop only if something needs those modules | 2026-10-03 | rarely | active | +| WA-017 | keyd right-hand modifier passthrough — `modules/nixos/desktop/peripherals.nix` | keyd folds right Ctrl/Shift/Meta into their left keys | Drop once keyd or fcitx5 keeps right-hand modifiers distinct | 2026-10-05 | on keyd bump | active | +| WA-018 | user-owned `~/.ssh/config` + disabled systemd ssh-proxy include — `home/base/tui/ssh.nix`, `modules/nixos/base/ssh.nix` | bwrap exposes root-owned store files as `nobody` | Drop once sandboxes preserve store ownership | 2026-10-06 | when sandbox layout changes | active | +| WA-019 | programs.steam.extraCompatPackages = [ pkgs.dwproton-bin pkgs.proton-ge-bin ] — modules/nixos/desktop/gaming.nix | ACE games fail on GE/mainline Proton, work with DW-Proton | Drop once mainline or GE-Proton runs these games (GE-Proton#433) | 2026-10-07 | on Proton/GE bump | active | +| WA-024 | Wi-Fi power saving disabled (`networking.networkmanager.wifi.powersave = false`) — `hosts/12kingdoms-shoukei/apple-silicon.nix` | On this BCM4378/brcmfmac laptop, powersave is a common source of intermittent ASSOC-REJECT / authentication-timeout failures; NetworkManager then mistakes the failure for a bad key, re-prompts, and the client (Noctalia, nmtui, nmcli) saves the answer as a duplicate ` 1` profile | Drop once the kernel carries the brcmfmac `port_authorized` fix (WA-025) and no association failures remain | 2026-10-07 | watch `journalctl -u wpa_supplicant` for ASSOC-REJECT / authentication timeout | active | +| WA-026 | guest deploys run `installOnHost` then `systemctl restart microvm@` on the physical host — `Justfile`, `lib/genMicrovmGuestModule.nix` | the sshSwitch step runs `nix-store --load-db` against the guest's read-only virtiofs `/nix/store` and dies on `cannot remount "/nix/store" writable` (nix 2.35) | Drop the manual path once microvm.nix or nix handles a read-only store switch. A host activation re-links `/var/lib/microvms//current` to the runner baked into the host's system, so switch the host after installing a guest from a newer tree | 2026-10-09 | on microvm.nix or nix bump | active | +| WA-029 | mihomo `fake-ip-range6` is `2001:2::/64`, not the shipped ULA default — `modules/nixos/desktop/networking/mihomo/generate.nu` | the default `fdfe:dcba:9876::/64` sits in `fc00::/7`, which Local Network Access calls local, so a public page loaded over the v4 fake pool and pulling a dual-stack subresource prompts for local-network access (Chrome 154, Firefox). The same range also stalled ssh: a session to a fake AAAA authenticated and then never closed, on both `gvisor` and `mips`. `2001:2::/48` is RFC 5180 benchmarking -- the v6 twin of the `198.18.0.0/15` v4 default, absent from that table -- and both symptoms went with the move | Go back to the upstream default once mihomo ships a pool outside the LNA non-public table; the ULA default brings the ssh stall back | 2026-10-09 | on a browser or mihomo bump -- that table already grew to cover `2001:db8::/32` and `198.18.0.0/15` | active | If WA-008 freezes recur, the fallback is to restore `nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 pcie_aspm=off`; this is a diff --git a/hosts/k8s/README.md b/hosts/k8s/README.md index 187ec0da..90b0f950 100644 --- a/hosts/k8s/README.md +++ b/hosts/k8s/README.md @@ -31,13 +31,15 @@ complete Nix closure to the physical host, installs it under `/var/lib/microvms/ activates the guest: ```sh -just microvm-deploy +just microvm-deploy ``` -For example, `just microvm-deploy k3s-test-1-master-3 shushou 192.168.5.116`. The recipe installs -the runner on the physical host first, then activates the guest with the microvm.nix SSH deployment -interface. WA-026 in [WORKAROUNDS.md](../../WORKAROUNDS.md) currently replaces that activation, -because the SSH switch cannot write the guest's read-only `/nix/store`. +For example, `just microvm-deploy k3s-test-1-master-3 shushou`. The recipe copies the runner to the +physical host, then restarts `microvm@` there; microvm.nix's SSH switch is not used because +it cannot write the guest's read-only `/nix/store` (WA-026 in +[WORKAROUNDS.md](../../WORKAROUNDS.md)). A later activation of the physical host re-points the guest +at the runner baked into the host's system, so switch the host after installing a guest from a newer +tree. Use `just build-microvm ` to build a runner without deploying it. Preserve the guest's `etc.img`, `var.img`, and `home.img` when updating it; they contain its identity and K3s state.