mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:36 +01:00
fix(microvm): restart the guest unit on the physical host to deploy
microvm.nix's sshSwitch step runs nix-store --load-db against the guest's read-only virtiofs /nix/store and dies there, so the recipe installs the runner on the physical host and restarts microvm@<guest> instead.
This commit is contained in:
Ryan Yin committed
1 parent
a2cfa5549c
commit
60b80e64c8
5 files changed
+34
-30
No files matched your search
@@ -133,9 +133,10 @@ ssh root@<host> hostname # the host that actually answers
|
||||
- Remote servers: `just shoryu [mode]`, `just shushou`, `just youko`, `just ruby`, `just kana`
|
||||
- All VM hosts at once: `just lab [mode]`; any Colmena tag: `just col <tag> [mode]`
|
||||
- k3s test nodes: `just k3s-test [mode]`
|
||||
- MicroVM guest: `just microvm-deploy` currently fails at its activation step on the guest's
|
||||
read-only `/nix/store`; WA-026 in [WORKAROUNDS.md](../../../WORKAROUNDS.md) carries the manual
|
||||
path. Deploy guests serially and check each one before moving on.
|
||||
- MicroVM guest: `just microvm-deploy <guest> <physical-host>` installs the runner on the physical
|
||||
host and restarts the guest unit there (WA-026); switch that host afterwards, or the next
|
||||
activation reverts the guest to its own runner. Deploy guests serially and check each one before
|
||||
moving on.
|
||||
|
||||
Use `boot` plus a deliberate reboot for anything that can drop networking mid-flight: the VM hosts
|
||||
with the `br0` bridge, and broad nixpkgs bumps. See
|
||||
|
||||
@@ -84,10 +84,10 @@ introduces one, and re-evaluate entries when their `Revisit` trigger comes up.
|
||||
- `just shoryu`, `just shushou`, `just youko`, `just ruby`, `just kana`, `just lab`,
|
||||
`just k3s-test`, and `just col <tag>` activate systems through Colmena. Use the narrower recipe
|
||||
that matches the intended host scope.
|
||||
- `just microvm-deploy <guest> <physical-host> <guest-ip>` installs and activates one MicroVM guest.
|
||||
WA-026 currently replaces the activation step; check [WORKAROUNDS.md](./WORKAROUNDS.md) before
|
||||
deploying. Deploy guests serially and check the guest Node and host services after each
|
||||
activation.
|
||||
- `just microvm-deploy <guest> <physical-host>` installs one MicroVM guest's runner on the physical
|
||||
host and restarts its unit there. A later host activation re-points the guest at the runner baked
|
||||
into the host's system; see WA-026 in [WORKAROUNDS.md](./WORKAROUNDS.md) before deploying. Deploy
|
||||
guests serially and check the guest Node and host services after each activation.
|
||||
- VM hosts (`shoryu`, `shushou`, `youko`) carry the `br0` bridge for their guests. Use the
|
||||
`boot`-based host deployment procedure for network stack or broad nixpkgs changes; see
|
||||
[hosts/README.md](./hosts/README.md#deploying-vm-hosts).
|
||||
|
||||
@@ -39,7 +39,7 @@ eval-host host:
|
||||
build-host host:
|
||||
nix build $".#nixosConfigurations.{{host}}.config.system.build.toplevel" --no-link --print-build-logs
|
||||
|
||||
# Build a MicroVM runner locally. WA-026 in WORKAROUNDS.md covers how to deploy it.
|
||||
# Build a MicroVM runner locally. microvm-deploy copies the closure to the physical host.
|
||||
[group('nix')]
|
||||
build-microvm guest:
|
||||
nix build $".#nixosConfigurations.{{guest}}.config.microvm.declaredRunner" --no-link --print-build-logs
|
||||
@@ -210,14 +210,15 @@ reset-launchpad:
|
||||
col tag mode="switch":
|
||||
colmena apply {{mode}} --on '@{{tag}}' --verbose --show-trace
|
||||
|
||||
# Deploy one microVM guest through microvm.nix's SSH deployment workflow (WA-026: the
|
||||
# activation step currently fails; see WORKAROUNDS.md).
|
||||
# The host is the physical machine running the guest; the guest address is used for activation.
|
||||
# Deploy one microVM guest: copy its runner to the physical host, then restart the
|
||||
# guest unit there. microvm.nix's sshSwitch step is not used: it switches the guest over
|
||||
# SSH and dies on the read-only virtiofs /nix/store (WA-026 in WORKAROUNDS.md).
|
||||
# The host is the physical machine running the guest.
|
||||
[linux]
|
||||
[group('homelab')]
|
||||
microvm-deploy guest host guest_ip:
|
||||
microvm-deploy guest host:
|
||||
nix run $".#nixosConfigurations.{{guest}}.config.microvm.deploy.installOnHost" -- root@{{host}}
|
||||
nix run $".#nixosConfigurations.{{guest}}.config.microvm.deploy.sshSwitch" -- root@{{guest_ip}} --use-remote-sudo
|
||||
ssh root@{{host}} systemctl restart microvm@{{guest}}
|
||||
|
||||
# Deploy all the VM hosts (physical machines running the VMs)
|
||||
[linux]
|
||||
|
||||
+12
-12
@@ -33,18 +33,18 @@ A version or revision pinned for a reason that can go away.
|
||||
|
||||
A setting that works around a bug or a hardware quirk.
|
||||
|
||||
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
|
||||
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | --------------------------------------------------------------------------------------------------- | ------ |
|
||||
| WA-008 | Relaxed NVMe APST (`nvme_core.default_ps_max_latency_us=100000`) — `hosts/idols-ai/hardware-configuration.nix` | the KINGBANK KP260 (DRAM-less QLC) froze on slow APST wake-ups | Drop the override once kernel/firmware fixes or replacement hardware make default APST reliable | 2026-10-03 | watch NVMe timeout/reset/AER in the journal | active |
|
||||
| WA-009 | USB-SATA bridge mitigations (`usbcore.autosuspend=-1`, `usb-storage.delay_use=10`) — `hosts/12kingdoms-youko/default.nix` | flaky JMicron JMS567 resets with UDMA-CRC and I/O errors | Drop once the bridge/controller is replaced; the signal is smartctl CRC/media errors | 2026-09-21 | on hardware change or CRC errors | active |
|
||||
| WA-010 | Insta360 Link USB reset on S3 resume — `hosts/idols-ai/default.nix` | the camera stays enumerated with a stalled UVC endpoint after resume | Drop after the Insta360 firmware or the kernel `uvcvideo` fixes it | 2026-08-01 | on kernel or firmware change | active |
|
||||
| WA-011 | Kernel module blacklist (`esp4`, `esp6`, `rxrpc`) — `hardening/kernel-hardening.nix` | attack surface for the Dirty Frag LPE (CVE-2026-43284 / CVE-2026-43500) | Kept as defence in depth; drop only if something needs those modules | 2026-10-03 | rarely | active |
|
||||
| WA-017 | keyd right-hand modifier passthrough — `modules/nixos/desktop/peripherals.nix` | keyd folds right Ctrl/Shift/Meta into their left keys | Drop once keyd or fcitx5 keeps right-hand modifiers distinct | 2026-10-05 | on keyd bump | active |
|
||||
| WA-018 | user-owned `~/.ssh/config` + disabled systemd ssh-proxy include — `home/base/tui/ssh.nix`, `modules/nixos/base/ssh.nix` | bwrap exposes root-owned store files as `nobody` | Drop once sandboxes preserve store ownership | 2026-10-06 | when sandbox layout changes | active |
|
||||
| WA-019 | programs.steam.extraCompatPackages = [ pkgs.dwproton-bin pkgs.proton-ge-bin ] — modules/nixos/desktop/gaming.nix | ACE games fail on GE/mainline Proton, work with DW-Proton | Drop once mainline or GE-Proton runs these games (GE-Proton#433) | 2026-10-07 | on Proton/GE bump | active |
|
||||
| WA-024 | Wi-Fi power saving disabled (`networking.networkmanager.wifi.powersave = false`) — `hosts/12kingdoms-shoukei/apple-silicon.nix` | On this BCM4378/brcmfmac laptop, powersave is a common source of intermittent ASSOC-REJECT / authentication-timeout failures; NetworkManager then mistakes the failure for a bad key, re-prompts, and the client (Noctalia, nmtui, nmcli) saves the answer as a duplicate `<SSID> 1` profile | Drop once the kernel carries the brcmfmac `port_authorized` fix (WA-025) and no association failures remain | 2026-10-07 | watch `journalctl -u wpa_supplicant` for ASSOC-REJECT / authentication timeout | active |
|
||||
| WA-026 | guest deploys run `installOnHost` then `systemctl restart microvm@<guest>`, not `just microvm-deploy` — `lib/genMicrovmGuestModule.nix` | the sshSwitch step runs `nix-store --load-db` against the guest's read-only virtiofs `/nix/store` and dies on `cannot remount "/nix/store" writable` (nix 2.35) | Drop the manual path once microvm.nix or nix handles a read-only store switch | 2026-10-09 | on microvm.nix or nix bump | active |
|
||||
| WA-029 | mihomo `fake-ip-range6` is `2001:2::/64`, not the shipped ULA default — `modules/nixos/desktop/networking/mihomo/generate.nu` | the default `fdfe:dcba:9876::/64` sits in `fc00::/7`, which Local Network Access calls local, so a public page loaded over the v4 fake pool and pulling a dual-stack subresource prompts for local-network access (Chrome 154, Firefox). The same range also stalled ssh: a session to a fake AAAA authenticated and then never closed, on both `gvisor` and `mips`. `2001:2::/48` is RFC 5180 benchmarking -- the v6 twin of the `198.18.0.0/15` v4 default, absent from that table -- and both symptoms went with the move | Go back to the upstream default once mihomo ships a pool outside the LNA non-public table; the ULA default brings the ssh stall back | 2026-10-09 | on a browser or mihomo bump -- that table already grew to cover `2001:db8::/32` and `198.18.0.0/15` | active |
|
||||
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
|
||||
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------- | ------ |
|
||||
| WA-008 | Relaxed NVMe APST (`nvme_core.default_ps_max_latency_us=100000`) — `hosts/idols-ai/hardware-configuration.nix` | the KINGBANK KP260 (DRAM-less QLC) froze on slow APST wake-ups | Drop the override once kernel/firmware fixes or replacement hardware make default APST reliable | 2026-10-03 | watch NVMe timeout/reset/AER in the journal | active |
|
||||
| WA-009 | USB-SATA bridge mitigations (`usbcore.autosuspend=-1`, `usb-storage.delay_use=10`) — `hosts/12kingdoms-youko/default.nix` | flaky JMicron JMS567 resets with UDMA-CRC and I/O errors | Drop once the bridge/controller is replaced; the signal is smartctl CRC/media errors | 2026-09-21 | on hardware change or CRC errors | active |
|
||||
| WA-010 | Insta360 Link USB reset on S3 resume — `hosts/idols-ai/default.nix` | the camera stays enumerated with a stalled UVC endpoint after resume | Drop after the Insta360 firmware or the kernel `uvcvideo` fixes it | 2026-08-01 | on kernel or firmware change | active |
|
||||
| WA-011 | Kernel module blacklist (`esp4`, `esp6`, `rxrpc`) — `hardening/kernel-hardening.nix` | attack surface for the Dirty Frag LPE (CVE-2026-43284 / CVE-2026-43500) | Kept as defence in depth; drop only if something needs those modules | 2026-10-03 | rarely | active |
|
||||
| WA-017 | keyd right-hand modifier passthrough — `modules/nixos/desktop/peripherals.nix` | keyd folds right Ctrl/Shift/Meta into their left keys | Drop once keyd or fcitx5 keeps right-hand modifiers distinct | 2026-10-05 | on keyd bump | active |
|
||||
| WA-018 | user-owned `~/.ssh/config` + disabled systemd ssh-proxy include — `home/base/tui/ssh.nix`, `modules/nixos/base/ssh.nix` | bwrap exposes root-owned store files as `nobody` | Drop once sandboxes preserve store ownership | 2026-10-06 | when sandbox layout changes | active |
|
||||
| WA-019 | programs.steam.extraCompatPackages = [ pkgs.dwproton-bin pkgs.proton-ge-bin ] — modules/nixos/desktop/gaming.nix | ACE games fail on GE/mainline Proton, work with DW-Proton | Drop once mainline or GE-Proton runs these games (GE-Proton#433) | 2026-10-07 | on Proton/GE bump | active |
|
||||
| WA-024 | Wi-Fi power saving disabled (`networking.networkmanager.wifi.powersave = false`) — `hosts/12kingdoms-shoukei/apple-silicon.nix` | On this BCM4378/brcmfmac laptop, powersave is a common source of intermittent ASSOC-REJECT / authentication-timeout failures; NetworkManager then mistakes the failure for a bad key, re-prompts, and the client (Noctalia, nmtui, nmcli) saves the answer as a duplicate `<SSID> 1` profile | Drop once the kernel carries the brcmfmac `port_authorized` fix (WA-025) and no association failures remain | 2026-10-07 | watch `journalctl -u wpa_supplicant` for ASSOC-REJECT / authentication timeout | active |
|
||||
| WA-026 | guest deploys run `installOnHost` then `systemctl restart microvm@<guest>` on the physical host — `Justfile`, `lib/genMicrovmGuestModule.nix` | the sshSwitch step runs `nix-store --load-db` against the guest's read-only virtiofs `/nix/store` and dies on `cannot remount "/nix/store" writable` (nix 2.35) | Drop the manual path once microvm.nix or nix handles a read-only store switch. A host activation re-links `/var/lib/microvms/<guest>/current` to the runner baked into the host's system, so switch the host after installing a guest from a newer tree | 2026-10-09 | on microvm.nix or nix bump | active |
|
||||
| WA-029 | mihomo `fake-ip-range6` is `2001:2::/64`, not the shipped ULA default — `modules/nixos/desktop/networking/mihomo/generate.nu` | the default `fdfe:dcba:9876::/64` sits in `fc00::/7`, which Local Network Access calls local, so a public page loaded over the v4 fake pool and pulling a dual-stack subresource prompts for local-network access (Chrome 154, Firefox). The same range also stalled ssh: a session to a fake AAAA authenticated and then never closed, on both `gvisor` and `mips`. `2001:2::/48` is RFC 5180 benchmarking -- the v6 twin of the `198.18.0.0/15` v4 default, absent from that table -- and both symptoms went with the move | Go back to the upstream default once mihomo ships a pool outside the LNA non-public table; the ULA default brings the ssh stall back | 2026-10-09 | on a browser or mihomo bump -- that table already grew to cover `2001:db8::/32` and `198.18.0.0/15` | active |
|
||||
|
||||
If WA-008 freezes recur, the fallback is to restore
|
||||
`nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 pcie_aspm=off`; this is a
|
||||
|
||||
+7
-5
@@ -31,13 +31,15 @@ complete Nix closure to the physical host, installs it under `/var/lib/microvms/
|
||||
activates the guest:
|
||||
|
||||
```sh
|
||||
just microvm-deploy <guest> <physical-host> <guest-ip>
|
||||
just microvm-deploy <guest> <physical-host>
|
||||
```
|
||||
|
||||
For example, `just microvm-deploy k3s-test-1-master-3 shushou 192.168.5.116`. The recipe installs
|
||||
the runner on the physical host first, then activates the guest with the microvm.nix SSH deployment
|
||||
interface. WA-026 in [WORKAROUNDS.md](../../WORKAROUNDS.md) currently replaces that activation,
|
||||
because the SSH switch cannot write the guest's read-only `/nix/store`.
|
||||
For example, `just microvm-deploy k3s-test-1-master-3 shushou`. The recipe copies the runner to the
|
||||
physical host, then restarts `microvm@<guest>` there; microvm.nix's SSH switch is not used because
|
||||
it cannot write the guest's read-only `/nix/store` (WA-026 in
|
||||
[WORKAROUNDS.md](../../WORKAROUNDS.md)). A later activation of the physical host re-points the guest
|
||||
at the runner baked into the host's system, so switch the host after installing a guest from a newer
|
||||
tree.
|
||||
|
||||
Use `just build-microvm <guest>` to build a runner without deploying it. Preserve the guest's
|
||||
`etc.img`, `var.img`, and `home.img` when updating it; they contain its identity and K3s state.
|
||||
|
||||
Reference in new issue
Block a user