nix-config-ryan4yin/hosts/idols-aquamarine

Host - Aquamarine

Headless AI inference server. Revives the aquamarine name, retired on 2026-09-21 when the VM of that name was decommissioned; this is a different, physical machine that reuses the name and the 192.168.5.101 address.

Hardware

Manual BIOS and power-saving settings are recorded in docs/bios.md.

Part	Model	Notes
Board	Colorful CVN Z890 ARK FROZEN V20 (BIOS 1013)	moved from ai on 2026-10-13, so its EFI vars come too
CPU	Intel Core Ultra 7 270K Plus	Arrow Lake; the NPU serves whisper-asr (./asr.nix)
Memory	2x48G JUHOR (96G), 4800 MT/s	the 2x16G kit stayed with ai
GPU	2x AMD Radeon AI PRO R9700, 32G each (RDNA4, gfx1201)	64G of VRAM total
Root	Fanxiang S790 2TB	was ai's /data disk until 2026-10-13
NIC	Realtek RTL8125BG 2.5GbE, enp133s0 today	matched by the en* prefix

Two things about this board set the ceiling for the GPU work:

• The second full-length x16 slot is chipset-attached: its root port negotiates Gen4 x4, while the card side reports Gen5 x16 either way (./ROCM.md has the link measurements). The chipset card probes 7.1 GB/s host->device against 46.3 GB/s for the CPU-attached card, so a two-card layer split is 8-27% slower per request and ~9% ahead on six concurrent requests. Read the inference engine's PCIe probe line rather than lspci, and prefer a CPU-attached slot for a second card.
• Multi-GPU work should split by layer (RPC / pipeline parallel), not by tensor parallelism: the R9700 has no xGMI link between cards, and a dual-R9700 report has RCCL deadlocking with vLLM TP=2 even with NCCL_P2P_DISABLE=1. See ROCM.md.

Idle power accounting

node_exporter exports the RAPL package/core/uncore counters and the R9700 hwmon power sensors, so VictoriaMetrics/Grafana can show whether the host is actually idle. For a one-off breakdown:

sudo turbostat --quiet --show PkgWatt,RAMWatt,GFXWatt,CorWatt --interval 2 --iterations 5
sudo powertop
rocm-smi --showpower --showuse --showclocks

RAPL and hwmon only cover silicon domains. They do not include PSU, VRM, fan, NVMe or board losses; measuring total wall power needs a metering smart plug/PDU or a UPS with per-outlet power.

Monitoring overhead

The scrape is not a meaningful load: node_exporter samples every 30 s, and the measured process cost is a few CPU ticks per scrape (well below 1 W even after accounting for wakeups). Grafana queries run on youko, not on this host, so opening a dashboard does not wake aquamarine's CPU. Keep the 30 s interval; lowering it would lose useful resolution for no measurable power saving.

CPU idle-power settings

• hardware-intel-cpu.nix uses NixOS's native boot.kernel.sysfs option to set Intel P-state's energy_performance_preference to balance_power on every CPU. This is separate from the existing x86-wide powerManagement.cpuFreqGovernor = "powersave": the governor chooses the frequency policy, while EPP tells intel_pstate how aggressively to trade power for performance. A runtime A/B on 2026-11-01 lowered the whole-home meter minimum from 160–170 W to 137 W, though the reading occasionally returned to the old range. This is a whole-rack number, not a host-only measurement.
• processor.max_cstate=2 is intentionally absent so the kernel can use deeper package C-states. If inference tail latency becomes more important than idle power, restore that kernel parameter; it was set for decode jitter, not for hardware stability. After the first deployment on 2026-11-01, intel_idle still exposed only POLL/C1/C2/C3 on BIOS 1013, so deeper CPU/package C-states still need to be enabled in BIOS before this knob can save more power.
• Safe BIOS-side changes: enable C1E, Intel Speed Shift and CPU/package C-states (limit Auto or C10), and enable iGPU Render Standby/RC6. Disable unused Wi-Fi/Bluetooth, onboard audio, SATA, serial ports and RGB lighting. Do not change the storage mode.

The iGPU carries display and video work: the kernel binds i915, and hardware-intel-igpu.nix installs intel-media-driver (VA-API), vpl-gpu-rt (oneVPL/QSV) and intel-compute-runtime (OpenCL/Level-Zero), so transcoding can use its media engine while the R9700s stay on inference. VA-API targets /dev/dri/renderD128; IGPU.md records the static-shape workaround Whisper needs on GPU, and the measured NPU/iGPU/CPU timings. A monitor plugged into the board's HDMI/DP ports still gets the text console (multi-user.target, no display manager) when SSH is unreachable.

iommu=pt is set in hardware-amd-gpu.nix because ROCm documents hangs on multi-GPU systems without IOMMU passthrough. The same file disables amdgpu runtime power management: this board's DSDT advertises ATPX power control that its own ACPI methods cannot deliver, so the first idle resume of the headless R9700 kills it until reboot. ROCM.md records whose defect that is, the supported ROCm versions, and the inference notes.

Disk layout

Declarative via disko-fs.nix: ESP + LUKS (aqua-luks) + btrfs subvolumes (@nix, @persistent, @snapshots, @tmp, @swap with a 32G swapfile), ephemeral tmpfs root, state under /persistent via preservation.nix. Secure Boot via lanzaboote (secureboot.nix).

The mapper is aqua-luks, not nixos-luks: the disk is formatted while the staging host is running, and that host already owns the nixos-luks name.

Install (staged from a running host)

This disk is installed into a machine that is not the one it gets formatted on: disko runs while the disk is still attached to a running NixOS host (ai below), and the disk then moves to its own machine. The target machine is the board the install runs on, so the EFI boot entry and the Secure Boot key enrollment the host already has carry over. That is what makes a staged install possible; it also means these steps run on a machine whose own root disk is one disko destroy away.

Steps 1 to 6 only touch this disk and leave ai running. ai's own deploy and shutdown are step 7, and they belong together: the machine comes back on the new board with this disk gone.

1. Confirm the target disk and release it: lsblk -f must show no mount and no open mapper on it. A disk reused from another role still carries that role's mounts:

   ls -l /dev/disk/by-id/nvme-Fanxiang_S790_2TB_FXS790254050582
   lsblk -f
   sudo umount /data                 # the previous role's mountpoint
   sudo cryptsetup close data-luks   # and its mapper
   

2. Partition, encrypt and mount. This destroys everything on that disk — re-read the device line in disko-fs.nix first. Run it as root: unprivileged, disko still walks through the confirmation prompt and then fails on every step with Permission denied.

   cd ~/nix-config
   sudo nix run github:nix-community/disko -- --mode destroy,format,mount hosts/idols-aquamarine/disko-fs.nix
   

   Success looks like two new partitions — disk-nixos-aquamarine-ESP (vfat, 600M) and disk-nixos-aquamarine-root (crypto_LUKS) — and seven mounts under /mnt. Check before moving on:

   lsblk -o NAME,SIZE,FSTYPE,PARTLABEL /dev/nvme0n1
   findmnt -R /mnt
   

3. Reuse the staging host's Secure Boot PKI bundle, so lanzaboote signs this system with a key the firmware already trusts. The bundle contains a private signing key: copy it yourself, and never commit it or send it through chat.

   sudo mkdir -p /mnt/persistent/etc/secureboot
   sudo cp -a /persistent/etc/secureboot/. /mnt/persistent/etc/secureboot/
   
   # lanzaboote reads the key from /etc/secureboot inside the target, and preservation
   # only bind-mounts that directory at boot — so expose it during the install too.
   sudo mkdir -p /mnt/etc/secureboot
   sudo mount --bind /mnt/persistent/etc/secureboot /mnt/etc/secureboot
   

   nixos-install installs the boot loader through nixos-enter --root /mnt and NIXOS_INSTALL_BOOTLOADER=1 ... switch-to-configuration boot, so lanzaboote's install hook reads /etc/secureboot/keys/db/db.key from the target root. Without the bind mount that path is an empty directory and the install fails at "installing the boot loader", however good the copy in /mnt/persistent is.

   Skipping this is fine only if you would rather start over: delete the firmware keys, sbctl create-keys + sbctl enroll-keys in setup mode. The fresh keys must end up in /etc/secureboot — lanzaboote's pkiBundle, while sbctl create-keys writes to its own keydir — or the firmware trusts a key this system's UKI was not signed with. That changes PCR7, so any TPM2 LUKS enrollment made before it stops unlocking.

4. Enroll the new LUKS volume with this board's TPM2, keeping the passphrase slot as the fallback:

   sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+7 \
     /dev/disk/by-id/nvme-Fanxiang_S790_2TB_FXS790254050582-part2
   sudo systemd-cryptenroll /dev/disk/by-id/nvme-Fanxiang_S790_2TB_FXS790254050582-part2  # slots: expect tpm2 + passphrase
   

   PCR7 measures the Secure Boot key database, so enroll after step 3, not before. Re-run this whenever the Secure Boot keys change.

5. Install the real system. The closure is mostly cached on the staging host, and step 3's bind mount is what lets lanzaboote sign into this disk's ESP:

   cd ~/nix-config
   sudo nixos-install --root /mnt --flake .#aquamarine --no-root-password
   

   If a private flake input is not in the host's store yet, fetch it as root: sudo does not pass SSH_AUTH_SOCK through on this host, so run the install from a root shell with its own agent (sudo su, then eval "$(ssh-agent)" && ssh-add ~/.ssh/<github-key>).

   The installer writes an EFI boot entry into this board's NVRAM and may put it first. Do not reboot the staging host with both disks attached — it would boot aquamarine's system.

   The install generates the target's /etc, but preservation only bind-mounts /persistent at boot, so the state it looks for has to be in place before the disk is unmounted. Move the generated /etc/ssh and machine-id in, then read the listing:

   sudo nixos-enter --root /mnt
   mkdir -p /persistent/etc
   mv /etc/ssh /persistent/etc/
   mv /etc/machine-id /persistent/etc/
   ls -l /persistent/etc/ssh /persistent/etc/ssh/authorized_keys.d
   exit
   

   authorized_keys.d/<user> must be a regular file with mode 0444: sshd's StrictModes rejects a key file whose path crosses /nix/store, which is group-writable (1775), so a symlink to /etc/static/... there locks every login out. sshd_config is a symlink to /etc/static/ssh/sshd_config on purpose — that one tracks the current generation. If a file is missing, copy the real one in and create its directory if needed: cp -L /etc/static/ssh/authorized_keys.d/. /persistent/etc/ssh/authorized_keys.d/.

6. Unmount the target: umount -R /mnt also drops step 3's bind mount. This disk is done; leave the staging host running for step 7.

   sudo umount -R /mnt
   findmnt -R /mnt          # nothing left mounted
   

7. On ai, stage its next generation and shut it down — one step, never one without the other. The machine boots on the new board with this disk gone, so the generation it comes back on must not wait for a LUKS volume here; and staging only takes effect at that boot, so shutting down first would leave the current generation as the default.

   # the host's initrd must no longer list this disk's mapper
   nix eval --json .#nixosConfigurations.ai-niri.config.boot.initrd.luks.devices --apply builtins.attrNames
   # [ "nixos-luks" ]
   
   just niri boot      # stage only: `switch` would apply the NIC rename now and cut the LAN
   sudo poweroff       # only after the deploy succeeded
   

   Then move the disk to its machine, in the same M.2 slot: the EFI entry names the disk by its device path, and a different slot can invalidate it.

First boot on aquamarine

1. The initrd should unlock aqua-luks from the TPM; if it prompts, use the passphrase.

2. If the firmware has no boot entry for the disk, pick it once from the boot menu (the ESP also carries the \EFI\BOOT\BOOTX64.EFI fallback), then let the first activation write the entry.

3. From another host:

   ssh aquamarine ip addr
   ssh aquamarine systemctl --failed
   ssh aquamarine journalctl -b -p err
   ssh aquamarine rocminfo
   ssh aquamarine rocm-smi --showproductname
   

   If ssh refuses the connection, or sshd logs bad ownership or mode for directory /nix/store, the /etc/ssh copy under /persistent is incomplete — repeat step 5's check from the console before deploying anything else.

   ip addr must show 192.168.5.101 on the wired interface. The networkd match uses the en* prefix, so a renamed NIC cannot strand the host; if the address is missing, check the cable and ip -br link first.

After the first boot

The install created a fresh SSH host key; the old aquamarine VM key is dead and the new one is already in vars/networking.nix under knownHosts (2026-10-13). Redo these steps only after a reinstall or a key rotation:

1. ssh aquamarine cat /etc/ssh/ssh_host_ed25519_key.pub.
2. Put that key in vars/networking.nix under knownHosts (the entry was removed with the old key) and, if this host should read agenix secrets, in ~/codes/nix-secrets/agenix-rules.nix under application_server_keys.
3. Rekey and bump the pin: just -f secrets/Justfile rekey, push nix-secrets, then just upp mysecrets here.

This config enables no modules.secrets.* group, so aquamarine decrypts nothing today; adding a group is what requires the recipient step above.

Remote builds

Aquamarine is the fleet's remote Nix builder: 24C/96G, serving native x86_64-linux only. It runs no boot.binfmt.emulatedSystems: emulated builds are ~10x slower per core, shoukei builds aarch64 natively, and this flake has no riscv64 output. Clients opt in with modules.remote-builders.enable, defined in modules/nixos/base/remote-builders.nix.

The builder's login user is root, which the daemon trusts by default, so this host needs no nix.settings.trusted-users entry of its own. Each client's daemon authenticates with its own SSH host key, which is passphrase-free and root-readable; hosts/idols-aquamarine/default.nix authorizes those keys for root.

Deploy

just aquamarine          # colmena, from any other host
just local               # on aquamarine itself

Rollback

The install only touches the target disk; the staging host keeps booting from its own. To undo, move the disk back, restore the hosts/idols-ai/disko-fs-data.nix module this change deleted, and re-enroll that machine's TPM2 slot — see ../idols-ai/README.md.

Idle suspend

autosuspend checks every minute and suspends into S3 after 60 minutes without an active user session, meaningful CPU load, or an established connection to one of the request-serving ports: 18080 (ASR), 63081 (llama-swap/Strata), 63083/63084 (MagPie), and 65002 (direct Strata). It calls systemctl suspend, so systemd sleep inhibitors are also honored. Wake-on-LAN is armed by the host's systemd.network.links configuration; see docs/wake-on-lan.md. No separate WoL service is needed.

The daemon resets its idle timer after resume, so a wake does not immediately go back to sleep. To keep the host awake during maintenance:

systemd-inhibit --what=sleep --mode=block --who=maintenance --why="manual work" sleep infinity

Cancel it with Ctrl-C when finished.