feat(aquamarine): cap each R9700 power limit at 250 W #453

hosts/idols-aquamarine/ROCM.md

@@ -3,8 +3,9 @@
 `aquamarine` runs Radeon AI PRO R9700 cards (Navi 48, RDNA4, PCI ID `1002:7551`, ROCm target
 `gfx1201`) as headless compute devices; the Intel iGPU drives the console. This note records which
 versions support the card, why amdgpu runtime power management leaves it unusable, and which
-decisions that produced. Claims are marked **verified** (primary source), **community-reported**, or
-**inference** (read from kernel source, not confirmed by a maintainer).
+decisions that produced. Claims are marked **verified** (primary source), **community-reported**,
+**measured here** (taken on this machine), or **inference** (read from kernel source, not confirmed
+by a maintainer).
 
 ## Versions
 
@@ -238,6 +239,9 @@ equivalent-generation consumer cards are "known to work", and never mentions
   `pp_od_clk_voltage`, and writes watts to hwmon `power1_cap` from a systemd oneshot. No NixOS
   module exists. **community-reported**
   ([tune_r9700.sh](https://github.com/stew675/r9700-setup/blob/HEAD/tune_r9700.sh))
+- **What a power cap costs here**: against the 300 W default, a 250 W cap costs roughly 4% of
+  prefill and decode throughput and a 230 W cap roughly 7%, the same order in both phases — measured
+  with a 27B model on the second card. **measured here**
 
 ## What this repository does, and what is still open
 
@@ -251,8 +255,10 @@ equivalent-generation consumer cards are "known to work", and never mentions
    `Fail to open libdrm_amdgpu.so` message only appeared while the card was wedged (see "ROCm
    userland in nixpkgs"). Expect `Card Series: N/A` until upstream names the R9700.
 5. `hardware.amdgpu.overdrive` stays off unless undervolting is wanted; if it is turned on, set
-   `ppfeaturemask = "0xfff7ffff"` (nixpkgs#564601). Power-cap scripting stays out of Nix.
-   `amdgpu.gpu_recovery=1` is set, because the driver default is auto/off except SRIOV.
+   `ppfeaturemask = "0xfff7ffff"` (nixpkgs#564601). A oneshot unit in `hardware-amd-gpu.nix` caps
+   both cards at 250 W through hwmon `power1_cap` to keep their fans quiet, and exits non-zero when
+   a card does not take the value; the cap needs no overdrive. `amdgpu.gpu_recovery=1` is set,
+   because the driver default is auto/off except SRIOV.
 6. **Open:** set `HIP_VISIBLE_DEVICES` (and `ROCR_VISIBLE_DEVICES` where honoured) so the iGPU is
    never selected; for llama.cpp prefer `--split-mode layer` or an RPC/pipeline split over tensor
    parallelism, and build with `-DGGML_HIP=ON -DGPU_TARGETS=gfx1201`, no `HSA_OVERRIDE_GFX_VERSION`.

hosts/idols-aquamarine/hardware-amd-gpu.nix

@@ -69,6 +69,46 @@
     pkgs.nvtopPackages.full
   ];
 
+  # Cap both cards at 250 W to quiet their fans; the smaller power draw is a side effect. Measured on
+  # this host (27B, second card), that costs ~4% of prefill and decode throughput against the 300 W
+  # default. amdgpu exposes the cap in microwatts and accepts anything inside [power1_cap_min,
+  # power1_cap_max] — 210-300 W on these cards. The set path (amdgpu_dpm_set_power_limit) has no
+  # overdrive gate, so the ppfeaturemask default stands.
+  systemd.services.amdgpu-power-cap = {
+    description = "Cap the power limit of every amdgpu card at 250 W";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "systemd-modules-load.service" ];
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      path = [ pkgs.coreutils ];
+      ExecStart = pkgs.writeShellScript "amdgpu-power-cap" ''
+        cap=250000000 # microwatts
+        want=2 # both R9700s; this host has no other amdgpu device
+        # The cards probe asynchronously, so retry briefly instead of racing the driver.
+        for _ in $(seq 1 60); do
+          capped=0
+          for hwmon in /sys/class/hwmon/hwmon*; do
+            [ "$(cat "$hwmon/name" 2>/dev/null)" = amdgpu ] || continue
+            # The subshell keeps a rejected write from printing one shell error per retry; the
+            # summary below is the line an operator needs.
+            if (echo "$cap" > "$hwmon/power1_cap") 2>/dev/null &&
+              [ "$(cat "$hwmon/power1_cap" 2>/dev/null)" = "$cap" ]; then
+              capped=$((capped + 1))
+            fi
+          done
+          if [ "$capped" -ge "$want" ]; then
+            echo "capped $capped amdgpu cards at 250 W"
+            exit 0
+          fi
+          sleep 1
+        done
+        echo "amdgpu power cap: only $capped of $want cards took the cap" >&2
+        exit 1
+      '';
+    };
+  };
+
   # GPU panel in btop, and utilization/VRAM in node_exporter for the shared
   # VictoriaMetrics scrape — the same shape as the Radeon iGPU hosts.
   # perfmon: CAP_PERFMON is what btop needs for the i915 iGPU.