diff --git a/hosts/idols-aquamarine/ROCM.md b/hosts/idols-aquamarine/ROCM.md index e4219d2b..a7092ac7 100644 --- a/hosts/idols-aquamarine/ROCM.md +++ b/hosts/idols-aquamarine/ROCM.md @@ -3,8 +3,9 @@ `aquamarine` runs Radeon AI PRO R9700 cards (Navi 48, RDNA4, PCI ID `1002:7551`, ROCm target `gfx1201`) as headless compute devices; the Intel iGPU drives the console. This note records which versions support the card, why amdgpu runtime power management leaves it unusable, and which -decisions that produced. Claims are marked **verified** (primary source), **community-reported**, or -**inference** (read from kernel source, not confirmed by a maintainer). +decisions that produced. Claims are marked **verified** (primary source), **community-reported**, +**measured here** (taken on this machine), or **inference** (read from kernel source, not confirmed +by a maintainer). ## Versions @@ -238,6 +239,9 @@ equivalent-generation consumer cards are "known to work", and never mentions `pp_od_clk_voltage`, and writes watts to hwmon `power1_cap` from a systemd oneshot. No NixOS module exists. **community-reported** ([tune_r9700.sh](https://github.com/stew675/r9700-setup/blob/HEAD/tune_r9700.sh)) +- **What a power cap costs here**: against the 300 W default, a 250 W cap costs roughly 4% of + prefill and decode throughput and a 230 W cap roughly 7%, the same order in both phases — measured + with a 27B model on the second card. **measured here** ## What this repository does, and what is still open @@ -251,8 +255,10 @@ equivalent-generation consumer cards are "known to work", and never mentions `Fail to open libdrm_amdgpu.so` message only appeared while the card was wedged (see "ROCm userland in nixpkgs"). Expect `Card Series: N/A` until upstream names the R9700. 5. `hardware.amdgpu.overdrive` stays off unless undervolting is wanted; if it is turned on, set - `ppfeaturemask = "0xfff7ffff"` (nixpkgs#564601). Power-cap scripting stays out of Nix. - `amdgpu.gpu_recovery=1` is set, because the driver default is auto/off except SRIOV. + `ppfeaturemask = "0xfff7ffff"` (nixpkgs#564601). A oneshot unit in `hardware-amd-gpu.nix` caps + both cards at 250 W through hwmon `power1_cap` to keep their fans quiet, and exits non-zero when + a card does not take the value; the cap needs no overdrive. `amdgpu.gpu_recovery=1` is set, + because the driver default is auto/off except SRIOV. 6. **Open:** set `HIP_VISIBLE_DEVICES` (and `ROCR_VISIBLE_DEVICES` where honoured) so the iGPU is never selected; for llama.cpp prefer `--split-mode layer` or an RPC/pipeline split over tensor parallelism, and build with `-DGGML_HIP=ON -DGPU_TARGETS=gfx1201`, no `HSA_OVERRIDE_GFX_VERSION`. diff --git a/hosts/idols-aquamarine/hardware-amd-gpu.nix b/hosts/idols-aquamarine/hardware-amd-gpu.nix index a5ff6551..67b10725 100644 --- a/hosts/idols-aquamarine/hardware-amd-gpu.nix +++ b/hosts/idols-aquamarine/hardware-amd-gpu.nix @@ -69,6 +69,46 @@ pkgs.nvtopPackages.full ]; + # Cap both cards at 250 W to quiet their fans; the smaller power draw is a side effect. Measured on + # this host (27B, second card), that costs ~4% of prefill and decode throughput against the 300 W + # default. amdgpu exposes the cap in microwatts and accepts anything inside [power1_cap_min, + # power1_cap_max] — 210-300 W on these cards. The set path (amdgpu_dpm_set_power_limit) has no + # overdrive gate, so the ppfeaturemask default stands. + systemd.services.amdgpu-power-cap = { + description = "Cap the power limit of every amdgpu card at 250 W"; + wantedBy = [ "multi-user.target" ]; + after = [ "systemd-modules-load.service" ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + path = [ pkgs.coreutils ]; + ExecStart = pkgs.writeShellScript "amdgpu-power-cap" '' + cap=250000000 # microwatts + want=2 # both R9700s; this host has no other amdgpu device + # The cards probe asynchronously, so retry briefly instead of racing the driver. + for _ in $(seq 1 60); do + capped=0 + for hwmon in /sys/class/hwmon/hwmon*; do + [ "$(cat "$hwmon/name" 2>/dev/null)" = amdgpu ] || continue + # The subshell keeps a rejected write from printing one shell error per retry; the + # summary below is the line an operator needs. + if (echo "$cap" > "$hwmon/power1_cap") 2>/dev/null && + [ "$(cat "$hwmon/power1_cap" 2>/dev/null)" = "$cap" ]; then + capped=$((capped + 1)) + fi + done + if [ "$capped" -ge "$want" ]; then + echo "capped $capped amdgpu cards at 250 W" + exit 0 + fi + sleep 1 + done + echo "amdgpu power cap: only $capped of $want cards took the cap" >&2 + exit 1 + ''; + }; + }; + # GPU panel in btop, and utilization/VRAM in node_exporter for the shared # VictoriaMetrics scrape — the same shape as the Radeon iGPU hosts. # perfmon: CAP_PERFMON is what btop needs for the i915 iGPU.