mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:46 +01:00
fix(aquamarine): install only clr's OpenCL ICD instead of the openvino-gpu wrapper
hardware.amdgpu.opencl.enable also adds clr, whose OpenCL 2.2 libOpenCL.so.1 lands in /run/opengl-driver/lib, first in OpenVINO's RUNPATH, and shadows the ocl-icd loader the GPU plugin links. With clr.icd alone the plugin finds ocl-icd, so the wrapper and WA-038 go. Assisted-by: Claude Code (claude-opus-5-5)
This commit is contained in:
Ryan Yin authored and ryan4yin | 二花 committed
1 parent
fe280d8fa2
commit
eaa53004e5
8 files changed
+30
-49
No files matched your search
+7
-8
@@ -63,14 +63,13 @@ only delete inactive duplicates.
|
||||
|
||||
A known limitation with no fix yet.
|
||||
|
||||
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
|
||||
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------- | ------ |
|
||||
| WA-016 | subscription rule snapshots — `modules/nixos/desktop/networking/mihomo/` (`rules_from:`) | mihomo's `proxy-providers` imports proxies only, so the `rules` and `proxy-groups` a subscription ships with are dropped by the core; the generator re-imports them from a local YAML copy that has to be refreshed by hand when the provider changes its routing policy | Drop the snapshots once mihomo can import rules from a proxy-provider, or the providers publish their rules as rule-providers | 2026-10-04 | when mihomo gains rule import | active |
|
||||
| WA-022 | qemu-user binfmt removed — `hosts/12kingdoms-shoukei/hardware-configuration.nix` | the x86-on-arm module owns the x86 ELF handlers and this host does not need riscv64, but dropping `boot.binfmt.emulatedSystems` also drops `nix.settings.extra-platforms` and qemu-user, so `podman run --platform=linux/amd64`, x86 chroots and local builds of uncached x86 derivations need a remote x86_64 builder | Restore `boot.binfmt.emulatedSystems` (without x86_64/i686, which the module asserts against) if such a workload is needed | 2026-10-07 | when an amd64 container/chroot or uncached x86 build is needed | active |
|
||||
| WA-025 | brcmfmac 4-way-handshake offload authentication — upstream kernel/wpa_supplicant, no code in this repo | Since hostap commit 416386060, wpa_supplicant 2.11+ expects `NL80211_CMD_PORT_AUTHORIZED` from drivers that offload the 4-way handshake; brcmfmac never reports it, so authentication intermittently fails on Apple Silicon (BRCM4378/4387/4388). nixpkgs already carries the Broadcom revert in wpa_supplicant 2.12; the proper fix, Janne Grunau's `wifi: brcmfmac: cfg80211: Report port_authorized for 4-way HS offload` (2026-09), is still under review. Entering the key again after such a failure only creates a duplicate profile, so cancel and retry instead | Fix lands via a nixpkgs kernel update (or by carrying the patch through `nixpkgs-patched`) | 2026-10-07 | on every kernel bump | active |
|
||||
| WA-033 | USB-C/XHCI port `_PLD` ACPI noise — this board's DSDT, no code in this repo | every boot logs 16 `ACPI Error: Aborting method \_SB.PC00.TXHC.RHUB.SS01._PLD ... (AE_NOT_FOUND)` lines, plus `\_SB.PC02.XHCI.RHUB.HS01._PLD`; the ports work, and these messages are unrelated to the GPU's ATPX failure, which names `\_SB.PC00.GFX0.ATPX` instead | Drop if a BIOS update defines the `_PLD` descriptors | 2026-10-14 | on a BIOS update | active |
|
||||
| WA-035 | SSH retry + multiplexing to LAN peers, ARP fallback tuning — `home/base/tui/ssh.nix`, `hosts/idols-ai/neighbour-probe.nix` | the desktop reaches the LAN over a Mesh WiFi backhaul, which drops unicast to a peer whose MAC it has aged out of its forwarding table while still flooding broadcast: a capture shows seven SYNs from one socket addressed to the cached MAC with no answer, three unicast ARP probes unanswered, then a broadcast ARP answered in 0.4 ms after ~12 s of failed connects. The ssh settings retry a short connect and keep a master warm so the entry stays fresh; the neighbour-probe tunings walk the kernel to that broadcast fallback in ~1 s | Drop once the backhaul is wired or the mesh stops dropping unknown unicast | 2026-10-16 | when the backhaul changes | active |
|
||||
| WA-038 | OpenVINO GPU plugin needs an `ocl-icd` loader — [hosts/idols-aquamarine/hardware-intel-igpu.nix](./hosts/idols-aquamarine/hardware-intel-igpu.nix) (`openvino-gpu` wrapper) | `/run/opengl-driver/lib/libOpenCL.so.1` comes from ROCm's clr, which provides OpenCL 2.2 symbols, while OpenVINO's GPU plugin links the OpenCL 3.0 loader (`ocl-icd`): the plugin fails to load and OpenVINO silently omits GPU from `available_devices`. The wrapper runs a command with `ocl-icd` first on `LD_LIBRARY_PATH` and `OCL_ICD_VENDORS` pinned to the Intel ICD | Drop once nixpkgs' openvino GPU plugin carries an explicit `ocl-icd` RUNPATH, or ROCm's clr ships an OpenCL 3.0 loader | 2026-10-18 | on an openvino or ROCm bump | active |
|
||||
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
|
||||
| ------ | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------- | ------ |
|
||||
| WA-016 | subscription rule snapshots — `modules/nixos/desktop/networking/mihomo/` (`rules_from:`) | mihomo's `proxy-providers` imports proxies only, so the `rules` and `proxy-groups` a subscription ships with are dropped by the core; the generator re-imports them from a local YAML copy that has to be refreshed by hand when the provider changes its routing policy | Drop the snapshots once mihomo can import rules from a proxy-provider, or the providers publish their rules as rule-providers | 2026-10-04 | when mihomo gains rule import | active |
|
||||
| WA-022 | qemu-user binfmt removed — `hosts/12kingdoms-shoukei/hardware-configuration.nix` | the x86-on-arm module owns the x86 ELF handlers and this host does not need riscv64, but dropping `boot.binfmt.emulatedSystems` also drops `nix.settings.extra-platforms` and qemu-user, so `podman run --platform=linux/amd64`, x86 chroots and local builds of uncached x86 derivations need a remote x86_64 builder | Restore `boot.binfmt.emulatedSystems` (without x86_64/i686, which the module asserts against) if such a workload is needed | 2026-10-07 | when an amd64 container/chroot or uncached x86 build is needed | active |
|
||||
| WA-025 | brcmfmac 4-way-handshake offload authentication — upstream kernel/wpa_supplicant, no code in this repo | Since hostap commit 416386060, wpa_supplicant 2.11+ expects `NL80211_CMD_PORT_AUTHORIZED` from drivers that offload the 4-way handshake; brcmfmac never reports it, so authentication intermittently fails on Apple Silicon (BRCM4378/4387/4388). nixpkgs already carries the Broadcom revert in wpa_supplicant 2.12; the proper fix, Janne Grunau's `wifi: brcmfmac: cfg80211: Report port_authorized for 4-way HS offload` (2026-09), is still under review. Entering the key again after such a failure only creates a duplicate profile, so cancel and retry instead | Fix lands via a nixpkgs kernel update (or by carrying the patch through `nixpkgs-patched`) | 2026-10-07 | on every kernel bump | active |
|
||||
| WA-033 | USB-C/XHCI port `_PLD` ACPI noise — this board's DSDT, no code in this repo | every boot logs 16 `ACPI Error: Aborting method \_SB.PC00.TXHC.RHUB.SS01._PLD ... (AE_NOT_FOUND)` lines, plus `\_SB.PC02.XHCI.RHUB.HS01._PLD`; the ports work, and these messages are unrelated to the GPU's ATPX failure, which names `\_SB.PC00.GFX0.ATPX` instead | Drop if a BIOS update defines the `_PLD` descriptors | 2026-10-14 | on a BIOS update | active |
|
||||
| WA-035 | SSH retry + multiplexing to LAN peers, ARP fallback tuning — `home/base/tui/ssh.nix`, `hosts/idols-ai/neighbour-probe.nix` | the desktop reaches the LAN over a Mesh WiFi backhaul, which drops unicast to a peer whose MAC it has aged out of its forwarding table while still flooding broadcast: a capture shows seven SYNs from one socket addressed to the cached MAC with no answer, three unicast ARP probes unanswered, then a broadcast ARP answered in 0.4 ms after ~12 s of failed connects. The ssh settings retry a short connect and keep a master warm so the entry stays fresh; the neighbour-probe tunings walk the kernel to that broadcast fallback in ~1 s | Drop once the backhaul is wired or the mesh stops dropping unknown unicast | 2026-10-16 | when the backhaul changes | active |
|
||||
|
||||
## WIP
|
||||
|
||||
|
||||
@@ -1,19 +1,15 @@
|
||||
# The Intel iGPU on aquamarine
|
||||
|
||||
The Arrow Lake-S iGPU (i915) does this host's video work and can run OpenVINO models; the R9700s
|
||||
stay on inference. [hardware-intel-igpu.nix](./hardware-intel-igpu.nix) installs the stack and the
|
||||
`openvino-gpu` wrapper. Measured 2026-10-18, OpenVINO 2026.4.0 / openvino-genai 2026.4.0.0.
|
||||
stay on inference. [hardware-intel-igpu.nix](./hardware-intel-igpu.nix) installs the stack. Measured
|
||||
2026-10-18, OpenVINO 2026.4.0 / openvino-genai 2026.4.0.0.
|
||||
|
||||
## Devices
|
||||
|
||||
`openvino-gpu` puts the OpenCL 3.0 loader (`ocl-icd`) ahead of ROCm's clr and pins the Intel ICD;
|
||||
without it the GPU plugin fails to load and GPU is missing from the device list (WA-038). Use it for
|
||||
every OpenVINO run:
|
||||
|
||||
```bash
|
||||
openvino-gpu python -c 'import openvino as ov; c = ov.Core(); print(c.available_devices)'
|
||||
python -c 'import openvino as ov; c = ov.Core(); print(c.available_devices)'
|
||||
# ['CPU', 'GPU', 'NPU']
|
||||
openvino-gpu python -c 'import openvino as ov; print(ov.Core().get_property("GPU", "FULL_DEVICE_NAME"))'
|
||||
python -c 'import openvino as ov; print(ov.Core().get_property("GPU", "FULL_DEVICE_NAME"))'
|
||||
# Intel(R) Graphics (iGPU)
|
||||
```
|
||||
|
||||
@@ -31,7 +27,7 @@ Models with static or bounded-dynamic shapes compile on `GPU`; unbounded dynamic
|
||||
`dynamic shape without upper bound`. Inspect a model before choosing the device:
|
||||
|
||||
```bash
|
||||
openvino-gpu python - /path/to/model.xml <<'PY'
|
||||
python - /path/to/model.xml <<'PY'
|
||||
import sys
|
||||
import openvino as ov
|
||||
|
||||
@@ -62,7 +58,7 @@ for f in "$MODEL"/*.bin "$MODEL"/*.xml; do
|
||||
esac
|
||||
done
|
||||
|
||||
openvino-gpu "$ENVPY" - <<PY
|
||||
"$ENVPY" - <<PY
|
||||
import openvino as ov
|
||||
|
||||
# 128 mel bins for large-v3/turbo, 3000 frames = 30 s window, batch bound 0..1.
|
||||
@@ -75,7 +71,7 @@ PY
|
||||
Transcribe (mono 16 kHz 16-bit WAV):
|
||||
|
||||
```bash
|
||||
openvino-gpu "$ENVPY" - "$OUT" /tmp/jfk.wav en <<'PY'
|
||||
"$ENVPY" - "$OUT" /tmp/jfk.wav en <<'PY'
|
||||
import sys
|
||||
import wave
|
||||
|
||||
|
||||
@@ -68,11 +68,10 @@ run on `youko`, not on this host, so opening a dashboard does not wake `aquamari
|
||||
The iGPU carries display and video work: the kernel binds `i915`, and
|
||||
[hardware-intel-igpu.nix](./hardware-intel-igpu.nix) installs `intel-media-driver` (VA-API),
|
||||
`vpl-gpu-rt` (oneVPL/QSV) and `intel-compute-runtime` (OpenCL/Level-Zero), so transcoding can use
|
||||
its media engine while the R9700s stay on inference. VA-API targets `/dev/dri/renderD128`, and
|
||||
OpenVINO jobs run through `openvino-gpu <command>`; [IGPU.md](./IGPU.md) records the wrapper, the
|
||||
static-shape workaround Whisper needs on `GPU`, and the measured NPU/iGPU/CPU timings. A monitor
|
||||
plugged into the board's HDMI/DP ports still gets the text console (`multi-user.target`, no display
|
||||
manager) when SSH is unreachable.
|
||||
its media engine while the R9700s stay on inference. VA-API targets `/dev/dri/renderD128`;
|
||||
[IGPU.md](./IGPU.md) records the static-shape workaround Whisper needs on `GPU`, and the measured
|
||||
NPU/iGPU/CPU timings. A monitor plugged into the board's HDMI/DP ports still gets the text console
|
||||
(`multi-user.target`, no display manager) when SSH is unreachable.
|
||||
|
||||
`iommu=pt` is set in [hardware-amd-gpu.nix](./hardware-amd-gpu.nix) because ROCm documents hangs on
|
||||
multi-GPU systems without IOMMU passthrough. The same file disables amdgpu runtime power management:
|
||||
|
||||
@@ -249,8 +249,9 @@ equivalent-generation consumer cards are "known to work", and never mentions
|
||||
rule, which would leave the driver registered as a PX client. Revisit if a BIOS update defines
|
||||
the `_OFF`/`_ON` methods — or now that two cards may keep ATPX out of the picture (above).
|
||||
2. `iommu=pt` stays; `iommu=off` for P2P is a community claim with no AMD statement.
|
||||
3. `hardware.amdgpu.opencl.enable = true` is the shape the host uses — the module adds both `clr`
|
||||
and `clr.icd` — instead of a hand-written `hardware.graphics.extraPackages`.
|
||||
3. Only `rocmPackages.clr.icd` goes into `hardware.graphics.extraPackages`: the `clr` that
|
||||
`hardware.amdgpu.opencl.enable` also adds puts its OpenCL 2.2 `libOpenCL.so.1` in
|
||||
`/run/opengl-driver/lib`, ahead of the `ocl-icd` that OpenVINO's GPU plugin links.
|
||||
4. `rocmPackages.rocm-smi` is not wrapped for libdrm. It reads a healthy card completely; the
|
||||
`Fail to open libdrm_amdgpu.so` message only appeared while the card was wedged (see "ROCm
|
||||
userland in nixpkgs"). Expect `Card Series: N/A` until upstream names the R9700.
|
||||
|
||||
@@ -41,8 +41,8 @@
|
||||
# nixpkgs still ships 7.2.x while the upgrade awaits split reviews, LLVM/OpenMP
|
||||
# fixes and compatibility work: https://github.com/NixOS/nixpkgs/pull/573885.
|
||||
# Inference containers carry their own ROCm userspace.
|
||||
# The ROCm OpenCL ICD, which is also what HIP consumers resolve for compute.
|
||||
hardware.amdgpu.opencl.enable = true;
|
||||
# Not hardware.amdgpu.opencl.enable: its clr adds an OpenCL 2.2 libOpenCL.so.1 that shadows ocl-icd.
|
||||
hardware.graphics.extraPackages = [ pkgs.rocmPackages.clr.icd ];
|
||||
|
||||
# /dev/kfd is world-writable under nixpkgs' default udev rule, but
|
||||
# /dev/dri/renderD* is `render` 0660 and this headless host has no logged-in
|
||||
|
||||
@@ -14,20 +14,4 @@
|
||||
vpl-gpu-rt
|
||||
intel-compute-runtime
|
||||
];
|
||||
|
||||
# OpenVINO's GPU plugin links the OpenCL 3.0 loader (ocl-icd), but
|
||||
# `/run/opengl-driver/lib/libOpenCL.so.1` comes from ROCm's clr (OpenCL 2.2), so the plugin
|
||||
# fails to load and OpenVINO silently drops GPU from `available_devices`. Run OpenVINO jobs
|
||||
# through this wrapper to put ocl-icd first and pin the Intel ICD (WORKAROUNDS.md, WA-038).
|
||||
# See IGPU.md for running models on the iGPU.
|
||||
environment.systemPackages = [
|
||||
(pkgs.writeShellApplication {
|
||||
name = "openvino-gpu";
|
||||
text = ''
|
||||
export LD_LIBRARY_PATH="${pkgs.ocl-icd}/lib''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||
export OCL_ICD_VENDORS="${pkgs.intel-compute-runtime}/etc/OpenCL/vendors"
|
||||
exec "$@"
|
||||
'';
|
||||
})
|
||||
];
|
||||
}
|
||||
@@ -15,7 +15,7 @@
|
||||
btopShowsRadeon = true;
|
||||
btopReadsIgpu = true;
|
||||
igpuMediaStackInstalled = true;
|
||||
openvinoGpuWrapperInstalled = true;
|
||||
rocmOpenclIcdOnly = true;
|
||||
gpuMetricsExported = true;
|
||||
raplMetricsExported = true;
|
||||
raplPermissionsExposed = true;
|
||||
|
||||
@@ -76,10 +76,12 @@ in
|
||||
"vpl-gpu-rt"
|
||||
"intel-compute-runtime"
|
||||
];
|
||||
# OpenVINO's GPU plugin needs ocl-icd ahead of ROCm's clr loader (WA-038).
|
||||
openvinoGpuWrapperInstalled = builtins.any (
|
||||
package: (package.name or "") == "openvino-gpu"
|
||||
) cfg.environment.systemPackages;
|
||||
rocmOpenclIcdOnly =
|
||||
let
|
||||
graphics = cfg.hardware.graphics.extraPackages;
|
||||
in
|
||||
builtins.elem eval.pkgs.rocmPackages.clr.icd graphics
|
||||
&& !builtins.elem eval.pkgs.rocmPackages.clr graphics;
|
||||
gpuMetricsExported = builtins.elem "drm" cfg.services.prometheus.exporters.node.enabledCollectors;
|
||||
raplMetricsExported = builtins.elem "rapl" cfg.services.prometheus.exporters.node.enabledCollectors;
|
||||
raplPermissionsExposed = lib.hasInfix "chgrp rapl" cfg.services.udev.extraRules;
|
||||
|
||||
Reference in new issue
Block a user