mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:46 +01:00
Ryan Yin
eaa53004e5
hardware.amdgpu.opencl.enable also adds clr, whose OpenCL 2.2 libOpenCL.so.1 lands in /run/opengl-driver/lib, first in OpenVINO's RUNPATH, and shadows the ocl-icd loader the GPU plugin links. With clr.icd alone the plugin finds ocl-icd, so the wrapper and WA-038 go. Assisted-by: Claude Code (claude-opus-5-5)
132 lines
5.2 KiB
Markdown
132 lines
5.2 KiB
Markdown
# The Intel iGPU on aquamarine
|
|
|
|
The Arrow Lake-S iGPU (i915) does this host's video work and can run OpenVINO models; the R9700s
|
|
stay on inference. [hardware-intel-igpu.nix](./hardware-intel-igpu.nix) installs the stack. Measured
|
|
2026-10-18, OpenVINO 2026.4.0 / openvino-genai 2026.4.0.0.
|
|
|
|
## Devices
|
|
|
|
```bash
|
|
python -c 'import openvino as ov; c = ov.Core(); print(c.available_devices)'
|
|
# ['CPU', 'GPU', 'NPU']
|
|
python -c 'import openvino as ov; print(ov.Core().get_property("GPU", "FULL_DEVICE_NAME"))'
|
|
# Intel(R) Graphics (iGPU)
|
|
```
|
|
|
|
VA-API uses `/dev/dri/renderD128`:
|
|
|
|
```bash
|
|
nix shell nixpkgs#libva-utils -c vainfo --display drm --device /dev/dri/renderD128
|
|
nix shell nixpkgs#ffmpeg -c ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
|
|
-i in.mkv -c:v hevc_vaapi -qp 26 out.mkv
|
|
```
|
|
|
|
## Running OpenVINO GenAI models
|
|
|
|
Models with static or bounded-dynamic shapes compile on `GPU`; unbounded dynamic shapes fail with
|
|
`dynamic shape without upper bound`. Inspect a model before choosing the device:
|
|
|
|
```bash
|
|
python - /path/to/model.xml <<'PY'
|
|
import sys
|
|
import openvino as ov
|
|
|
|
for port in ov.Core().read_model(sys.argv[1]).inputs:
|
|
print(port.get_any_name(), port.get_partial_shape())
|
|
PY
|
|
```
|
|
|
|
### Whisper needs a static encoder
|
|
|
|
The published `whisper-large-v3-turbo-int8-ov` IRs are fully dynamic. The NPU pipeline reshapes them
|
|
internally and the CPU path runs them as-is, but the GPU path cannot compile the encoder. Reshape it
|
|
once, with batch bounded `0..1` so OpenVINO GenAI's `{0, 128, 3000}` input reset still matches, then
|
|
run the normal pipeline:
|
|
|
|
```bash
|
|
# Store paths change on every rebuild, so read them from the running service.
|
|
MODEL=$(systemctl cat whisper-asr | sed -n 's/^Environment="WHISPER_MODEL_DIR=\([^"]*\)"/\1/p')
|
|
ENVPY=$(systemctl cat whisper-asr | sed -n 's/^ExecStart=\([^ ]*\).*/\1/p')
|
|
OUT=/tmp/whisper-static
|
|
|
|
rm -rf "$OUT" && mkdir -p "$OUT"
|
|
cp "$MODEL"/*.json "$OUT"/
|
|
for f in "$MODEL"/*.bin "$MODEL"/*.xml; do
|
|
case "$(basename "$f")" in
|
|
openvino_encoder_model.*) ;;
|
|
*) ln -s "$f" "$OUT/$(basename "$f")" ;;
|
|
esac
|
|
done
|
|
|
|
"$ENVPY" - <<PY
|
|
import openvino as ov
|
|
|
|
# 128 mel bins for large-v3/turbo, 3000 frames = 30 s window, batch bound 0..1.
|
|
enc = ov.Core().read_model("$MODEL/openvino_encoder_model.xml")
|
|
enc.reshape({"input_features": ov.PartialShape([ov.Dimension(0, 1), 128, 3000])})
|
|
ov.save_model(enc, "$OUT/openvino_encoder_model.xml", compress_to_fp16=False)
|
|
PY
|
|
```
|
|
|
|
Transcribe (mono 16 kHz 16-bit WAV):
|
|
|
|
```bash
|
|
"$ENVPY" - "$OUT" /tmp/jfk.wav en <<'PY'
|
|
import sys
|
|
import wave
|
|
|
|
import numpy as np
|
|
import openvino_genai
|
|
|
|
models, wav_path, language = sys.argv[1:4]
|
|
with wave.open(wav_path) as wav:
|
|
samples = (
|
|
np.frombuffer(wav.readframes(wav.getnframes()), dtype=np.int16).astype(np.float32)
|
|
/ 32768.0
|
|
)
|
|
pipe = openvino_genai.WhisperPipeline(models, "GPU", CACHE_DIR="/tmp/whisper-gpu-cache")
|
|
config = pipe.get_generation_config()
|
|
config.task = "transcribe"
|
|
config.language = f"<|{language}|>"
|
|
print(pipe.generate(samples, config).texts[0].strip())
|
|
PY
|
|
```
|
|
|
|
## Timings and memory
|
|
|
|
Steady state, pinned language, 3-5 iterations after warmup. NPU through the `whisper-asr` service;
|
|
iGPU and CPU standalone. Audio: 11.0 s English, 5.6 s Chinese.
|
|
|
|
| Device | English | Chinese | CPU-s/request | Memory in use |
|
|
| ------ | ------------------ | ------- | ------------- | -------------------- |
|
|
| NPU | 1.47 s (1.46-1.57) | 1.36 s | 0.055 / 0.042 | ~2.2 GiB (resident) |
|
|
| iGPU | 1.49 s | 1.41 s | 0.23 / 0.14 | ~2.1 GiB (on demand) |
|
|
| CPU | 1.23 s | 1.18 s | 9.35 / 8.99 | ~1.1 GiB (on demand) |
|
|
|
|
| Device | Cold compile / first use | Peak memory |
|
|
| ------ | ----------------------------------------------------------------------------------- | ----------- |
|
|
| NPU | ~70 s VPUX compile on a cold `/var/cache/whisper-asr`; warm starts import the blobs | 6.2 GB |
|
|
| iGPU | load 2.2 s instead of 0.7 s warm; first inference 1.8 s instead of 1.5 s | 2.65 GB |
|
|
| CPU | load 0.95 s; first inference 1.85 s instead of 1.23 s | ~2.0 GB |
|
|
|
|
## Language switching
|
|
|
|
Pinned and auto-detect (no `language`) both work on the iGPU and CPU: alternating en/zh in one
|
|
process or service is correct, 1.44-1.79 s on iGPU, 1.20-1.34 s on CPU. On the NPU pinned works
|
|
(1.46/1.36 s); auto-detect works while the idle pipeline is fresh (>=4 s between requests), but an
|
|
auto request right after a pinned one inherited the pinned language and translated the English audio
|
|
to Chinese, correct again after the ~4 s rebuild. A pinned request during that rebuild took 5.25 s.
|
|
Pin the language per client before mixing modes.
|
|
|
|
## Cache
|
|
|
|
A `CACHE_DIR` written by a failed GPU compile produced repeated text; deleting it and recompiling
|
|
fixed the output. Wipe the cache after changing the model or after a failed attempt.
|
|
|
|
## Which device
|
|
|
|
- **Whisper/ASR: NPU.** Same latency as the iGPU, lowest CPU cost, leaves the iGPU free.
|
|
- **iGPU: OpenVINO models that compile on `GPU`, and ASR fallback.** It occupies the GPU for the
|
|
whole request, so transcodes and the console wait.
|
|
- **CPU: emergency only.** Fastest single request, but it keeps ~7.5 cores busy.
|