idols-ai and aquamarine stop preserving ~/.qwen; its existing data stays under /persistent until deleted by hand.
Assisted-by: DeepSeek Harness (claude-opus-5-5)
* chore(aquamarine): take the NPU stack from nixos-unstable-small
* refactor(aquamarine): load the NPU driver via hardware.cpu.intel.npu
Assisted-by: DeepSeek Harness (claude-opus-5-5)
Preserve ~/.raft across reboots so Raft Computer keeps its enrollment, machine identity, agent state and app data. ~/.local/bin and ~/.config/gh were already persisted.
Assisted-by: DeepSeek Harness (deepseek-flash)
- Add the agent profile (modules/nixos/agent.nix: rootless Podman, linger,
nix-ld) with home/linux/tui.nix as its home role; ruby, kana and
aquamarine use it, so hosts no longer import from hosts/idols-ruby.
- Gate nushell's ~/.secrets source behind modules.shell.secrets.enable,
off on agent hosts.
- Persist the TUI home state on aquamarine's tmpfs root.
- Document host profiles in hosts/README.md; add tests/agent-profile and
drop the x86_64 colmena-parity test, which needs mysecrets.
Assisted-by: Claude Code (claude-opus-5-5)
Move standalone repository docs from the root into docs/ and add a documentation index that links the moved docs and the code-local runbooks.
Update Markdown links, source comments, and the root entry points to the new locations. Keep README.md and AGENTS.md at the root as required entry points.
Assisted-by: DeepSeek Harness (deepseek-flash)
At boot whisper-asr could start before the LAN address was configured and
fail with "could not bind on any address". Let whisper-asr.socket bind it
with FreeBind and have uvicorn take fd 3, like the exporters.
Assisted-by: Claude Code (claude-opus-5-5)
hardware.amdgpu.opencl.enable also adds clr, whose OpenCL 2.2 libOpenCL.so.1
lands in /run/opengl-driver/lib, first in OpenVINO's RUNPATH, and shadows the
ocl-icd loader the GPU plugin links. With clr.icd alone the plugin finds
ocl-icd, so the wrapper and WA-038 go.
Assisted-by: Claude Code (claude-opus-5-5)
amdgpu registers power1_cap with its hwmon device, so a rule on each
card's hwmon add event replaces the oneshot that polled /sys/class/hwmon
for 60 s waiting for both cards to probe.
Assisted-by: Claude Code (claude-opus-5-5)
It was a precaution for the R9700 links, never an observed problem, and
on BIOS 1013 the FADT already disables ASPM ("ACPI FADT declares the
system doesn't support PCIe ASPM"), so the parameter did nothing.
Assisted-by: Claude Code (claude-opus-5-5)
bb899876 moved the proxy gateway to fake-IP for IPv6 and removed
aquamarine's getaddrinfo precedence table, so the row's removal condition
has been met since 2026-10-14; only the row and a comment pointer lingered.
Assisted-by: Claude Code (claude-opus-5-5)
drivers/nvme/host/core.c defaults default_ps_max_latency_us to 100000, so
the "relaxed" APST cap on ai and its copy on aquamarine set the value the
kernel already uses. Keep ai's history and rollback recipe as a comment,
and stop the aquamarine pcie_aspm=off comment from citing WA-008, which
never covered it. Removes WA-008.
Assisted-by: Claude Code (claude-opus-5-5)
Run nginx on youko as a loopback pull-through HTTP cache for
cache.nixos.org and cache.numtide.com, exposed by Caddy at
https://nix-cache.writefor.fun.
Assisted-by: DeepSeek Harness (deepseek-flash)
NixOS copies mode-carrying /etc entries such as authorized_keys.d during
stage-2 activation, which runs before systemd starts. preservation mounted the
persisted /etc/ssh from preservation.target afterwards and hid that copy, so a
key change only took effect on `switch`: revoking a key and rebooting silently
restored the old one.
Prepare the mount in initrd, before activation runs, and cover it with a VM
test that seeds a stale authorized_keys file, reboots, and asserts the declared
key replaced it. The same test fails on that assertion with inInitrd disabled.
Verified on all four hosts: /persistent/etc/ssh/authorized_keys.d/root is now
rewritten at boot instead of keeping the timestamp of the last switch.
Assisted-by: Claude Code (claude-opus-5-5)
The homelab hosts accept root SSH with long-lived file keys, run several
exposed services with almost no systemd sandboxing, and keep no audit trail of
privileged actions. Close those three gaps, weighing each measure against its
cost instead of adopting a policy wholesale.
SSH
- Turn off KbdInteractiveAuthentication: nixpkgs defaults it to true and PAM
then still accepts the Unix password despite PasswordAuthentication = false.
- authorizedKeysInHomedir = false: only keys declared in Nix take effect;
youko keeps Gitea's forced-command file through `Match User gitea`.
- Pin aquamarine's passphrase-free remote-builder keys to
`restrict,command="nix-daemon --stdio --store daemon"`, so builds run in the
system daemon instead of an audited root login session.
systemd sandboxing
- Add the mylib.systemdHardening preset and apply it to the exposed, weakly
sandboxed services: caddy 7.8 -> 1.5, vmalert 8.2 -> 1.4, rustfs 6.6 -> 1.4,
whisper-asr 7.7 -> 1.6. An eval test pins the key restrictions.
- Rejected: a global all-keys policy (<=0.2 score gains on services nixpkgs
already sandboxes) and per-service confinement (bind paths to maintain).
auditd on all four hosts, reusing nixpkgs' loader and unit
- Audit login sessions only: a first `-a never,task -F auid=unset` rule keeps
services out of the per-syscall audit path. A compromised service is
therefore unrecorded until it opens a session.
- Record root commands, credential and host-key access, writes to SSH, account,
PAM, Nix-trust and audit configuration, module loads and session denials.
- Rejected: global denial logging, service-UID watches and
--loginuid-immutable (the kernel already records loginuid changes).
Verified on all four hosts after deployment: zero failed units, k3s 6/6 Ready,
and the exposed services healthy.
Assisted-by: Claude Code (claude-opus-5-5)
* chore(nix): update nixpkgs to e62746b and fix 26.11 breakage
Bump nixpkgs to e62746b and preservation to 00be7f7, then adapt to the removals and renamed options in the new revision: drop the removed boot.initrd.systemd.enable settings, trim the stale emSign CA blacklist on Linux while keeping it for the older Darwin input, and migrate Avahi publishing to settings.publish. Work around sunshine still setting the legacy Avahi options and keep dsh-nu on its tested Nushell 0.115.1. Patch NUR dsh's pnpm call to pass nodejs = null, and update pyclipsync for the stdenv.hostPlatform.system rename.
* chore(nix): refresh flake inputs for 26.11
- update nixpkgs-darwin to the same e62746b revision as nixpkgs
- make dsh-nu follow main nixpkgs and patch its testedWith manifest for the resolved Nushell
- refresh floating inputs; update lanzaboote, tuios, and ponytail
- drop the nixpkgs-2505 and rust-overlay workarounds
- keep pyclipsync on main and simplify the cross-platform CA blacklist
- adopt home-manager's Nushell loadSessionVariables default
* fix(gitea): use ROOT_URL for nixpkgs 26.11
nixpkgs removed the legacy settings.server.DOMAIN option; the existing ROOT_URL already carries the public URL.
* chore(nix): update dsh-nu to 0.7.0
Assisted-by: DeepSeek Harness (deepseek-flash)
Update the mysecrets lock to 7f93490, which renames secrets.nix to agenix-rules.nix, drops the deprecated ./ key prefixes for agenix 0.18, and re-encrypts the access-token secret. Update the docs and agenix skill to the new filename and path convention.
* feat(aquamarine): make the iGPU usable for media and OpenVINO
The R9700s stay reserved for inference: VA-API/QSV transcoding moves to the iGPU media
engine, and `openvino-gpu` works around ROCm's OpenCL loader shadowing the plugin's
`ocl-icd` (WA-038).
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-v4.1-flash)
* docs(aquamarine): document OpenVINO on the iGPU
How to run models on the iGPU: the `openvino-gpu` wrapper, the encoder staticization Whisper needs
on `GPU`, compile/steady timings and memory, and the language and cache caveats.
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-v4.1-flash)
The goal is speech recognition on the NPU for vinput, and sherpa-onnx cannot
deliver it: every model it loads keeps a dynamic time axis, which the NPU
compiler rejects outright. Qwen3-ASR and SenseVoice both core-dumped in
vpux-compiler ("Got non broadcastable dimensions pair"), and reshaping the ONNX
inputs does not help because the dynamic dimensions sit inside the graph
(WA-037). Whisper is the only family OpenVINO publishes fully static IRs for,
and its fixed 30 s window is exactly what the compiler accepts.
The host now runs an OpenAI-compatible /v1/audio/transcriptions service backed
by whisper-large-v3-turbo int8 on the NPU: 1.5 s for 11 s of audio against
2.1 s on CPU, with the compiled graph cached in /var/cache so only a cold cache
pays the ~60 s NPU compile. vinput on the desktop sends raw PCM to it through a
small provider script.
Only the NPU driver and compiler still come from the unmerged branch in the
`nixpkgs-npu` input (NixOS/nixpkgs#570230), whose nixpkgs also happens to be
the one where openvino-genai 2026.4.0.0 builds at all.
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-v4.1-flash)
Aquamarine builds for ai and shoukei across the three platforms, with both
daemons logging in as root. They authenticate with their passphrase-free SSH
host keys, and ssh_known_hosts now carries each builder's address.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
On a tmpfs root the flake/fetch cache is empty at every boot, so
`nix run <flake-ref>` re-downloads its input and fails until DNS is up: a reboot
test showed llama-swap's first start failing on that. Other caches write into
RAM meanwhile.
Assisted-by: DeepSeek Harness (deepseek-flash)
disko creates the swapfile only when it formats the pool, so the live 96G files
keep their size until they are recreated with btrfs filesystem mkswapfile; that
frees 64G per host.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
systemd ignores `path` inside [Service], so the unit ran with the default PATH
instead of the coreutils one it asks for.
Assisted-by: DeepSeek Harness (deepseek-flash)
The Z890 board and its root SSD move here from `ai`, and the retired VM's name and 192.168.5.101 are reused.
Assisted-by: DeepSeek Harness (deepseek-flash)
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:
- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
drop the explicit import list, leaving aquamarine-services.nix with
just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload
The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.
- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
The restic job needs three agenix secrets that do not exist yet (restic-password, ssh-key-for-restic-backup and an rclone config for the smb-downloads remote). Comment it out until they are provisioned so it does not fail silently every night.
Load both *.yml and *.yaml alert rule files: the Loki rules were named .yaml while only *.yml was loaded, so they silently never ran. Fix the Grafana domain typo (writefo.fun -> writefor.fun). Comment out the postgres-playground datasource, which used the Grafana admin password for a role that has no password and could never authenticate.
* feat(btrbk): make snapshots explicit and guarded
Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.
Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.
Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.
* feat(monitoring): alert on btrbk snapshot failure
Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
Ryan Yin
ryan4yin | 二花