Aquamarine builds for ai and shoukei across the three platforms, with both
daemons logging in as root. They authenticate with their passphrase-free SSH
host keys, and ssh_known_hosts now carries each builder's address.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
On a tmpfs root the flake/fetch cache is empty at every boot, so
`nix run <flake-ref>` re-downloads its input and fails until DNS is up: a reboot
test showed llama-swap's first start failing on that. Other caches write into
RAM meanwhile.
Assisted-by: DeepSeek Harness (deepseek-flash)
disko creates the swapfile only when it formats the pool, so the live 96G files
keep their size until they are recreated with btrfs filesystem mkswapfile; that
frees 64G per host.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
* fix(ssh): retry and multiplex LAN connections
* fix(ai): shrink the neighbour-probe fallback to ~1s
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
systemd ignores `path` inside [Service], so the unit ran with the default PATH
instead of the coreutils one it asks for.
Assisted-by: DeepSeek Harness (deepseek-flash)
Enabling the module runs `openrgb --server` at boot and installs its udev rules,
so the GUI and the CLI reach the MSI controller without root.
Assisted-by: DeepSeek Harness (deepseek-flash)
The module loads i2c-dev and grants /dev/i2c-* to the i2c group and locally
logged-in users, so ddcutil no longer needs root.
Assisted-by: DeepSeek Harness (deepseek-flash)
The second NVMe and its LUKS volume leave with the Z890 board, so the data-disk disko module is deleted; `ai` must activate this change before the disk is pulled, or its initrd blocks on the missing volume.
Assisted-by: DeepSeek Harness (deepseek-flash)
The Z890 board and its root SSD move here from `ai`, and the retired VM's name and 192.168.5.101 are reused.
Assisted-by: DeepSeek Harness (deepseek-flash)
* flake.lock: Update
nixpkgs only. The new revision builds VictoriaMetrics with cgo
(CGO_ENABLED=1); its pure-Go zstd decoder rejected a stored values block, so
every small-part merge panicked on youko.
* fix(colmena): address the VM hosts by name, not by their IP
`~/.ssh/config` pins the passphrase-protected builder key with IdentitiesOnly
for 192.168.*, so IP-addressed ssh cannot authenticate without a terminal. The
name resolves through the aliases in /etc/ssh/ssh_config.
* fix(monitoring): repair two alert rules VictoriaMetrics rejects
`on(uid)` matches on a label kube-state-metrics no longer emits, and
`!= ON(__name__, instance)` treats every pg_settings series as a duplicate.
They now join on (namespace, pod) and compare with `changes(...[5m])`.
* fix(monitoring): keep disk latency alerts off mechanical disks
Both rules averaged per-op latency with no device or sample guard, so the 4 TB
USB HDD volumes paged constantly (>100 ms/write in 26% of the last 30 days).
They now require a non-rotational physical device and >=5 ops/s.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
No host defines a *-nixos-tests output; the pod firewall drops 5900-5903
from pod sources; the mihomo core keeps geodata and cache in its
StateDirectory; WA-020 was used twice and the cited WA-028 had no row.
microvm.nix's sshSwitch step runs nix-store --load-db against the guest's
read-only virtiofs /nix/store and dies there, so the recipe installs the
runner on the physical host and restarts microvm@<guest> instead.
* feat(mihomo): keep ranges that can never be proxied out of TUN
The list lives in policy.yaml as tun_exclude_address; sources.yaml may override
it. 198.18.0.0/15 stays in TUN: that is the fake-ip range itself.
* feat(mihomo): match Valve's content servers by ASN
The 20 hand-copied prefixes had drifted from what AS32590 announces. The rule
type needs GeoLite2-ASN.mmdb, so geox-url.asn points at the same mirror.
* feat(mihomo): render the gateway config from the same sources
The hand-kept copy had drifted: a manual sync carried the desktop's secret onto
the gateway and dropped external-ui-url. --redact yields a shareable template.
* fix(mihomo): open the link DNS takeover only once the core serves it
The core opens :1053 after parsing rules, and rule parsing is where missing
geodata gets fetched. Flipping at fork time made that fetch SERVFAIL, the config
fatal, and the unit unstartable.
* fix(justfile): keep local endpoints out of the agent proxy
With only HTTP_PROXY set, a request to this host's own llama-swap endpoint went
to mihomo. Node matches exact hosts only, so the literals carry it; the CIDRs
serve curl and Go.
* feat(justfile): seed mihomo geodata into the service state dir
So a cold start never needs the download that the DNS takeover made unstartable.
`just test` has enforced the suite result since #350: the recipe raises unless
`evalTests` is `true`, so a zero exit already means pass. Update the places that
still described the old raw-`nix eval` behaviour: AGENTS.md, README.md,
SECURITY.md, outputs/README.md, and the nix-config-debug / nix-config-new-host /
nix-config-update skills.
The exit-0 caveat is kept only where the raw `nix eval .#evalTests` command is
documented, because that command genuinely exits 0 even on `false`.
BCM4378/brcmfmac intermittently fails Wi-Fi association (ASSOC-REJECT /
authentication timeout) on this laptop. NetworkManager misreads the failure
as a bad key and re-prompts; whichever client reconnects (Noctalia, nmtui,
nmcli) then saves the answer as a duplicate "<SSID> 1" profile. The problem
predates the Noctalia v5 shell and is driven by the driver, not the shell.
- disable Wi-Fi powersave on shoukei, a common source of brcmfmac auth timeouts
- WORKAROUNDS.md WA-024: the powersave mitigation and its removal condition
- WORKAROUNDS.md WA-025: the pending upstream brcmfmac port_authorized fix
- document how to recover a wedged supplicant without rebooting, and that
deleting an active NM profile is what turned a reconnect into a dead link
Refs: https://patchew.org/linux/20260915-brcmfmac-4way-handshake-offload-authenticated-event-v2-1-fbb9cc7ac177@jannau.net/
Apple Silicon's 16K pages mean FEX cannot run natively and qemu-user cannot
run complex x86_64 binaries. Wire the maintained rowanG077/nix-x86-on-aarch64
runtime (FEX inside a 4K muvm microVM) into shoukei so games, Steam, Wine and
other x86/x86_64 programs work, and let the module own the x86 ELF handlers
instead of qemu-user.
The input is pinned to our fork (ryan4yin/nix-x86-on-aarch64) at the reviewed
commit. Dropping boot.binfmt.emulatedSystems also removes x86_64 from
nix.settings.extra-platforms, so the qemu-user/uncached-x86-build gap is
recorded as WA-022.
- flake.nix: add nix-x86-on-aarch64 input (WA-021)
- hosts/12kingdoms-shoukei/game/: programs.x86-on-arm + steam/wine, 6 GiB VM
- hardware-configuration.nix: drop emulatedSystems/preferStaticEmulators
- apple-silicon.nix: drop the now-redundant box64
- docs: WA-012 resolved; WA-021 and WA-022 added; host README updated
* feat(restic): verify the repository after each backup
* feat(youko): snapshot the restic store hourly
* docs(backup): document the snapshot and check layers
* fix(postgres): require a password for superuser over TCP
* test: assert the postgres loopback auth rules
* fix(darwin): keep root-only agenix secrets root-only
* docs(security): record the local auth boundaries and retire WA-015
* fix(nfs): restrict the golden export to its real clients
* fix(libvirt): bind VNC consoles to loopback
* fix(firewall): drop VNC ports on every host
* docs(security): record the NFS and VNC boundaries
KubernetesApiServerLatency fired continuously on the p99 of
apiserver_request_duration_seconds_bucket. Every firing series was
verb="GET", subresource="openapi/v3" (kubelet, kube-controller-manager,
kube-scheduler, apiserver) at ~8s, while real verbs (POST/LIST/DELETE)
stay around 0.2s.
openapi/v3 is a streaming discovery endpoint, like WATCH, so exclude
subresource=~"^openapi/" from the rule.
The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.