* fix(ssh): retry and multiplex LAN connections
* fix(ai): shrink the neighbour-probe fallback to ~1s
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
systemd ignores `path` inside [Service], so the unit ran with the default PATH
instead of the coreutils one it asks for.
Assisted-by: DeepSeek Harness (deepseek-flash)
Enabling the module runs `openrgb --server` at boot and installs its udev rules,
so the GUI and the CLI reach the MSI controller without root.
Assisted-by: DeepSeek Harness (deepseek-flash)
The module loads i2c-dev and grants /dev/i2c-* to the i2c group and locally
logged-in users, so ddcutil no longer needs root.
Assisted-by: DeepSeek Harness (deepseek-flash)
The second NVMe and its LUKS volume leave with the Z890 board, so the data-disk disko module is deleted; `ai` must activate this change before the disk is pulled, or its initrd blocks on the missing volume.
Assisted-by: DeepSeek Harness (deepseek-flash)
The Z890 board and its root SSD move here from `ai`, and the retired VM's name and 192.168.5.101 are reused.
Assisted-by: DeepSeek Harness (deepseek-flash)
* flake.lock: Update
nixpkgs only. The new revision builds VictoriaMetrics with cgo
(CGO_ENABLED=1); its pure-Go zstd decoder rejected a stored values block, so
every small-part merge panicked on youko.
* fix(colmena): address the VM hosts by name, not by their IP
`~/.ssh/config` pins the passphrase-protected builder key with IdentitiesOnly
for 192.168.*, so IP-addressed ssh cannot authenticate without a terminal. The
name resolves through the aliases in /etc/ssh/ssh_config.
* fix(monitoring): repair two alert rules VictoriaMetrics rejects
`on(uid)` matches on a label kube-state-metrics no longer emits, and
`!= ON(__name__, instance)` treats every pg_settings series as a duplicate.
They now join on (namespace, pod) and compare with `changes(...[5m])`.
* fix(monitoring): keep disk latency alerts off mechanical disks
Both rules averaged per-op latency with no device or sample guard, so the 4 TB
USB HDD volumes paged constantly (>100 ms/write in 26% of the last 30 days).
They now require a non-rotational physical device and >=5 ops/s.
Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
No host defines a *-nixos-tests output; the pod firewall drops 5900-5903
from pod sources; the mihomo core keeps geodata and cache in its
StateDirectory; WA-020 was used twice and the cited WA-028 had no row.
microvm.nix's sshSwitch step runs nix-store --load-db against the guest's
read-only virtiofs /nix/store and dies there, so the recipe installs the
runner on the physical host and restarts microvm@<guest> instead.
* feat(mihomo): keep ranges that can never be proxied out of TUN
The list lives in policy.yaml as tun_exclude_address; sources.yaml may override
it. 198.18.0.0/15 stays in TUN: that is the fake-ip range itself.
* feat(mihomo): match Valve's content servers by ASN
The 20 hand-copied prefixes had drifted from what AS32590 announces. The rule
type needs GeoLite2-ASN.mmdb, so geox-url.asn points at the same mirror.
* feat(mihomo): render the gateway config from the same sources
The hand-kept copy had drifted: a manual sync carried the desktop's secret onto
the gateway and dropped external-ui-url. --redact yields a shareable template.
* fix(mihomo): open the link DNS takeover only once the core serves it
The core opens :1053 after parsing rules, and rule parsing is where missing
geodata gets fetched. Flipping at fork time made that fetch SERVFAIL, the config
fatal, and the unit unstartable.
* fix(justfile): keep local endpoints out of the agent proxy
With only HTTP_PROXY set, a request to this host's own llama-swap endpoint went
to mihomo. Node matches exact hosts only, so the literals carry it; the CIDRs
serve curl and Go.
* feat(justfile): seed mihomo geodata into the service state dir
So a cold start never needs the download that the DNS takeover made unstartable.
`just test` has enforced the suite result since #350: the recipe raises unless
`evalTests` is `true`, so a zero exit already means pass. Update the places that
still described the old raw-`nix eval` behaviour: AGENTS.md, README.md,
SECURITY.md, outputs/README.md, and the nix-config-debug / nix-config-new-host /
nix-config-update skills.
The exit-0 caveat is kept only where the raw `nix eval .#evalTests` command is
documented, because that command genuinely exits 0 even on `false`.
BCM4378/brcmfmac intermittently fails Wi-Fi association (ASSOC-REJECT /
authentication timeout) on this laptop. NetworkManager misreads the failure
as a bad key and re-prompts; whichever client reconnects (Noctalia, nmtui,
nmcli) then saves the answer as a duplicate "<SSID> 1" profile. The problem
predates the Noctalia v5 shell and is driven by the driver, not the shell.
- disable Wi-Fi powersave on shoukei, a common source of brcmfmac auth timeouts
- WORKAROUNDS.md WA-024: the powersave mitigation and its removal condition
- WORKAROUNDS.md WA-025: the pending upstream brcmfmac port_authorized fix
- document how to recover a wedged supplicant without rebooting, and that
deleting an active NM profile is what turned a reconnect into a dead link
Refs: https://patchew.org/linux/20260915-brcmfmac-4way-handshake-offload-authenticated-event-v2-1-fbb9cc7ac177@jannau.net/
Apple Silicon's 16K pages mean FEX cannot run natively and qemu-user cannot
run complex x86_64 binaries. Wire the maintained rowanG077/nix-x86-on-aarch64
runtime (FEX inside a 4K muvm microVM) into shoukei so games, Steam, Wine and
other x86/x86_64 programs work, and let the module own the x86 ELF handlers
instead of qemu-user.
The input is pinned to our fork (ryan4yin/nix-x86-on-aarch64) at the reviewed
commit. Dropping boot.binfmt.emulatedSystems also removes x86_64 from
nix.settings.extra-platforms, so the qemu-user/uncached-x86-build gap is
recorded as WA-022.
- flake.nix: add nix-x86-on-aarch64 input (WA-021)
- hosts/12kingdoms-shoukei/game/: programs.x86-on-arm + steam/wine, 6 GiB VM
- hardware-configuration.nix: drop emulatedSystems/preferStaticEmulators
- apple-silicon.nix: drop the now-redundant box64
- docs: WA-012 resolved; WA-021 and WA-022 added; host README updated
* feat(restic): verify the repository after each backup
* feat(youko): snapshot the restic store hourly
* docs(backup): document the snapshot and check layers
* fix(postgres): require a password for superuser over TCP
* test: assert the postgres loopback auth rules
* fix(darwin): keep root-only agenix secrets root-only
* docs(security): record the local auth boundaries and retire WA-015
* fix(nfs): restrict the golden export to its real clients
* fix(libvirt): bind VNC consoles to loopback
* fix(firewall): drop VNC ports on every host
* docs(security): record the NFS and VNC boundaries
KubernetesApiServerLatency fired continuously on the p99 of
apiserver_request_duration_seconds_bucket. Every firing series was
verb="GET", subresource="openapi/v3" (kubelet, kube-controller-manager,
kube-scheduler, apiserver) at ~8s, while real verbs (POST/LIST/DELETE)
stay around 0.2s.
openapi/v3 is a streaming discovery endpoint, like WATCH, so exclude
subresource=~"^openapi/" from the rule.
The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
The bring-up runs at the default 1.25 V; afterwards Gear mode is set to
Gear2 and VDD/VDDQ are raised to 1.28 V, changing nothing else. 1.29 V and
up do not train. Also record that the 1.25 V config passed stressapptest
but still hard-froze under a real llama.cpp load, so a passing test is not
proof of stability.
- replace the btop screenshot with the 2026-10-06 annotated capture
(btop + fastfetch + the strata/llama.cpp serving logs); the old
2026-10-01 file is gone, so the README link moves to the new name
- drop the Helix/AI-agents screenshot and its README reference
- fix stale hardware descriptions: the hosts/README.md inventory row and
the idols-ai header comment still said i5-13600KF, and the zswap pool
comment sized its percentages against the old 96G of RAM (now 128G)
- record the hardware history: the platform was rebuilt on 2026-04-27
(PR #257/#258) onto the Colorful CVN Z890 ARK FROZEN + Core Ultra 7
270K Plus, replacing the MSI board and i5-13600KF the host was added
with in 2023-05; the 2x48G kit predates that rebuild, the 2x16G kit
was added 2026-10-06
- make the DIMM bring-up order mandatory and record its failure mode:
with the 48G pair in first, or all four DIMMs from the start, the
board does not POST
- gloss the Chinese BIOS labels in English, marked as translations of
the labels rather than the firmware's own English wording
Left alone on purpose: the 96G swapfile in hosts/idols-ai/disko-fs.nix.
It was grown from 20G to 1x-RAM on 2026-10-02 (a26ab3be), when the host
still had 96G; resizing it again means re-running disko on the machine,
so that stays a separate decision.
Document the 4-DIMM kit mix (2×48G JUHOR + 2×16G GLOWY) on the Colorful
CVN Z890 ARK FROZEN V20:
- hardware (CPU, board, and both kits)
- the install order that trains (16G pair first)
- the only stable settings (XMP off, DDR5-4800 @ 1.25 V)
- the stress-test recipe and results
- the XMP @ 5600 failure and how to recover a board that will not POST
- re-enrolling the LUKS TPM unlock after a BIOS change