* fix(microvm): pin guest IPv6 identity to a cluster ULA
The ISP retires delegated prefixes, so a guest that registers its WAN prefix
loses it and Cilium never refreshes the node address. Derive the identity from
the IPv4 octets inside fd05:5::/64 and trust that ULA in the guest firewall.
* feat(mihomo): answer AAAA and keep the tailnet off the TUN
fake-ip needs one pool per family, and the family a client picks does not pick the
egress family. Exclude the cluster ULA and the Tailscale ranges from the TUN; the ssh
pin and the WA-028 record cover the fake-AAAA stall.
* feat(mihomo): generate the config from the repo, not a copy in the live dir
`just mihomo-gen` runs generate.nu from the module and validates with the core
the service runs. ~/.config/mihomo keeps only sources.yaml and config.yaml.
* fix(mihomo): keep the fake IPv6 pool out of fc00::/7
The v6 fake pool is a lookup key, not the egress path, so it can be a public range: the
shipped ULA default reads as "local" to browsers and prompts for local-network access on
every site. The gvisor stack override goes with it -- mips is upstream's default, and
nftables rules out the kernel stack here.
* feat(mihomo): sniff on the gateway, not on the desktop
LAN clients using DoH dial the IPs they resolved, and a node may not reach an IP a
foreign resolver handed out. The gateway recovers the name from the handshake; on the
desktop the fake-ip table already names every flow. Its binary is a static release
build, not a package: update it by hand and validate the live config with the new core.
* fix(mihomo): name the Singapore nodes instead of matching 新
The single-char keyword also matched 新西兰, so the Singapore group held a New
Zealand node and url-test flipped the exit country mid-session.
LAN clients using DoH dial the IPs they resolved, and a node may not reach an IP a
foreign resolver handed out. The gateway recovers the name from the handshake; on the
desktop the fake-ip table already names every flow. Its binary is a static release
build, not a package: update it by hand and validate the live config with the new core.
The v6 fake pool is a lookup key, not the egress path, so it can be a public range: the
shipped ULA default reads as "local" to browsers and prompts for local-network access on
every site. The gvisor stack override goes with it -- mips is upstream's default, and
nftables rules out the kernel stack here.
`just mihomo-gen` runs generate.nu from the module and validates with the core
the service runs. ~/.config/mihomo keeps only sources.yaml and config.yaml.
fake-ip needs one pool per family, and the family a client picks does not pick the
egress family. Exclude the cluster ULA and the Tailscale ranges from the TUN; the ssh
pin and the WA-028 record cover the fake-AAAA stall.
The ISP retires delegated prefixes, so a guest that registers its WAN prefix
loses it and Cilium never refreshes the node address. Derive the identity from
the IPv4 octets inside fd05:5::/64 and trust that ULA in the guest firewall.
* feat(mihomo): keep ranges that can never be proxied out of TUN
The list lives in policy.yaml as tun_exclude_address; sources.yaml may override
it. 198.18.0.0/15 stays in TUN: that is the fake-ip range itself.
* feat(mihomo): match Valve's content servers by ASN
The 20 hand-copied prefixes had drifted from what AS32590 announces. The rule
type needs GeoLite2-ASN.mmdb, so geox-url.asn points at the same mirror.
* feat(mihomo): render the gateway config from the same sources
The hand-kept copy had drifted: a manual sync carried the desktop's secret onto
the gateway and dropped external-ui-url. --redact yields a shareable template.
* fix(mihomo): open the link DNS takeover only once the core serves it
The core opens :1053 after parsing rules, and rule parsing is where missing
geodata gets fetched. Flipping at fork time made that fetch SERVFAIL, the config
fatal, and the unit unstartable.
* fix(justfile): keep local endpoints out of the agent proxy
With only HTTP_PROXY set, a request to this host's own llama-swap endpoint went
to mihomo. Node matches exact hosts only, so the literals carry it; the CIDRs
serve curl and Go.
* feat(justfile): seed mihomo geodata into the service state dir
So a cold start never needs the download that the DNS takeover made unstartable.
The comment trim dropped them. Without a domain rule above the Valve IP ranges,
steamcommunity.com and steampowered.com match those ranges and go DIRECT.
Steam picks download caches from the IP it sees on the CM login. Pin the CM hosts
and content CDN DIRECT, keep the community and store proxied, and add Valve's
AS32590 ranges because Steam dials cached CM IPs without a lookup. Sources in
README.md.
The abort needs havePrivateMountNs == false AND /nix/store still ro:
remountReadOnlyWritable() returns early while the store is writable. Nix
<= 2.34 left it rw until reboot; 2.35 remounts in a private namespace, so
only a freshly booted host hits it. The repro command also uses an open-only
exec 7<..., since cat fails on the read even when the open succeeds.
While here, trim the comments and drop the disabled default_deny/nix
policies plus the explicit killUnconfinedConfinables=false (nixpkgs
default). 82 -> 34 lines.
`just test` has enforced the suite result since #350: the recipe raises unless
`evalTests` is `true`, so a zero exit already means pass. Update the places that
still described the old raw-`nix eval` behaviour: AGENTS.md, README.md,
SECURITY.md, outputs/README.md, and the nix-config-debug / nix-config-new-host /
nix-config-update skills.
The exit-0 caveat is kept only where the raw `nix eval .#evalTests` command is
documented, because that command genuinely exits 0 even on `false`.
BCM4378/brcmfmac intermittently fails Wi-Fi association (ASSOC-REJECT /
authentication timeout) on this laptop. NetworkManager misreads the failure
as a bad key and re-prompts; whichever client reconnects (Noctalia, nmtui,
nmcli) then saves the answer as a duplicate "<SSID> 1" profile. The problem
predates the Noctalia v5 shell and is driven by the driver, not the shell.
- disable Wi-Fi powersave on shoukei, a common source of brcmfmac auth timeouts
- WORKAROUNDS.md WA-024: the powersave mitigation and its removal condition
- WORKAROUNDS.md WA-025: the pending upstream brcmfmac port_authorized fix
- document how to recover a wedged supplicant without rebooting, and that
deleting an active NM profile is what turned a reconnect into a dead link
Refs: https://patchew.org/linux/20260915-brcmfmac-4way-handshake-offload-authenticated-event-v2-1-fbb9cc7ac177@jannau.net/
Nix 2.35's root nix-env -- used by nixos-rebuild to set the system profile --
calls saveMountNamespace(), which opens /proc/self/ns/mnt before remounting
the read-only /nix/store writable. Those magic links have no connected path,
so AppArmor cannot match them against any rule and denies the open with
EACCES even in complain mode. The private mount namespace is then never
created and nixos-rebuild aborts with:
cannot remount "/nix/store" writable: not in a private mount namespace
Adding flags=(attach_disconnected) lets AppArmor re-root those paths on / so
the existing `file /**` rule matches. This also fixes any other sudo'd tool
that needs namespaces (unshare, nsenter, bwrap, ...).
Rolling this out to a host that already has the old profile loaded needs one
deploy with --no-reexec, so the current system's Nix performs nix-env --set
instead of the new system's; after that generation is active, plain
`just niri` works and the Nix version is never rolled back.
Gitee's SSH gateway intermittently accepts the connection and key, then never
answers the initial git-receive-pack and drops the session ~100s later with
"the remote end hung up upon initial contact". The same failure mode shows up
sporadically since 2026-10-02 and spiked on 2026-10-07 (9 of 14 runs failed).
Bound each push attempt with `timeout`, retry up to six times with exponential
backoff, and add ssh connect/keepalive timeouts. Serialize runs (a push to main
and a tag on the same commit used to race for Gitee's per-repo push lock), cap
the job at 30 minutes, and allow manual workflow_dispatch reruns.
* feat(shoukei): run x86/x86_64 apps and games via muvm+FEX
Apple Silicon's 16K pages mean FEX cannot run natively and qemu-user cannot
run complex x86_64 binaries. Wire the maintained rowanG077/nix-x86-on-aarch64
runtime (FEX inside a 4K muvm microVM) into shoukei so games, Steam, Wine and
other x86/x86_64 programs work, and let the module own the x86 ELF handlers
instead of qemu-user.
The input is pinned to our fork (ryan4yin/nix-x86-on-aarch64) at the reviewed
commit. Dropping boot.binfmt.emulatedSystems also removes x86_64 from
nix.settings.extra-platforms, so the qemu-user/uncached-x86-build gap is
recorded as WA-022.
- flake.nix: add nix-x86-on-aarch64 input (WA-021)
- hosts/12kingdoms-shoukei/game/: programs.x86-on-arm + steam/wine, 6 GiB VM
- hardware-configuration.nix: drop emulatedSystems/preferStaticEmulators
- apple-silicon.nix: drop the now-redundant box64
- docs: WA-012 resolved; WA-021 and WA-022 added; host README updated
* refactor(nixpkgs): drop the repo-wide pkgs-x64 package set
Every remaining pkgs-x64 consumer runs only on x86_64-linux hosts, where the
native pkgs already carries the same allowUnfree config
(modules/nixos/base/nix.nix) and overlays (modules/base/overlays.nix), so
replace the references with pkgs and drop the extra nixpkgs instantiation
(which also removes an eval on aarch64-darwin).
media.nix keeps the native Zoom app on x86_64 hosts only, since zoom-us has
no aarch64 Linux build.
Every remaining pkgs-x64 consumer runs only on x86_64-linux hosts, where the
native pkgs already carries the same allowUnfree config
(modules/nixos/base/nix.nix) and overlays (modules/base/overlays.nix), so
replace the references with pkgs and drop the extra nixpkgs instantiation
(which also removes an eval on aarch64-darwin).
media.nix keeps the native Zoom app on x86_64 hosts only, since zoom-us has
no aarch64 Linux build.
Apple Silicon's 16K pages mean FEX cannot run natively and qemu-user cannot
run complex x86_64 binaries. Wire the maintained rowanG077/nix-x86-on-aarch64
runtime (FEX inside a 4K muvm microVM) into shoukei so games, Steam, Wine and
other x86/x86_64 programs work, and let the module own the x86 ELF handlers
instead of qemu-user.
The input is pinned to our fork (ryan4yin/nix-x86-on-aarch64) at the reviewed
commit. Dropping boot.binfmt.emulatedSystems also removes x86_64 from
nix.settings.extra-platforms, so the qemu-user/uncached-x86-build gap is
recorded as WA-022.
- flake.nix: add nix-x86-on-aarch64 input (WA-021)
- hosts/12kingdoms-shoukei/game/: programs.x86-on-arm + steam/wine, 6 GiB VM
- hardware-configuration.nix: drop emulatedSystems/preferStaticEmulators
- apple-silicon.nix: drop the now-redundant box64
- docs: WA-012 resolved; WA-021 and WA-022 added; host README updated
* feat(restic): verify the repository after each backup
* feat(youko): snapshot the restic store hourly
* docs(backup): document the snapshot and check layers
* fix(postgres): require a password for superuser over TCP
* test: assert the postgres loopback auth rules
* fix(darwin): keep root-only agenix secrets root-only
* docs(security): record the local auth boundaries and retire WA-015
* fix(nfs): restrict the golden export to its real clients
* fix(libvirt): bind VNC consoles to loopback
* fix(firewall): drop VNC ports on every host
* docs(security): record the NFS and VNC boundaries
* fix(k3s): deny credential-bearing host ports to pod sources
* test: assert the k3s pod firewall drops sensitive ports
* docs(security): record the pod firewall boundary
KubernetesApiServerLatency fired continuously on the p99 of
apiserver_request_duration_seconds_bucket. Every firing series was
verb="GET", subresource="openapi/v3" (kubelet, kube-controller-manager,
kube-scheduler, apiserver) at ~8s, while real verbs (POST/LIST/DELETE)
stay around 0.2s.
openapi/v3 is a streaming discovery endpoint, like WATCH, so exclude
subresource=~"^openapi/" from the rule.
The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
The bring-up runs at the default 1.25 V; afterwards Gear mode is set to
Gear2 and VDD/VDDQ are raised to 1.28 V, changing nothing else. 1.29 V and
up do not train. Also record that the 1.25 V config passed stressapptest
but still hard-froze under a real llama.cpp load, so a passing test is not
proof of stability.
- replace the btop screenshot with the 2026-10-06 annotated capture
(btop + fastfetch + the strata/llama.cpp serving logs); the old
2026-10-01 file is gone, so the README link moves to the new name
- drop the Helix/AI-agents screenshot and its README reference
- fix stale hardware descriptions: the hosts/README.md inventory row and
the idols-ai header comment still said i5-13600KF, and the zswap pool
comment sized its percentages against the old 96G of RAM (now 128G)
- record the hardware history: the platform was rebuilt on 2026-04-27
(PR #257/#258) onto the Colorful CVN Z890 ARK FROZEN + Core Ultra 7
270K Plus, replacing the MSI board and i5-13600KF the host was added
with in 2023-05; the 2x48G kit predates that rebuild, the 2x16G kit
was added 2026-10-06
- make the DIMM bring-up order mandatory and record its failure mode:
with the 48G pair in first, or all four DIMMs from the start, the
board does not POST
- gloss the Chinese BIOS labels in English, marked as translations of
the labels rather than the firmware's own English wording
Left alone on purpose: the 96G swapfile in hosts/idols-ai/disko-fs.nix.
It was grown from 20G to 1x-RAM on 2026-10-02 (a26ab3be), when the host
still had 96G; resizing it again means re-running disko on the machine,
so that stays a separate decision.
Document the 4-DIMM kit mix (2×48G JUHOR + 2×16G GLOWY) on the Colorful
CVN Z890 ARK FROZEN V20:
- hardware (CPU, board, and both kits)
- the install order that trains (16G pair first)
- the only stable settings (XMP off, DDR5-4800 @ 1.25 V)
- the stress-test recipe and results
- the XMP @ 5600 failure and how to recover a board that will not POST
- re-enrolling the LUKS TPM unlock after a BIOS change
ryan4yin | 二花
Ryan Yin