The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
Document the 4-DIMM kit mix (2×48G JUHOR + 2×16G GLOWY) on the Colorful
CVN Z890 ARK FROZEN V20:
- hardware (CPU, board, and both kits)
- the install order that trains (16G pair first)
- the only stable settings (XMP off, DDR5-4800 @ 1.25 V)
- the stress-test recipe and results
- the XMP @ 5600 failure and how to recover a board that will not POST
- re-enrolling the LUKS TPM unlock after a BIOS change
`dsh-llm-pi-ai` registers its built-in providers once, so a second instance
in the profile patch fails on mount with `provider "..." is already declared`.
Extend the base `llm-pi-ai` row instead.
dsh rewrites a profile's cordis.patch.yml at runtime, so linking the web
profile put GUI state in the checkout and let runtime writes clobber manual
edits. Link the stable policy as the $DSH_HOME home layer instead: dsh applies
it after every profile and only reads it.
The host-plane compaction-basic row is disabled on the web surface and each
agent preset owns its own copy, so the top-level override was inert. Drop it;
the model contextWindow/maxTokens are the effective knobs.
- add home/base/tui/agents/dsh/cordis.patch.yml, linked from default.nix
- drop the ~/.dsh/profiles/web link and the tracked web/ profile
- keep runtime state and private/local model routes in the profile patch
- update the eval test and README
* feat(rime): share the Rime config and user dictionary across Linux and macOS
Move the Rime data, preferences, and user dictionary into home/base/gui/rime so fcitx5-rime and Squirrel read the same files.
* feat(vinput): open the command palette with right Ctrl
Pin keyd's right-hand modifiers (rightcontrol/rightshift/rightmeta) to their own keycodes; keyd otherwise folds them into the left keys. Set Vinput's MenuKey to right Ctrl and clear CommandKeys, which would shadow it.
* feat(fcitx5): switch CN/EN with Ctrl+Space and right Shift
Clear AltTriggerKeys, whose Shift_L default toggled on left Shift. Left Shift is now a plain modifier and right Shift switches through the shared Rime config.
* fix(desktop): align Qt app fonts with GTK
Qt apps kept their built-in default font ("Sans Serif" 9) while GTK apps use
`gtk-font-name` (Noto Sans 11), so Telegram and other Qt apps rendered
noticeably smaller than GTK apps like Firefox. Set
QT_QPA_PLATFORMTHEME=gtk3; the plugin ships with qtbase, so it needs no
extra package and also gives Qt apps GTK file dialogs.
* chore(desktop): drop unused Xft.dpi from the Wayland GTK profile
Xft.dpi/ *.dpi are X11 X-resource settings. This file is only imported by
the niri (Wayland) hosts, where xwayland-satellite sets its own Xft.dpi
(96 x output scale, currently 144), and the i3 hosts are headless 1080p
computer-use sessions that do not import it. So the setting had no effect
anywhere; remove it instead of keeping a misleading knob.
Vinput's streaming Zipformer models have unbounded dynamic shapes, so
OpenVINO cannot compile them for the GPU or the NPU; the `openvino`
provider and ai's Intel NPU enablement could never run on the NPU (noted
as unverified in WORKAROUNDS WA-017). Remove the backend option, the
daemon's OpenVINO library path, and the intel-npu module enable, and
update the eval test, README and WORKAROUNDS (drop WA-016 and WA-017).
Keep Vinput's stock package and its bundled sherpa-onnx (1.13.4): the
fork-based and nixpkgs-master sherpa overrides existed only to provide the
`openvino` provider, so drop package.nix with them.
Assisted-by: opencode (DeepSeek V4.1 Flash)
metacubexd hardcodes its tab title in the SPA, so dashboards on
different hosts are indistinguishable. Patch the static <title> and
re-apply the hostname from config.js, since Nuxt's useHead resets
document.title on every route change.
The config deliberately stays out of the store, under the home directory, which is a persistent bind mount on tmpfs-root hosts. When the service and the mount are introduced in the same activation, mihomo can start before the mount is up and fail with "Failed to set up credentials: No such file or directory", making the switch report status 4. Add RequiresMountsFor so the mount is brought up and ordered before the service.
Vinput stores its core config in ~/.config/vinput/config.json and its
downloaded sherpa-onnx models in ~/.local/share/vinput/models/. Neither
was bind-mounted from /persistent on the tmpfs-root hosts, so a reboot
wiped both and vinput-daemon started with "Local ASR provider model is
not configured". Persist both directories and document the one-shot CLI
model setup in the vinput README.
* fix(desktop): tie ai's link DNS takeover to mihomo's lifecycle
sing-tun normally points systemd-resolved at the TUN DNS by running
resolvectl (dns/domain/default-route). Under nixpkgs' hardened
DynamicUser mihomo unit that hand-off fails silently: the unit's
RestrictAddressFamilies blocks resolvectl's D-Bus socket and resolve1
requires privileged authorization, so resolved keeps querying the
physical-link DNS, fake-ip is bypassed, and domain rules degrade to
MATCH.
Do the hand-off as '+' ExecStartPost/ExecStopPost hooks instead: '+'
commands run outside the sandbox (systemd.exec: address-family
restrictions do not apply to them) and ExecStopPost reverts even on
crashes, so a dead mihomo leaves DNS usable.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
* docs(mihomo): record sing-tun's resolvectl hand-off as the leak cause
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
* refactor(desktop): keep mihomo only on ai and shoukei
ruby/kana never ran the core (service inactive live), and their
resolved + TUN combo would reintroduce the resolver leak once enabled.
Drop the enable and the explicit module import from computer-use; the
eval test now pins mihomo=false for both guests.
---------
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
* refactor: consume computer-use packages from NUR
* fix: expose flake inputs to package overlays
Package overlays are imported from two places with different argument
shapes: outputs/default.nix passes a set containing `inputs`, while
modules/base/overlays.nix passes the NixOS module args, where genSpecialArgs
spreads each flake input as a top-level argument. The new NUR-backed
overlays read `inputs.nur-ryan4yin`, so evaluation failed on the module
path. Also expose the whole flake input set as `inputs` in genSpecialArgs.
* chore: bump nur-ryan4yin input
Update the locked nur-packages revision to the current main (nixpkgs
26.05, unused packages removed).
node_exporter listened on 0.0.0.0, so access control rested entirely on
the nftables allowlist. Bind node/smartctl exporters to the host static
LAN IPv4 (the same address VictoriaMetrics scrapes), keeping the socket
off tailscale/bridge interfaces even if the firewall regresses.
Hosts without a static LAN address (shoukei, akane) now run no exporters
at all instead of loopback-bound ones nobody scrapes; akane, a UTM VM,
previously ran a loopback smartctl exporter with no physical disks.
LAN-bound exporters also get the boot-race treatment from #372
(network-online ordering plus patient restart); loopback-bound hosts
must not pull in network-online because NM wait-online would stall
their offline boots.
The exporter binds to the host LAN address, which systemd-networkd
assigns during boot. Without ordering, the service starts before the
address exists, `bind: cannot assign requested address` trips the
start limit, and the unit stays failed until restarted manually.
Order the unit after network-online.target, and since wait-online is
best-effort (120s timeout), also retry patiently so the unit self-heals
when the link comes up late.
Add SECURITY.md covering the threat model, trust boundaries,
proportionate kernel/process hardening, the update/reboot routine and
remaining prioritized work. Record the dated read-only audit findings
(listeners, privileged workloads, effective RBAC, secret metadata and
sandbox/SSH boundaries) separately from planned controls and runtime
verification.
Define agent-first diagnosis: scoped root read-only access on homelab
hosts, and normal-user diagnostics on core desktops with no new sudo
grants. Add canonical domain-based k3s API endpoint regression tests,
record the unverified Darwin secret-ownership concern as WA-015, and
correct comments claiming mode 0000 blocks privileged root.
test-instrumentation.nix sets serial-getty@ttyS0.enable = false at normal priority to keep ttyS0 free for the test backdoor console, while qemu-guest.nix set it to true at normal priority. Two conflicting normal-priority definitions made `nix flake check` fail while evaluating packages.x86_64-linux.ruby-nixos-tests.
Use lib.mkDefault so the test framework wins in tests while ordinary QEMU guests still enable the serial getty.
* fix(colmena): expose colmenaHive for the new evaluator
nixpkgs now ships the nix-community Colmena rewrite, whose default
evaluator reads an evaluated colmenaHive flake output instead of the raw
colmena hive. Without it every colmena apply recipe failed with
"does not provide attribute colmenaHive".
Add a colmena input pinned to the same v0.5.0 as nixpkgs and expose
colmenaHive = colmena.lib.makeHive <raw hive>, with the raw hive kept as
an internal binding rather than a flake output of its own.
* chore: clean up stale colmena and aquamarine references
- point the Colmena links at nix-community/colmena after the move
- rename the security-aquamarine-metrics test to youko-metrics (it
already tested youko) and update the outputs README
- drop the commented aquamarine remote-builder example
* chore(workarounds): re-evaluate stale hardware/kernel fixes
Prune workarounds whose root cause is gone, fix one vendor typo, and
document the ones we keep:
- drop `transparent_hugepage=madvise` on shoryu/shushou/youko: it was tied to
the old fixed 1G hugetlb pool, and the VMs now use ordinary memory
- shoryu: `hardware.cpu.intel.updateMicrocode` -> `amd` (it is an AMD APU)
- ai: drop `i915.enable_guc=3` (GuC/HuC submission is the default on modern
i915/Xe)
- kernel-hardening: name the Dirty Frag CVEs (CVE-2026-43284 / CVE-2026-43500)
and the re-evaluation condition
- document re-eval conditions for the Insta360 resume fix, the darwin
auto-optimise-store workaround, and the youko USB-SATA bridge mitigations
* chore(workarounds): keep the Dirty Frag module blacklist as defence in depth
* chore(ai): correct the i915.enable_guc rationale (verified against the kernel)
* chore(darwin): drop the auto-optimise-store workaround (nix#7273 fixed)
* fix(ai): relax NVMe power-management workarounds and enable smartd
Logs show no NVMe errors, so allow APST again (capped at 100 ms), restore
the default io_timeout, and drop pcie_aspm=off. Add smartd for drive
health visibility.
* feat(storage): add drive/SSD SMART monitoring on physical hosts
node_exporter's nvme collector only exposes node_nvme_info and namespace
capacity, not SMART health, so SSD wear/lifespan and drive temperature were
not observable anywhere.
- enable services.prometheus.exporters.smartctl on physical hosts, gated on
modules.hardwareTools (off on QEMU/MicroVM guests), bound to the host's LAN
address on port 9633
- firewall: drop 9633 from anything but the monitoring host
- youko: scrape smartctl-<host> for the physical hosts
- alert rules (storage.yml): SMART health, NVMe wear, available spare, media
errors, critical warning, and drive temperature (warn 70 / crit 80 C)
- Grafana dashboard "Drive SMART (smartctl)"
agenix was pinned to a May 2025 revision, so `just up` could never move
it. Move to the 0.18.0 release tag and collapse its duplicate
nix-darwin/home-manager lock nodes via follows.
Remove a dead .gitattributes rule, bump the mirror workflow's checkout,
document the daemon test settings, drop unused JACK, and align the
installer's disko version.
* fix(ci): fail the eval-test run when the suite does not pass
`nix eval .#evalTests` prints the suite result but exits 0 either way, so
both `just test` and the CI workflow reported success even when a test
failed. Compare the raw boolean and fail instead.
* fix(ci): use --json for the boolean eval result
`nix eval --raw` cannot coerce a Boolean to a string, and Just runs each
recipe line as a separate `nu -c`, so the result variable must stay on one
line.
* feat!: replace zellij with tuios and switch to ghostty + kitty
Terminal environment overhaul:
- add the tuios flake input (upstream 0.8.5; nixpkgs has 0.8.1) and link
the checked-in config out-of-store; auto-start tuios in nushell instead
of zellij
- remove zellij entirely: both modules, config, README and nushell
completions
- remove foot and alacritty, which lack the Kitty graphics protocol;
* feat(idols-ai): persist /var/tmp and small state dirs, wipe /var/tmp at boot
Move the strata model staging area (~3G) off the root tmpfs onto the
persistent btrfs volume to free RAM, preserve btrbk/cups/logrotate
state, and wipe /var/tmp at boot so it only accumulates between
reboots.
* chore(idols-ai): add nosuid,nodev to root tmpfs and tmp btrfs mounts
Hardens /, /tmp and /snapshots; safe because /dev, /run, /nix and
/run/setuid-programs are separate mounts, so nothing there needs suid
bits or device nodes.
* chore(idols-ai): add noatime to the @tmp btrfs mount
Temp files do not need access times; matches the @nix subvolume style
and saves btrfs metadata writes.
* chore(idols-ai): harden the /var/tmp bind mount with nosuid,nodev
World-writable staging data should not honor suid bits or device
nodes, same as /tmp and /snapshots.
* fix(idols-ai): preserve logrotate state as a file, fix persisted dir ownership
- /var/lib/logrotate.status is a regular file, not a directory: move it from
preservation directories to files. A missing state entry makes logrotate
treat every log as freshly rotated, so it must survive reboots.
- keep /var/lib/btrbk owned by btrbk:btrbk 0750 so preservation does not
override the btrbk module's rule with root:root.
- set /var/tmp mode 1777 explicitly instead of relying on systemd's tmp.conf
ordering.
- rewrite comments to describe the paths generically.
* chore(idols-ai): cap journal, wipe coredumps at boot, persist upower/noctalia
- services.journald.settings.Journal: SystemMaxUse=10G, RuntimeMaxUse=256M,
matching the cap already set on idols-kana/idols-ruby.
- wipe /var/lib/systemd/coredump at boot: it lives in the persisted
/var/lib/systemd and crash dumps can contain secrets copied from memory.
- persist /var/lib/upower (device battery history) and
~/.local/state/noctalia (shell settings, notification/clipboard history).
* feat(secrets): add work gitconfig secret, symlink it in home-manager
* feat(secrets): add a no-sudo replace recipe for new secrets
* refactor(secrets): move work gitconfig symlink into the desktop layer
* docs(skills): point nix-config-secrets at the private repo under ~/codes
Record where the private nix-secrets checkout lives and that agents should
do the mechanical secret work themselves, handing back only the
interactive/sudo agenix operations and pushes.
* docs(agents): add related repositories to nix-config
* chore(agents): adopt ~/src for source checkouts
Move third-party/source checkouts from ~/codes/src to ~/src so the home
layout reads as three siblings: ~/codes (personal), ~/work (work),
~/src (checkouts). Preserve and back up the new path, and update the
global rule and the two nixpkgs skill references.
* docs(agents): tighten the code layout rule
* docs(agents): fold the work-confidentiality note into the layout line
* docs(agents): base the confidentiality rule on repo visibility
* docs(agents): drop the stale KubeVirt mention
nixvim has been disabled for a long time (programs.nixvim.enable = false), so
drop the flake input (and prune flake.lock). Port the settings it carried to
Home Manager's programs.neovim: opts and per-plugin setup become an initLua
block plus a plugin list. The catppuccin module already provides the
colorscheme, vi stays unmapped, and the plain system neovim package is
untouched so sudo/nvim --clean keep using it.
* fix(youko): add ROCm OpenCL so Jellyfin can tone-map HDR
Jellyfin's HDR/DoVi transcodes use ffmpeg's `tonemap_opencl`, but only
Mesa was installed via `hardware.graphics.enable`, so no OpenCL ICD was
registered. ffmpeg then failed to create an OpenCL device ("Failed to
get number of OpenCL platforms: -1001") and aborted with exit code 237,
leaving HDR titles unplayable.
Add `rocmPackages.clr.icd` to `hardware.graphics.extraPackages`; it
ships the ICD for the Barceló iGPU (reported as `gfx90c`) under
/run/opengl-driver/etc/OpenCL/vendors, where `ocl-icd` (built with
`--enable-custom-vendordir`) finds it.
The Barceló iGPU only manages ~0.8x realtime for a 4K HDR -> 1080p
tone-mapped transcode, so playback can still buffer; this only makes
such titles playable at all.
* fix(youko): allow /dev/kfd so Jellyfin OpenCL can reach the GPU
The jellyfin module's `DeviceAllow` whitelist only permits the VA-API
render node (`/dev/dri/renderD128`). ROCm's OpenCL runtime opens
`/dev/kfd`, so even with the ICD installed OpenCL device creation failed
("No matching devices found") and HDR transcodes still aborted with exit
code 237.
Add `/dev/kfd rw` to the whitelist; list definitions merge with the
module's entry. Verified under the unit's sandbox (PrivateUsers,
PrivateTmp) that OpenCL init then succeeds as the jellyfin user.
* fix(youko): stop homepage from probing the authenticated transmission RPC
The siteMonitor hit /transmission/rpc unauthenticated, so every probe returned 401 and counted against transmission's anti-brute-force threshold.
* feat(youko): tune transmission and firewall its network namespace
Raise the upload limit and queues, relax required encryption to preferred, and add a default-drop nftables input chain inside the netns so the peer port stays open while everything else is closed. Allowlist the nftables 'iif' keyword for typos.
* feat(youko): export transmission metrics and add a Grafana dashboard
nixpkgs has no transmission exporter, so add a small stdlib Python exporter that queries the authenticated netns RPC on loopback, scrape it from VictoriaMetrics, and ship a Transmission dashboard.
* fix(youko): cap transmission upload at 2500 kB/s
30 MB/s ignored the 50 Mbps uplink; keep seeding around 20 Mbps.
* Revert "feat(youko): export transmission metrics and add a Grafana dashboard"
This reverts commit 9df7a26526.
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.
- base: nftables backend; deny inbound by default; trust the home LAN by
source and the tailnet by interface (restrict the tailnet with ACLs);
suppress per-service `openFirewall` global ports; allow node_exporter only
from the monitoring host. The local podman bridge, the tailnet and the pod
CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
(caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
redundant desktop firewall module; aarch64/riscv64 import the firewall
explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
the LAN CIDR rule and no global ports.
* feat(youko): add Jellyfin media server
Jellyfin reads the existing Transmission download tree on the public HDD
(via the shared `fileshare` group) and transcodes with the AMD Barceló
iGPU through VA-API. Encoding settings are declared in Nix
(`forceEncodingConfig`), so the web UI cannot silently drift.
Replace the per-service public ACME certificates with a single
`*.writefor.fun` wildcard (plus the apex) shared by every public vhost,
so adding a service needs no new ACME entry; a wildcard also keeps
individual hostnames out of public CT logs.
Pin the jellyfin uid/gid to avoid the dynamic-id drift fixed for the
other services in #320.
* fix(youko): enable graphics for Jellyfin VA-API
VA-API needs Mesa's radeonsi VA driver under /run/opengl-driver, but
youko is headless so hardware.graphics was off. Without it Jellyfin's
ffmpeg fails with 'va_openDriver() returns -1'. Enable it so the iGPU
can be used for hardware transcoding.
* fix(youko): use VA-API hardware encoding for Jellyfin
Enabling hardware decoding alone left encoding on the CPU (libx264), which
pins ~3 cores per 720p stream. Set transcoding.enableHardwareEncoding so
the iGPU (Vega VCN) does the H.264 encode too.
* feat(youko): harden Jellyfin proxy trust, groups, and HEVC encode
Add jellyfin to the render/video groups for robust VA-API access (was
relying on the world-writable render node), enable HEVC hardware encoding
(Vega's VCN supports it; AV1 does not), and patch network.xml on start to
trust Caddy as KnownProxies and advertise the public URL.
* fix(youko): run Jellyfin network.xml patch every start; allow LAN discovery
The module's preStart exits early once encoding.xml matches, so the
mkAfter network.xml patch only ran when encoding settings changed. Run it
with mkBefore instead. Also allow Jellyfin app auto-discovery (UDP 7359 /
TCP 8096) from the home network only.
Service users/groups get a dynamically allocated uid/gid, which shifts when the user set changes: adding `immich` moved gitea 993:985 -> 996:993 and rustfs 987:977 -> 988:983. Files on the persistent volumes keep the old numeric owner, so gitea and rustfs lost access to their data and failed to start after a restart. Pin every dynamically allocated service id on youko to its current value. The ids are host-specific, so this is a host file.
timerConfig was set with lowercase onCalendar/randomizedDelaySec, which systemd treats as unknown directives. The generated timer had no schedule, so `systemctl list-timers` showed NEXT=- and no backup ever ran. Use the real keys OnCalendar/RandomizedDelaySec.
Add Immich via the native nixpkgs services.immich module on youko, reusing the host PostgreSQL 16 (pgvector + vectorchord) and a shared Valkey instance for Redis-compatible caching and BullMQ queues (Dragonfly rejected BullMQ's dynamic Lua keys). Media lives on the encrypted HDD at /data/apps/immich; Caddy publishes immich.writefor.fun with a publicly-trusted Let's Encrypt cert (security.acme + Cloudflare DNS-01) so the mobile app connects. OTEL metrics are scraped by VictoriaMetrics, and /var/cache is now preserved on the tmpfs-root hosts so the ML model cache survives reboots.
The s3-console Caddy vhost redirected every request to / to /rustfs/console/. The RustFS console frontend signs its own S3/STS requests to / (POST / for STS, GET /?x-id=ListBuckets), so the 302 broke the console session and login failed server-side with SignatureDoesNotMatch ("missing signed header: content-type"). Restrict the redirect to unsigned browser GET /; signed requests now reach RustFS unchanged.
- run two autocutsel instances so the X PRIMARY/CLIPBOARD selections stay in
sync, making copy/paste work in the session and through VNC
- drop --disable-blink-features=AutomationControlled: nothing drives the
browser over CDP, so navigator.webdriver is already false and the flag only
marked the setup as automated
- document why the driver stays on real X11/AT-SPI input and is never granted
a CDP attach, and note the SSH-tunnel path for manual VNC access
- extend the computer-use eval test
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
The script pointed the macOS nix-daemon at the homelab v2ray HTTP proxy (192.168.5.101:7890), but that proxy was never running. Remove it together with the just darwin-set-proxy recipe and the README step that referenced it.
Reserve 192.168.5.192/27 as a dedicated kube-vip block, outside the router's DHCP pool (.2-.99): one /29 per cluster, k3s-test-1 gets .192/29 with the API VIP at .193.
Advertise only youko itself (.183/32) and that block over Tailscale, instead of the whole /24. The /24 route shadowed hosts that are already reachable directly, e.g. idols-ai at 192.168.5.100.
Deploy prerequisites (not in this repo): approve the new Tailscale routes; point kube-vip at the new VIP; move the *.writefor.fun wildcard from .101 to .183 and update scripts/darwin_set_proxy.py, since .101 is no longer advertised.
Only three secrets are readable as the login user (nix-access-tokens,
ssh-key-romantic and the nushell one); everything else is root-only and
the filesystem already keeps agents out.
* feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
* docs(backup): say 'cloud copy' rather than 'offsite'
Clearer for a reader who does not live in backup jargon; the requirement
(not in the homelab, immutable) is spelled out in the text.
* docs(secrets): state the recipient invariant
A recipient set must answer two questions: who may read it, and how it is
recovered once those hosts are gone. Dropping recovery_key while narrowing
a secret is easy and silent.
* docs(secrets): state that every secret must be decryptable by the desktops
The desktops hold the trusted admin keys and are where secrets are edited and
rekeyed from; narrowing a set to the servers alone breaks agenix -r and drops
the offline recovery key.
* docs(secrets): desktops and recovery_key have identical access
recovery_key is a member of desktop_keys, so every group written as
desktop_keys ++ ... covers both.
* feat(monitoring): scrape the restic rest-server
Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.
* docs(backup): record what the metrics actually expose
rest-server only ships the go/promhttp defaults, so repository growth is
charted from node-exporter's filesystem metrics instead.
* feat(hosts): per-host restic backups to the SMB remote
Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:
- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
(rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
excludes for the regenerable/bulky paths (podman overlay layers, microvm
and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
and dumps all postgres databases first (a consistent SQL artifact beside
the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token
Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.
* refactor(hosts): only restic does off-host; btrbk stays local
btrbk works at the subvolume level, so a would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.
So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.
Update the btrbk module's comment to say so.
* fix(restic): never back up keys; drop the implicit /etc/agenix
restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).
* refactor(restic): make the backup a proper module
Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.
* refactor(restic): exclude the whole podman storage, trim retention
Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.
Retention: daily 3, weekly 2, monthly 2 - no yearly.
* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump
A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.
* feat(secrets): restic-password for modules.restic-backup
With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.
* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix
* fix(restic): take the password path from agenix, not a hardcoded one
config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
* refactor(youko): let preservation own the service state dirs
The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:
- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind
Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.
* refactor(youko): drop caddy's redundant tmpfiles rules
The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.
* refactor(youko): use a podman named volume for uptime-kuma
podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
The VMs now declare their root disk as a DataVolume with `source: upload`, so
upload straight into `<vm>-disk` (CDI only accepts an upload into a DataVolume)
instead of a shared golden PVC. Takes the disk size as an argument.
* feat(k3s-test-1): add three worker nodes and taint the control plane
- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too
* chore(k3s-test-1): move the workers to .111-.113
The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.
- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
* feat(kubevirt-youko): export a shared golden-image store over NFS
* feat(vm-images): upload golden images into the cluster
Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
services.earlyoom.freeMemThresholdPercent does not exist in this nixpkgs; the module exposes freeMemThreshold/freeSwapThreshold. Just enable earlyoom with its defaults (trigger at 10% free mem/swap).
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
ryan4yin | 二花