Commit Graph
570 Commits
Author SHA1 Message Date
Ryan Yin 00aab5f562 feat(computer-use): add a headless X11/i3 environment
Give the idols-ruby and idols-kana VMs a headless computer-use
environment: Xvfb plus i3, driven by cua-driver (and computer-use-linux
as a fallback). Includes the system module (AT-SPI, fonts, Mesa, Clash
Verge proxy, proxy-region timezone), the home i3 module (Xvfb/i3/x11vnc
services, browser flags), host wiring and an eval test.
2026-09-21 21:31:27 +08:00
Ryan Yin 842cd26f64 feat(monitoring): scrape the restic rest-server
Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.
2026-09-21 20:56:01 +08:00
Ryan Yin 096607a5c5 feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
2026-09-21 20:42:39 +08:00
Ryan Yin c3f0dbf8bd revert(backup): drop append-only for the desktop endpoint
A desktop holds both the plaintext and the repository password, so blocking
deletion on the server protects little while preventing the automatic
retention the client needs. Immutability belongs in the offsite copy.
2026-09-21 20:38:08 +08:00
Ryan Yin 5a017666f6 refactor(backup): name the REST endpoint restic.writefor.fun
'backup' did not say what the service is.
2026-09-21 20:38:08 +08:00
Ryan Yin d322f4d2b6 docs: add BACKUP.md, and stagger the restic schedule per host
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
2026-09-21 20:38:08 +08:00
Ryan Yin 2ff93c16ce feat(restic): move idols-ai's off-host backup to the REST server
Split the repository password into desktop and homelab groups, add the
environmentFile/pruneOpts module options, and point the desktop at youko's
REST endpoint.
2026-09-21 20:38:08 +08:00
Ryan Yin 1e2cef2550 feat(youko): serve restic backups over an append-only REST server
Run restic-rest-server behind caddy (append-only, private repos, htpasswd
from agenix) and register youko's host key centrally.
2026-09-21 20:38:08 +08:00
Ryan Yin 12ac5f45aa feat(idols-ai): restic backup to a local repository
Allowlist of the irreplaceable directories (Documents, work, Pictures,
nix-config, .local/state); project checkouts under ~/codes live in git and
are not backed up. Kept the excludes generic, since this repo is public.

Also move the restic-password secret to a desktop.enable || server.storage
gate so desktops get it without enabling a server group.
2026-09-21 19:04:58 +08:00
ryan4yin | 二花 e37042f61c feat(youko): restic backup of the service state to the HDD (part 1) (#299)
* feat(hosts): per-host restic backups to the SMB remote

Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:

- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
  (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
  excludes for the regenerable/bulky paths (podman overlay layers, microvm
  and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
  and dumps all postgres databases first (a consistent SQL artifact beside
  the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token

Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.

* refactor(hosts): only restic does off-host; btrbk stays local

btrbk works at the subvolume level, so a  would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.

So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.

Update the btrbk module's comment to say so.

* fix(restic): never back up keys; drop the implicit /etc/agenix

restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).

* refactor(restic): make the backup a proper module

Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.

* refactor(restic): exclude the whole podman storage, trim retention

Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.

Retention: daily 3, weekly 2, monthly 2 - no yearly.

* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump

A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.

* feat(secrets): restic-password for modules.restic-backup

With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.

* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix

* fix(restic): take the password path from agenix, not a hardcoded one

config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
2026-09-21 19:00:35 +08:00
ryan4yin | 二花 bb33cca83a refactor(youko): let preservation own the service state dirs (#298)
* refactor(youko): let preservation own the service state dirs

The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:

- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
  postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind

Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.

* refactor(youko): drop caddy's redundant tmpfiles rules

The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.

* refactor(youko): use a podman named volume for uptime-kuma

podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
2026-09-21 18:06:40 +08:00
Ryan Yin a755b6fa11 feat(youko): move the service data from the HDD to the SSD
The services now run directly on the host, so their state does not belong
on the HDD behind the flaky USB-SATA bridge: /data/apps was small-file
heavy and latency/consistency sensitive. Point every dataDir at
/persistent/apps (the LUKS nvme) instead.

The HDD keeps the bulk data that suits it: /data/fileshare (173G) and
/data/backups (24G). The HDD's old @apps subvolume and its data are left
in place for now.
2026-09-21 17:56:35 +08:00
ryan4yin | 二花 12966d326c Merge pull request #296 from ryan4yin/fix/usb-hdd-mitigations
fix(youko): mitigate the flaky USB-SATA HDD bridge
2026-09-21 17:56:08 +08:00
Ryan Yin 5bbe17ebcd fix(youko): also try disabling the WD idle3 (Intellipark) timer
-B 255 turns APM off and it stuck, but the disks still reach standby, so
the bridge firmware is likely doing the spindown. -J 0 additionally asks
the drive to stop parking its heads (sdb has ~1M load cycles).
2026-09-21 17:43:05 +08:00
Ryan Yin 899162055a fix(youko): drop pcie_aspm=off, re-assert the hdparm settings hourly
pcie_aspm=off was the least-justified mitigation (the flakiness is on the
USB/SATA side) and costs power. Also log the disks' APM level/state and
re-apply the settings hourly, since the bridge or the disk firmware can
drop them (the disks were still reaching standby).
2026-09-21 17:42:21 +08:00
Ryan Yin 45a35a080d fix(youko): address the HDDs by-id, not /dev/sdX
Follows the disko config (ata-WDC_...-<serial>); the sdX names are not
stable.
2026-09-21 17:38:54 +08:00
Ryan Yin ce1645619f fix(youko): mitigate the flaky USB-SATA HDD bridge
The two 4TB HDDs sit behind a JMicron JMS567 that resets and throws
UDMA-CRC / I/O errors (the disks themselves are healthy). Best-effort
mitigations while the enclosure stays in use:

- kernel: usbcore.autosuspend=-1, usb-storage.delay_use=10, pcie_aspm=off
- udev: keep the bridge's power/control=on (never autosuspend)
- disable APM/standby on the HDDs (spindown is a common reset trigger)
- enable smartd (watch the CRC count and disk health)

The kernel parameters need a reboot to take effect.
2026-09-21 17:37:52 +08:00
ryan4yin | 二花 9ef2fefc25 Merge pull request #295 from ryan4yin/fix/homepage-allowed-hosts
fix(homepage): allow the home.writefor.fun host
2026-09-21 17:37:35 +08:00
Ryan Yin e939c54d26 chore: refresh the remaining KubeVirt references
Update the user-facing bits that still pointed at KubeVirt: the homepage
dashboard group and entries (and drop the dead LongHorn one), the docs,
and remove the now-unneeded virtctl package.
2026-09-21 17:25:38 +08:00
Ryan Yin 6180072646 chore: drop the KubeVirt guest bits
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:

- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
  latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
  upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
  restic backup path must stay as it is)
2026-09-21 17:24:40 +08:00
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
Ryan Yin b9c5d3e5c9 docs: describe the k3s-test microVMs
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
2026-09-21 16:37:53 +08:00
Ryan Yin e428d66abf feat(microvm): make the guest volume sizes configurable
etcSize/varSize/homeSize (MiB) with the old defaults. The masters get a
20G /var: k3s server + containerd + logs already filled the old 4G.
2026-09-21 16:37:53 +08:00
Ryan Yin 4ad359696e feat(microvm): run the k3s-test-1 masters as microVMs
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.

- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
  and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
  nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
2026-09-21 16:32:23 +08:00
Ryan Yin bc7d62b063 refactor(kubevirt-youko): resolve cert paths with relativeToRoot
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
2026-09-21 15:50:15 +08:00
Ryan Yin f9c902691b refactor: name the youko service modules by purpose
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
2026-09-21 15:46:55 +08:00
Ryan Yin c292509351 refactor: retire the aquamarine VM host
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:

- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
  drop the explicit import list, leaving aquamarine-services.nix with
  just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload

The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
2026-09-21 15:45:02 +08:00
Ryan Yin dfc3c2884a fix(aquamarine): order transmission after its data mount 2026-09-21 15:39:52 +08:00
Ryan Yin abd28347ab fix(aquamarine): qualify the uptime-kuma image for podman 2026-09-21 15:39:52 +08:00
Ryan Yin 37ebd7a5f8 feat(kubevirt-youko): run aquamarine's services on the host 2026-09-21 15:39:52 +08:00
ryan4yin | 二花 787b51df25 feat(vms): migrate the idol desktops from KubeVirt to libvirt (#287)
* wip(kubevirt-shoryu): enable libvirt/qemu

* feat(kubevirt-shoryu): enable libvirt/qemu with br0 + no NAT network

* chore: lock the microvm flake input

* feat(kubevirt-shoryu): define the libvirt domains declaratively

* fix(kubevirt-shoryu): pin the kana domain UUID

* feat(kubevirt-shoryu): migrate idols-ruby to libvirt
2026-09-21 15:12:37 +08:00
ryan4yin | 二花 3753396a01 docs(k8s): note the flaky USB bridge and the aquamarine microVM plan (#286) 2026-09-21 13:46:39 +08:00
ryan4yin | 二花 d40ed42f6c feat(k3s-test-1): add three worker nodes and taint the control plane (#283)
* feat(k3s-test-1): add three worker nodes and taint the control plane

- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
  never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too

* chore(k3s-test-1): move the workers to .111-.113

The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
2026-09-21 12:56:49 +08:00
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 64a81f16ec fix(victoriametrics): enable dedup to accept duplicate samples (#279)
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
2026-09-21 01:21:09 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
Ryan Yin 40b3786f14 docs(kubevirt): refine switch vs boot guidance 2026-09-20 22:45:22 +08:00
Ryan Yin 54016a9da2 fix: address nixpkgs rename/deprecation warnings 2026-09-20 22:42:06 +08:00
Ryan Yin fb522af5c0 feat(kubevirt): cap journald disk usage on all VMs 2026-09-20 22:22:50 +08:00
Ryan Yin 68bea13405 docs(kubevirt): document boot+reboot for KubeVirt hosts 2026-09-20 22:22:47 +08:00
Ryan Yin 4e66a6de57 chore: remove insecure modelscope package 2026-09-20 21:36:23 +08:00
Ryan Yin 9bb608a577 fix(idols-aquamarine): migrate journald.extraConfig to settings.Journal 2026-09-20 21:36:21 +08:00
Ryan Yin 69f60095ae fix(networking): drop shoukei static address
shoukei is a laptop using DHCP and never consumes hostsAddr.shoukei.
Its 192.168.5.108 collided with k3s-prod-1-master-1 and produced a wrong
ssh Host shoukei entry.
2026-09-20 20:26:17 +08:00
Ryan Yin 98fbf010b8 docs: fix broken links and editor references in readmes
- root README: VS Code config moved from home/linux/gui/base/editors.nix to vscode.nix
- READMEs now list both GUI editors (Zed, the primary one, plus VS Code)
- base README: xdg is a directory now, nvidia.nix no longer exists, editors.nix -> vscode.nix
- idols-ai README: disko-fs-media.nix -> disko-fs-data.nix (/persistent/data)
- secrets README: password-store moved from home/base/desktop to home/base/tui
- drop unused link definitions (AstroNvim, wl-clipboard, Catppuccin)
- fix a pre-existing typo flagged by the typos hook (keeped -> kept)
2026-09-19 15:23:09 +08:00
Ryan Yin 33383414a2 fix(niri): match outputs by EDID to survive connector renumbering
The 2026-09 nixpkgs update changed DRM probe order, renaming the Intel connectors DP-1 -> DP-4 and HDMI-A-2 -> HDMI-A-3. The connector-name based output and open-on-output rules stopped matching, which dropped monitor scale/mode/position and sent every named workspace to the first output. Match by make/model/serial instead, which is stable across connector renumbering.
2026-09-19 13:26:24 +08:00
Ryan Yin f23d5d4245 docs(monitoring): note dead-man switch and backup freshness TODOs 2026-09-19 12:14:20 +08:00
Ryan Yin 3318103f8a chore(guix): remove the unused Guix integration
Guix was enabled on every desktop host, but shoukei had no /gnu mount (store and state lived on the tmpfs root) and it was not used. Drop the module, the modules.guix option, the @guix subvolume and the /var/guix preservation entry.
2026-09-19 12:11:15 +08:00