Commit Graph
556 Commits
Author SHA1 Message Date
Ryan Yin 5bbe17ebcd fix(youko): also try disabling the WD idle3 (Intellipark) timer
-B 255 turns APM off and it stuck, but the disks still reach standby, so
the bridge firmware is likely doing the spindown. -J 0 additionally asks
the drive to stop parking its heads (sdb has ~1M load cycles).
2026-09-21 17:43:05 +08:00
Ryan Yin 899162055a fix(youko): drop pcie_aspm=off, re-assert the hdparm settings hourly
pcie_aspm=off was the least-justified mitigation (the flakiness is on the
USB/SATA side) and costs power. Also log the disks' APM level/state and
re-apply the settings hourly, since the bridge or the disk firmware can
drop them (the disks were still reaching standby).
2026-09-21 17:42:21 +08:00
Ryan Yin 45a35a080d fix(youko): address the HDDs by-id, not /dev/sdX
Follows the disko config (ata-WDC_...-<serial>); the sdX names are not
stable.
2026-09-21 17:38:54 +08:00
Ryan Yin ce1645619f fix(youko): mitigate the flaky USB-SATA HDD bridge
The two 4TB HDDs sit behind a JMicron JMS567 that resets and throws
UDMA-CRC / I/O errors (the disks themselves are healthy). Best-effort
mitigations while the enclosure stays in use:

- kernel: usbcore.autosuspend=-1, usb-storage.delay_use=10, pcie_aspm=off
- udev: keep the bridge's power/control=on (never autosuspend)
- disable APM/standby on the HDDs (spindown is a common reset trigger)
- enable smartd (watch the CRC count and disk health)

The kernel parameters need a reboot to take effect.
2026-09-21 17:37:52 +08:00
Ryan Yin e939c54d26 chore: refresh the remaining KubeVirt references
Update the user-facing bits that still pointed at KubeVirt: the homepage
dashboard group and entries (and drop the dead LongHorn one), the docs,
and remove the now-unneeded virtctl package.
2026-09-21 17:25:38 +08:00
Ryan Yin 6180072646 chore: drop the KubeVirt guest bits
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:

- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
  latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
  upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
  restic backup path must stay as it is)
2026-09-21 17:24:40 +08:00
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
Ryan Yin b9c5d3e5c9 docs: describe the k3s-test microVMs
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
2026-09-21 16:37:53 +08:00
Ryan Yin e428d66abf feat(microvm): make the guest volume sizes configurable
etcSize/varSize/homeSize (MiB) with the old defaults. The masters get a
20G /var: k3s server + containerd + logs already filled the old 4G.
2026-09-21 16:37:53 +08:00
Ryan Yin 4ad359696e feat(microvm): run the k3s-test-1 masters as microVMs
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.

- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
  and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
  nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
2026-09-21 16:32:23 +08:00
Ryan Yin bc7d62b063 refactor(kubevirt-youko): resolve cert paths with relativeToRoot
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
2026-09-21 15:50:15 +08:00
Ryan Yin f9c902691b refactor: name the youko service modules by purpose
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
2026-09-21 15:46:55 +08:00
Ryan Yin c292509351 refactor: retire the aquamarine VM host
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:

- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
  drop the explicit import list, leaving aquamarine-services.nix with
  just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload

The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
2026-09-21 15:45:02 +08:00
Ryan Yin dfc3c2884a fix(aquamarine): order transmission after its data mount 2026-09-21 15:39:52 +08:00
Ryan Yin abd28347ab fix(aquamarine): qualify the uptime-kuma image for podman 2026-09-21 15:39:52 +08:00
Ryan Yin 37ebd7a5f8 feat(kubevirt-youko): run aquamarine's services on the host 2026-09-21 15:39:52 +08:00
ryan4yin | 二花 787b51df25 feat(vms): migrate the idol desktops from KubeVirt to libvirt (#287)
* wip(kubevirt-shoryu): enable libvirt/qemu

* feat(kubevirt-shoryu): enable libvirt/qemu with br0 + no NAT network

* chore: lock the microvm flake input

* feat(kubevirt-shoryu): define the libvirt domains declaratively

* fix(kubevirt-shoryu): pin the kana domain UUID

* feat(kubevirt-shoryu): migrate idols-ruby to libvirt
2026-09-21 15:12:37 +08:00
ryan4yin | 二花 3753396a01 docs(k8s): note the flaky USB bridge and the aquamarine microVM plan (#286) 2026-09-21 13:46:39 +08:00
ryan4yin | 二花 d40ed42f6c feat(k3s-test-1): add three worker nodes and taint the control plane (#283)
* feat(k3s-test-1): add three worker nodes and taint the control plane

- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
  never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too

* chore(k3s-test-1): move the workers to .111-.113

The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
2026-09-21 12:56:49 +08:00
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 64a81f16ec fix(victoriametrics): enable dedup to accept duplicate samples (#279)
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
2026-09-21 01:21:09 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
Ryan Yin 40b3786f14 docs(kubevirt): refine switch vs boot guidance 2026-09-20 22:45:22 +08:00
Ryan Yin 54016a9da2 fix: address nixpkgs rename/deprecation warnings 2026-09-20 22:42:06 +08:00
Ryan Yin fb522af5c0 feat(kubevirt): cap journald disk usage on all VMs 2026-09-20 22:22:50 +08:00
Ryan Yin 68bea13405 docs(kubevirt): document boot+reboot for KubeVirt hosts 2026-09-20 22:22:47 +08:00
Ryan Yin 4e66a6de57 chore: remove insecure modelscope package 2026-09-20 21:36:23 +08:00
Ryan Yin 9bb608a577 fix(idols-aquamarine): migrate journald.extraConfig to settings.Journal 2026-09-20 21:36:21 +08:00
Ryan Yin 69f60095ae fix(networking): drop shoukei static address
shoukei is a laptop using DHCP and never consumes hostsAddr.shoukei.
Its 192.168.5.108 collided with k3s-prod-1-master-1 and produced a wrong
ssh Host shoukei entry.
2026-09-20 20:26:17 +08:00
Ryan Yin 98fbf010b8 docs: fix broken links and editor references in readmes
- root README: VS Code config moved from home/linux/gui/base/editors.nix to vscode.nix
- READMEs now list both GUI editors (Zed, the primary one, plus VS Code)
- base README: xdg is a directory now, nvidia.nix no longer exists, editors.nix -> vscode.nix
- idols-ai README: disko-fs-media.nix -> disko-fs-data.nix (/persistent/data)
- secrets README: password-store moved from home/base/desktop to home/base/tui
- drop unused link definitions (AstroNvim, wl-clipboard, Catppuccin)
- fix a pre-existing typo flagged by the typos hook (keeped -> kept)
2026-09-19 15:23:09 +08:00
Ryan Yin 33383414a2 fix(niri): match outputs by EDID to survive connector renumbering
The 2026-09 nixpkgs update changed DRM probe order, renaming the Intel connectors DP-1 -> DP-4 and HDMI-A-2 -> HDMI-A-3. The connector-name based output and open-on-output rules stopped matching, which dropped monitor scale/mode/position and sent every named workspace to the first output. Match by make/model/serial instead, which is stable across connector renumbering.
2026-09-19 13:26:24 +08:00
Ryan Yin f23d5d4245 docs(monitoring): note dead-man switch and backup freshness TODOs 2026-09-19 12:14:20 +08:00
Ryan Yin 3318103f8a chore(guix): remove the unused Guix integration
Guix was enabled on every desktop host, but shoukei had no /gnu mount (store and state lived on the tmpfs root) and it was not used. Drop the module, the modules.guix option, the @guix subvolume and the /var/guix preservation entry.
2026-09-19 12:11:15 +08:00
Ryan Yin fc8b9e5ba0 chore(aquamarine): disable the broken restic backup
The restic job needs three agenix secrets that do not exist yet (restic-password, ssh-key-for-restic-backup and an rclone config for the smb-downloads remote). Comment it out until they are provisioned so it does not fail silently every night.
2026-09-19 12:11:11 +08:00
Ryan Yin d78f6badd4 fix(aquamarine): repair monitoring config
Load both *.yml and *.yaml alert rule files: the Loki rules were named .yaml while only *.yml was loaded, so they silently never ran. Fix the Grafana domain typo (writefo.fun -> writefor.fun). Comment out the postgres-playground datasource, which used the Grafana admin password for a role that has no password and could never authenticate.
2026-09-19 12:11:07 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin 501c6af220 chore(agents): remove rtk integration
RTK compresses only shell output and independent evaluations show little to no end-to-end cost reduction. Drop the package and its install snippets; runtime artifacts were uninstalled separately.
2026-09-19 09:47:52 +08:00
Ryan Yin d80efddd17 fix(agents): leave Codex config unmanaged 2026-09-18 21:20:33 +08:00
Ryan Yin d47710ebe2 fix(agents): merge Codex privacy settings 2026-09-18 21:20:33 +08:00
Ryan Yin 8c62b5433d chore(agents): remove Cursor integrations 2026-09-18 21:20:33 +08:00
Ryan Yin bcee9f2ac8 feat(agents): add Crush and disable usage tracking 2026-09-18 21:20:33 +08:00
Ryan Yin 23258a06d4 feat(agents): add Pi and OMP 2026-09-18 21:20:33 +08:00
Ryan Yin 46266dd8ea chore(agents): remove Claude integrations 2026-09-18 21:20:33 +08:00
Ryan Yin b10b059d1f chore: add microsoft-edge, chromium and brave-origin
Install microsoft-edge on x86_64 Linux via Home Manager and on macOS via
a brew cask. Enable the chromium and brave-origin Home Manager options
alongside google-chrome, dropping the google-chrome aarch64 fallback to
chromium since google-chrome supports aarch64-linux.

Persist the Brave Origin and microsoft-edge profile directories on the
ephemeral-root hosts.
2026-09-16 18:51:23 +08:00
Ryan Yin c8642fcac4 feat(herdr): declare config in home-manager and persist state 2026-09-14 13:18:57 +08:00
Ryan Yin 77e31bd4cb fix(idols-ai): keep network config across S3 resume
The r8169 NIC drops carrier on S3 resume, and systemd-networkd reacts by reconfiguring enp130s0: it dropped the static address/default route and deleted foreign routing policy rules (mihomo's 9000-9010 and Tailscale's 5210-5270). clash's TUN then lost its outbound interface and networking stayed broken until clash was restarted.

- IgnoreCarrierLoss=10s keeps the static address/route across the short carrier loss.
- ManageForeignRoutingPolicyRules=false stops networkd from deleting the TUN/VPN ip rules during the reconfiguration.
2026-09-14 10:54:33 +08:00
Ryan Yin 324e5d8820 feat(moonlight): persist client data 2026-09-04 20:21:52 +08:00