Commit Graph
550 Commits
Author SHA1 Message Date
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
Ryan Yin b9c5d3e5c9 docs: describe the k3s-test microVMs
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
2026-09-21 16:37:53 +08:00
Ryan Yin e428d66abf feat(microvm): make the guest volume sizes configurable
etcSize/varSize/homeSize (MiB) with the old defaults. The masters get a
20G /var: k3s server + containerd + logs already filled the old 4G.
2026-09-21 16:37:53 +08:00
Ryan Yin 4ad359696e feat(microvm): run the k3s-test-1 masters as microVMs
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.

- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
  and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
  nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
2026-09-21 16:32:23 +08:00
Ryan Yin bc7d62b063 refactor(kubevirt-youko): resolve cert paths with relativeToRoot
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
2026-09-21 15:50:15 +08:00
Ryan Yin f9c902691b refactor: name the youko service modules by purpose
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
2026-09-21 15:46:55 +08:00
Ryan Yin c292509351 refactor: retire the aquamarine VM host
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:

- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
  drop the explicit import list, leaving aquamarine-services.nix with
  just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload

The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
2026-09-21 15:45:02 +08:00
Ryan Yin dfc3c2884a fix(aquamarine): order transmission after its data mount 2026-09-21 15:39:52 +08:00
Ryan Yin abd28347ab fix(aquamarine): qualify the uptime-kuma image for podman 2026-09-21 15:39:52 +08:00
Ryan Yin 37ebd7a5f8 feat(kubevirt-youko): run aquamarine's services on the host 2026-09-21 15:39:52 +08:00
ryan4yin | 二花 787b51df25 feat(vms): migrate the idol desktops from KubeVirt to libvirt (#287)
* wip(kubevirt-shoryu): enable libvirt/qemu

* feat(kubevirt-shoryu): enable libvirt/qemu with br0 + no NAT network

* chore: lock the microvm flake input

* feat(kubevirt-shoryu): define the libvirt domains declaratively

* fix(kubevirt-shoryu): pin the kana domain UUID

* feat(kubevirt-shoryu): migrate idols-ruby to libvirt
2026-09-21 15:12:37 +08:00
ryan4yin | 二花 3753396a01 docs(k8s): note the flaky USB bridge and the aquamarine microVM plan (#286) 2026-09-21 13:46:39 +08:00
ryan4yin | 二花 d40ed42f6c feat(k3s-test-1): add three worker nodes and taint the control plane (#283)
* feat(k3s-test-1): add three worker nodes and taint the control plane

- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
  never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too

* chore(k3s-test-1): move the workers to .111-.113

The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
2026-09-21 12:56:49 +08:00
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 64a81f16ec fix(victoriametrics): enable dedup to accept duplicate samples (#279)
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
2026-09-21 01:21:09 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
Ryan Yin 40b3786f14 docs(kubevirt): refine switch vs boot guidance 2026-09-20 22:45:22 +08:00
Ryan Yin 54016a9da2 fix: address nixpkgs rename/deprecation warnings 2026-09-20 22:42:06 +08:00
Ryan Yin fb522af5c0 feat(kubevirt): cap journald disk usage on all VMs 2026-09-20 22:22:50 +08:00
Ryan Yin 68bea13405 docs(kubevirt): document boot+reboot for KubeVirt hosts 2026-09-20 22:22:47 +08:00
Ryan Yin 4e66a6de57 chore: remove insecure modelscope package 2026-09-20 21:36:23 +08:00
Ryan Yin 9bb608a577 fix(idols-aquamarine): migrate journald.extraConfig to settings.Journal 2026-09-20 21:36:21 +08:00
Ryan Yin 69f60095ae fix(networking): drop shoukei static address
shoukei is a laptop using DHCP and never consumes hostsAddr.shoukei.
Its 192.168.5.108 collided with k3s-prod-1-master-1 and produced a wrong
ssh Host shoukei entry.
2026-09-20 20:26:17 +08:00
Ryan Yin 98fbf010b8 docs: fix broken links and editor references in readmes
- root README: VS Code config moved from home/linux/gui/base/editors.nix to vscode.nix
- READMEs now list both GUI editors (Zed, the primary one, plus VS Code)
- base README: xdg is a directory now, nvidia.nix no longer exists, editors.nix -> vscode.nix
- idols-ai README: disko-fs-media.nix -> disko-fs-data.nix (/persistent/data)
- secrets README: password-store moved from home/base/desktop to home/base/tui
- drop unused link definitions (AstroNvim, wl-clipboard, Catppuccin)
- fix a pre-existing typo flagged by the typos hook (keeped -> kept)
2026-09-19 15:23:09 +08:00
Ryan Yin 33383414a2 fix(niri): match outputs by EDID to survive connector renumbering
The 2026-09 nixpkgs update changed DRM probe order, renaming the Intel connectors DP-1 -> DP-4 and HDMI-A-2 -> HDMI-A-3. The connector-name based output and open-on-output rules stopped matching, which dropped monitor scale/mode/position and sent every named workspace to the first output. Match by make/model/serial instead, which is stable across connector renumbering.
2026-09-19 13:26:24 +08:00
Ryan Yin f23d5d4245 docs(monitoring): note dead-man switch and backup freshness TODOs 2026-09-19 12:14:20 +08:00
Ryan Yin 3318103f8a chore(guix): remove the unused Guix integration
Guix was enabled on every desktop host, but shoukei had no /gnu mount (store and state lived on the tmpfs root) and it was not used. Drop the module, the modules.guix option, the @guix subvolume and the /var/guix preservation entry.
2026-09-19 12:11:15 +08:00
Ryan Yin fc8b9e5ba0 chore(aquamarine): disable the broken restic backup
The restic job needs three agenix secrets that do not exist yet (restic-password, ssh-key-for-restic-backup and an rclone config for the smb-downloads remote). Comment it out until they are provisioned so it does not fail silently every night.
2026-09-19 12:11:11 +08:00
Ryan Yin d78f6badd4 fix(aquamarine): repair monitoring config
Load both *.yml and *.yaml alert rule files: the Loki rules were named .yaml while only *.yml was loaded, so they silently never ran. Fix the Grafana domain typo (writefo.fun -> writefor.fun). Comment out the postgres-playground datasource, which used the Grafana admin password for a role that has no password and could never authenticate.
2026-09-19 12:11:07 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin 501c6af220 chore(agents): remove rtk integration
RTK compresses only shell output and independent evaluations show little to no end-to-end cost reduction. Drop the package and its install snippets; runtime artifacts were uninstalled separately.
2026-09-19 09:47:52 +08:00
Ryan Yin d80efddd17 fix(agents): leave Codex config unmanaged 2026-09-18 21:20:33 +08:00
Ryan Yin d47710ebe2 fix(agents): merge Codex privacy settings 2026-09-18 21:20:33 +08:00
Ryan Yin 8c62b5433d chore(agents): remove Cursor integrations 2026-09-18 21:20:33 +08:00
Ryan Yin bcee9f2ac8 feat(agents): add Crush and disable usage tracking 2026-09-18 21:20:33 +08:00
Ryan Yin 23258a06d4 feat(agents): add Pi and OMP 2026-09-18 21:20:33 +08:00
Ryan Yin 46266dd8ea chore(agents): remove Claude integrations 2026-09-18 21:20:33 +08:00
Ryan Yin b10b059d1f chore: add microsoft-edge, chromium and brave-origin
Install microsoft-edge on x86_64 Linux via Home Manager and on macOS via
a brew cask. Enable the chromium and brave-origin Home Manager options
alongside google-chrome, dropping the google-chrome aarch64 fallback to
chromium since google-chrome supports aarch64-linux.

Persist the Brave Origin and microsoft-edge profile directories on the
ephemeral-root hosts.
2026-09-16 18:51:23 +08:00
Ryan Yin c8642fcac4 feat(herdr): declare config in home-manager and persist state 2026-09-14 13:18:57 +08:00
Ryan Yin 77e31bd4cb fix(idols-ai): keep network config across S3 resume
The r8169 NIC drops carrier on S3 resume, and systemd-networkd reacts by reconfiguring enp130s0: it dropped the static address/default route and deleted foreign routing policy rules (mihomo's 9000-9010 and Tailscale's 5210-5270). clash's TUN then lost its outbound interface and networking stayed broken until clash was restarted.

- IgnoreCarrierLoss=10s keeps the static address/route across the short carrier loss.
- ManageForeignRoutingPolicyRules=false stops networkd from deleting the TUN/VPN ip rules during the reconfiguration.
2026-09-14 10:54:33 +08:00
Ryan Yin 324e5d8820 feat(moonlight): persist client data 2026-09-04 20:21:52 +08:00
Ryan Yin 83833bf871 fix: use valid shoukei logind keys 2026-09-03 23:26:03 +08:00
Ryan Yin 1ec29059ad fix(sunshine): stabilize remote Niri sessions 2026-09-01 22:32:19 +08:00
ryan4yin | 二花 9db935bc87 fix(sunshine): use Intel rendering and recover remote Niri sessions (#268)
- render Niri on the Intel iGPU while retaining NVIDIA-connected physical outputs
- encode Sunshine streams with Intel VAAPI and wait for a Niri output before startup
- load VKMS persistently for headless sessions and configure `Virtual-1` at 2560x1600@60
- let Moonlight request the stream bitrate instead of enforcing a 20 Mbps host cap
- remove unsupported Dynamic Boost and global NVIDIA session overrides
- grant Sunshine access to virtual input devices
- add a Nushell recovery helper that starts a temporary greetd/Niri session from SSH
- identify and safely replace normal, standalone, or previous transient Niri sessions
- launch the transient service in the background and verify greetd, Niri, and Sunshine before returning
- ignore local `.worktrees` directories and keep eval regression coverage focused on structural behavior
2026-09-01 03:40:48 -06:00
Ryan Yin c6bbb69b3d fix(shoukei): muvm failed to build currently 2026-08-30 23:02:51 +08:00
Ryan Yin 099752e89f fix(shoukei): restore tuigreet login, autologin bypassed the lock screen
greetd ran the niri session directly, so exiting the session (e.g.
logout from the lock screen) made greetd immediately restart the
desktop with no authentication.
2026-08-30 21:47:57 +08:00
Ryan Yin a0d55a298e feat(remote-desktop): enable Sunshine on ai and Moonlight on darwin 2026-08-30 12:57:09 +08:00