Commit Graph
87 Commits
Author SHA1 Message Date
ryan4yin | 二花 9c38819517 docs: review and correct comments and documentation (#436) 2026-10-12 10:02:38 +08:00
Ryan Yin 60b80e64c8 fix(microvm): restart the guest unit on the physical host to deploy
microvm.nix's sshSwitch step runs nix-store --load-db against the guest's
read-only virtiofs /nix/store and dies there, so the recipe installs the
runner on the physical host and restarts microvm@<guest> instead.
2026-10-10 08:42:38 +08:00
Ryan Yin a2cfa5549c fix(skills): correct stale guidance and widen task triggers 2026-10-10 08:42:38 +08:00
ryan4yin | 二花 5f6a1dadf4 docs: fix stale claims and AI-flavored prose (#397) 2026-10-07 14:48:36 +08:00
ryan4yin | 二花 3a5b9ec4c6 docs: README audit fixes and de-duplication (#371) 2026-10-03 18:28:58 +08:00
ryan4yin | 二花 6a1b4c699a chore(docs): prune stale TODOs (#370) 2026-10-03 17:50:28 +08:00
ryan4yin | 二花 68ea04f173 fix(docs): correct stale and mismatched documentation (#338) 2026-10-02 10:21:55 +08:00
Ryan Yin c7d8db2bf3 fix(k8s): rebalance k3s-test-1 VMs after the youko/shushou swap 2026-09-29 21:46:09 +08:00
Ryan Yin 9a6923b4e9 fix(k3s): drop Home Manager from test masters
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
2026-09-24 13:35:33 +08:00
ryan4yin | 二花 069e081fec fix(k3s): bind control-plane metrics to node addresses (#311)
* fix(k3s): bind control-plane metrics to node addresses

* docs(microvm): document remote deployment workflow

* fix(microvm): deploy runner before guest switch

* docs(microvm): standardize on just deployment commands

* docs(k3s): explain control-plane metrics address

* docs(hosts): move generic build commands to host guide

* docs: simplify command operation guidance
2026-09-24 11:13:59 +08:00
Ryan Yin 64cad91401 fix(k8s): unblock the k3s-test-1 workers
kubelet refuses to self-assign the reserved node-role.kubernetes.io/* labels, so the worker --node-label made the agent crash-loop. Drop it and rely on the masters' control-plane taint for placement; document that the taint only lands via registerWithTaints on first registration, so it was applied by hand here.
2026-09-23 12:07:28 +08:00
ryan4yin | 二花 b19818ef44 feat(k8s): add the k3s-test-1 worker nodes (#307)
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
2026-09-21 23:09:13 +08:00
ryan4yin | 二花 97c2d1325d docs: refresh stale docs after the VM and backup refactors 2026-09-21 22:03:21 +08:00
Ryan Yin e939c54d26 chore: refresh the remaining KubeVirt references
Update the user-facing bits that still pointed at KubeVirt: the homepage
dashboard group and entries (and drop the dead LongHorn one), the docs,
and remove the now-unneeded virtctl package.
2026-09-21 17:25:38 +08:00
Ryan Yin 6180072646 chore: drop the KubeVirt guest bits
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:

- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
  latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
  upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
  restic backup path must stay as it is)
2026-09-21 17:24:40 +08:00
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
Ryan Yin b9c5d3e5c9 docs: describe the k3s-test microVMs
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
2026-09-21 16:37:53 +08:00
Ryan Yin e428d66abf feat(microvm): make the guest volume sizes configurable
etcSize/varSize/homeSize (MiB) with the old defaults. The masters get a
20G /var: k3s server + containerd + logs already filled the old 4G.
2026-09-21 16:37:53 +08:00
Ryan Yin 4ad359696e feat(microvm): run the k3s-test-1 masters as microVMs
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.

- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
  and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
  nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
2026-09-21 16:32:23 +08:00
Ryan Yin bc7d62b063 refactor(kubevirt-youko): resolve cert paths with relativeToRoot
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
2026-09-21 15:50:15 +08:00
Ryan Yin f9c902691b refactor: name the youko service modules by purpose
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
2026-09-21 15:46:55 +08:00
Ryan Yin c292509351 refactor: retire the aquamarine VM host
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:

- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
  drop the explicit import list, leaving aquamarine-services.nix with
  just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload

The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
2026-09-21 15:45:02 +08:00
Ryan Yin 37ebd7a5f8 feat(kubevirt-youko): run aquamarine's services on the host 2026-09-21 15:39:52 +08:00
ryan4yin | 二花 787b51df25 feat(vms): migrate the idol desktops from KubeVirt to libvirt (#287)
* wip(kubevirt-shoryu): enable libvirt/qemu

* feat(kubevirt-shoryu): enable libvirt/qemu with br0 + no NAT network

* chore: lock the microvm flake input

* feat(kubevirt-shoryu): define the libvirt domains declaratively

* fix(kubevirt-shoryu): pin the kana domain UUID

* feat(kubevirt-shoryu): migrate idols-ruby to libvirt
2026-09-21 15:12:37 +08:00
ryan4yin | 二花 3753396a01 docs(k8s): note the flaky USB bridge and the aquamarine microVM plan (#286) 2026-09-21 13:46:39 +08:00
ryan4yin | 二花 d40ed42f6c feat(k3s-test-1): add three worker nodes and taint the control plane (#283)
* feat(k3s-test-1): add three worker nodes and taint the control plane

- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
  never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too

* chore(k3s-test-1): move the workers to .111-.113

The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
2026-09-21 12:56:49 +08:00
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin 3550c6af2a feat(monitoring): collect homelab UPS metrics 2026-07-24 12:44:17 +08:00
Ryan Yin 8b4e922e5b feat(networking): move tailscale subnet router to kubevirt-youko 2026-03-23 10:34:51 +08:00
Ryan Yin f6aa45aa00 feat: idols-ai - add new data disk via disko (#250)
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
2026-03-18 22:26:08 +08:00
Ryan Yin 69f77fecca feat: new host - idols-akane, hardens VFAT /boot mounts (#245)
* feat: new host - idols-akane

* fix: missing efi files

* fix: efi partition - permission issue
2026-03-07 23:54:13 +08:00
Ryan Yin 1222bb25d0 feat(preservation): add .openclaw, harden home dir permissions
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
2026-02-27 10:18:49 +08:00
Ryan Yin e3a33fdf03 fix: podman cross platform - binfmt 2025-09-24 11:47:23 +08:00
Ryan Yin 13bb77108c feat: format via 'ls **/*.nix | each { |it| nixfmt $it.name }' 2025-07-30 12:17:24 +08:00
Ryan Yin c231ba66f8 chore: nushell do not use history.txt now 2025-07-16 23:00:23 +08:00
Ryan Yin ffee47a2de chore: replace impermanence with preservation (#201) 2025-06-29 15:08:12 +08:00
Ryan Yin 0004bccc9d feat: hosts/k8s - fix k3s cidr (#200) 2025-06-29 15:00:53 +08:00
Ryan Yin aaabb5ed76 feat: hosts/k8s - replace impermanence with preservation (#199) 2025-06-29 14:56:08 +08:00
Ryan Yin 8690978d7e feat: replace impermanence with preservation (#198)
* feat: replace impermanence with preservation

* fix: This module cannot be used with scripted initrd.

* fix: permission for /var/lib/private

* chore: boot.initrd.systemd.enable

* docs: boot.initrd.luks.devices.<name>.fallbackToPassword is implied by systemd stage 1

* fix: typo
2025-06-29 13:02:36 +08:00
Ryan Yin 77ed0378d1 feat: IPv6 (#192)
* feat: custom ipv6 routes for all hosts
* fix: ipv6 - k3s+cilium
2025-06-08 19:12:55 +08:00
Ryan Yin b82395d603 docs: fix style 2024-09-22 22:55:10 +08:00
Ryan Yin 43db6bcf63 feat: kubevirt - use 75% of the memory for hugepages 2024-08-15 17:19:26 +08:00
Ryan Yin e2457e80aa docs: kubevirt-youko - pci passthrough 2024-07-30 10:22:42 +08:00
Ryan Yin 8ffaf4e3ae feat: decrease pre-allocated hugepages 2024-07-30 01:21:04 +08:00
Ryan Yin fed3bc981b fix: k3s test cluster - token path 2024-07-29 23:42:14 +08:00