Commit Graph
72 Commits
Author SHA1 Message Date
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
Ryan Yin b9c5d3e5c9 docs: describe the k3s-test microVMs
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
2026-09-21 16:37:53 +08:00
Ryan Yin e428d66abf feat(microvm): make the guest volume sizes configurable
etcSize/varSize/homeSize (MiB) with the old defaults. The masters get a
20G /var: k3s server + containerd + logs already filled the old 4G.
2026-09-21 16:37:53 +08:00
Ryan Yin 4ad359696e feat(microvm): run the k3s-test-1 masters as microVMs
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.

- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
  and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
  nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
2026-09-21 16:32:23 +08:00
Ryan Yin bc7d62b063 refactor(kubevirt-youko): resolve cert paths with relativeToRoot
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
2026-09-21 15:50:15 +08:00
Ryan Yin f9c902691b refactor: name the youko service modules by purpose
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
2026-09-21 15:46:55 +08:00
Ryan Yin c292509351 refactor: retire the aquamarine VM host
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:

- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
  drop the explicit import list, leaving aquamarine-services.nix with
  just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload

The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
2026-09-21 15:45:02 +08:00
Ryan Yin 37ebd7a5f8 feat(kubevirt-youko): run aquamarine's services on the host 2026-09-21 15:39:52 +08:00
ryan4yin | 二花 787b51df25 feat(vms): migrate the idol desktops from KubeVirt to libvirt (#287)
* wip(kubevirt-shoryu): enable libvirt/qemu

* feat(kubevirt-shoryu): enable libvirt/qemu with br0 + no NAT network

* chore: lock the microvm flake input

* feat(kubevirt-shoryu): define the libvirt domains declaratively

* fix(kubevirt-shoryu): pin the kana domain UUID

* feat(kubevirt-shoryu): migrate idols-ruby to libvirt
2026-09-21 15:12:37 +08:00
ryan4yin | 二花 3753396a01 docs(k8s): note the flaky USB bridge and the aquamarine microVM plan (#286) 2026-09-21 13:46:39 +08:00
ryan4yin | 二花 d40ed42f6c feat(k3s-test-1): add three worker nodes and taint the control plane (#283)
* feat(k3s-test-1): add three worker nodes and taint the control plane

- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
  never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too

* chore(k3s-test-1): move the workers to .111-.113

The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
2026-09-21 12:56:49 +08:00
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin 3550c6af2a feat(monitoring): collect homelab UPS metrics 2026-07-24 12:44:17 +08:00
Ryan Yin 8b4e922e5b feat(networking): move tailscale subnet router to kubevirt-youko 2026-03-23 10:34:51 +08:00
Ryan Yin f6aa45aa00 feat: idols-ai - add new data disk via disko (#250)
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
2026-03-18 22:26:08 +08:00
Ryan Yin 69f77fecca feat: new host - idols-akane, hardens VFAT /boot mounts (#245)
* feat: new host - idols-akane

* fix: missing efi files

* fix: efi partition - permission issue
2026-03-07 23:54:13 +08:00
Ryan Yin 1222bb25d0 feat(preservation): add .openclaw, harden home dir permissions
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
2026-02-27 10:18:49 +08:00
Ryan Yin e3a33fdf03 fix: podman cross platform - binfmt 2025-09-24 11:47:23 +08:00
Ryan Yin 13bb77108c feat: format via 'ls **/*.nix | each { |it| nixfmt $it.name }' 2025-07-30 12:17:24 +08:00
Ryan Yin c231ba66f8 chore: nushell do not use history.txt now 2025-07-16 23:00:23 +08:00
Ryan Yin ffee47a2de chore: replace impermanence with preservation (#201) 2025-06-29 15:08:12 +08:00
Ryan Yin 0004bccc9d feat: hosts/k8s - fix k3s cidr (#200) 2025-06-29 15:00:53 +08:00
Ryan Yin aaabb5ed76 feat: hosts/k8s - replace impermanence with preservation (#199) 2025-06-29 14:56:08 +08:00
Ryan Yin 8690978d7e feat: replace impermanence with preservation (#198)
* feat: replace impermanence with preservation

* fix: This module cannot be used with scripted initrd.

* fix: permission for /var/lib/private

* chore: boot.initrd.systemd.enable

* docs: boot.initrd.luks.devices.<name>.fallbackToPassword is implied by systemd stage 1

* fix: typo
2025-06-29 13:02:36 +08:00
Ryan Yin 77ed0378d1 feat: IPv6 (#192)
* feat: custom ipv6 routes for all hosts
* fix: ipv6 - k3s+cilium
2025-06-08 19:12:55 +08:00
Ryan Yin b82395d603 docs: fix style 2024-09-22 22:55:10 +08:00
Ryan Yin 43db6bcf63 feat: kubevirt - use 75% of the memory for hugepages 2024-08-15 17:19:26 +08:00
Ryan Yin e2457e80aa docs: kubevirt-youko - pci passthrough 2024-07-30 10:22:42 +08:00
Ryan Yin 8ffaf4e3ae feat: decrease pre-allocated hugepages 2024-07-30 01:21:04 +08:00
Ryan Yin fed3bc981b fix: k3s test cluster - token path 2024-07-29 23:42:14 +08:00
Ryan Yin 5ec72c848f fix: allocate only hugepages with size 1Gi 2024-07-29 22:31:17 +08:00
Ryan Yin 5abbd63284 docs: fix the maxium keyfile size of luks 2024-04-24 11:22:15 +08:00
Ryan Yin 9a0e41429a docs: nixos-install + persistent 2024-04-20 13:29:03 +08:00
Ryan Yin 5b2d09d0e7 feat(hosts): cifs mount 2024-04-12 21:29:17 +08:00
Ryan Yin e789e5aff3 docs: kubevirt cluster - screentshots 2024-04-02 00:19:36 +08:00
Ryan Yin a9e3dd0c53 docs: kubevirt installation 2024-04-02 00:12:07 +08:00
Ryan Yin 75202fb46c docs: kubevirt installation 2024-04-02 00:04:09 +08:00
Ryan Yin ed5328c8de docs: kubevirt installation 2024-04-01 23:39:21 +08:00
Ryan Yin 80c0ab327f docs: kubevirt installation 2024-04-01 23:17:57 +08:00
Ryan Yin 4dc50e7e8c feat(hosts/k8s): kubevirt - set cpu-manager's policy to static for all nodes 2024-04-01 23:12:16 +08:00
Ryan Yin 3bf5ff65de docs: kubevirt - system-reserved 2024-04-01 22:54:34 +08:00
Ryan Yin 4775cc2742 feat(hosts/k8s): kubevirt - set cpu-manager's policy to static 2024-04-01 22:49:07 +08:00
Ryan Yin c628eaa6c8 fix: run other k3s clusters as kubevirt quests, use the interface provided by kubevirt 2024-04-01 00:39:52 +08:00
Ryan Yin 68629e929c fix: kubevirt - failed to configure vmi network: setup failed, err: pod link (pod6b4853bd4f2) is missing 2024-03-31 22:42:48 +08:00
Ryan Yin 4da995fa88 feat(modules/nixos/base,hosts): add btrbk for filesystem backup 2024-03-31 15:35:08 +08:00