Files
nix-config/hosts/k8s
Ryan Yin cf2435e854 chore(hosts): retire the k3s cluster on the physical hosts
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:

- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
  physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
  journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
  longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
  virtctl package
- remove lib/genKubeVirtHostModule.nix
2026-09-21 16:57:42 +08:00
..
2026-09-21 16:37:53 +08:00

Kubernetes Clusters

WIP, not finished yet.

I'm running a Kubernetes cluster for testing and development.

I prefer to use k3s as the Kubernetes distribution, because it's lightweight, easy to install, and full featured(see what-have-k3s-removed-from-upstream-kubernetes for details).

KubeVirt Cluster

The KubeVirt cluster is running on physical machines, all my virtual machines are running on this cluster, including other Kubernetes clusters.

K3s Clusters

The control plane runs as NixOS microVMs (microvm.nix) on the physical hosts, not as KubeVirt VMs. Each guest keeps its state in three sparse ext4 images under the host's /var/lib/microvms/<name>/:

  • etc.img (64M) — the ssh host key (also the agenix age identity) and machine-id
  • var.img (20G) — k3s state (/var/lib/rancher/k3s) and the NixOS uid/gid maps
  • home.img (4G) — the user's home

The guest's root is a tmpfs and /nix/store is the host's store shared read-only (virtiofs), so no image has to be built or uploaded; only the volumes above persist. The guest's tap interface is bridged onto br0, and its name is derived from the guest IP (e.g. 192.168.5.114 -> vm114, as IFNAMSIZ caps interface names at 15 characters).

To move an existing KubeVirt VM over in place, copy the host key and machine-id from its disk into etc.img, and /var/lib/rancher/k3s/server into var.img, before the first boot — then it keeps its identity and etcd membership instead of bootstraping a new cluster.

  1. k3s-test-1-master-{1,2,3} — running as microVMs
  2. k3s-test-1-worker-{1,2,3} — to be re-added as microVMs

TODO / Known issues

  • Retire KubeVirt. aquamarine now runs natively on kubevirt-youko, and the k3s-test masters run as microVMs. Remove KubeVirt/CDI/multus from the cluster (k8s-gitops) once nothing else needs them, and the Windows VM is the only KubeVirt VM left to move.
  • The USB HDD bridge is flaky. aquamarine's two HDDs sit behind a JMicron JMS567 USB-SATA bridge that keeps resetting (dmesg on kubevirt-youko). Find out how often it resets and how much it matters before putting anything critical (e.g. an NFS export) on it.

Kubernetes Resources

Kubernetes resources are deployed and managed separately through ryan4yin/k8s-gitops.