Files
nix-config/hosts/k8s
Ryan Yin 2c7629000f refactor: rename the three VM hosts to 12kingdoms-*
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).

- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
  becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
  tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
  their DNS name changes with the hostname and is not resolvable until
  the router learns it
2026-09-21 17:10:14 +08:00
..

Kubernetes Clusters

WIP, not finished yet.

I'm running a Kubernetes cluster for testing and development.

I prefer to use k3s as the Kubernetes distribution, because it's lightweight, easy to install, and full featured(see what-have-k3s-removed-from-upstream-kubernetes for details).

KubeVirt Cluster

The KubeVirt cluster is running on physical machines, all my virtual machines are running on this cluster, including other Kubernetes clusters.

K3s Clusters

The control plane runs as NixOS microVMs (microvm.nix) on the physical hosts, not as KubeVirt VMs. Each guest keeps its state in three sparse ext4 images under the host's /var/lib/microvms/<name>/:

  • etc.img (64M) — the ssh host key (also the agenix age identity) and machine-id
  • var.img (20G) — k3s state (/var/lib/rancher/k3s) and the NixOS uid/gid maps
  • home.img (4G) — the user's home

The guest's root is a tmpfs and /nix/store is the host's store shared read-only (virtiofs), so no image has to be built or uploaded; only the volumes above persist. The guest's tap interface is bridged onto br0, and its name is derived from the guest IP (e.g. 192.168.5.114 -> vm114, as IFNAMSIZ caps interface names at 15 characters).

To move an existing KubeVirt VM over in place, copy the host key and machine-id from its disk into etc.img, and /var/lib/rancher/k3s/server into var.img, before the first boot — then it keeps its identity and etcd membership instead of bootstraping a new cluster.

  1. k3s-test-1-master-{1,2,3} — running as microVMs
  2. k3s-test-1-worker-{1,2,3} — to be re-added as microVMs

TODO / Known issues

  • Retire KubeVirt. aquamarine now runs natively on youko, and the k3s-test masters run as microVMs. Remove KubeVirt/CDI/multus from the cluster (k8s-gitops) once nothing else needs them, and the Windows VM is the only KubeVirt VM left to move.
  • The USB HDD bridge is flaky. aquamarine's two HDDs sit behind a JMicron JMS567 USB-SATA bridge that keeps resetting (dmesg on youko). Find out how often it resets and how much it matters before putting anything critical (e.g. an NFS export) on it.

Kubernetes Resources

Kubernetes resources are deployed and managed separately through ryan4yin/k8s-gitops.