0 Fork 0

mirror of https://github.com/ryan4yin/nix-config.git synced 2026-11-13 12:13:36 +01:00

Files

T

Ryan Yin 60b80e64c8 fix(microvm): restart the guest unit on the physical host to deploy

microvm.nix's sshSwitch step runs nix-store --load-db against the guest's
read-only virtiofs /nix/store and dies there, so the recipe installs the
runner on the physical host and restarts microvm@<guest> instead.

2026-10-10 08:42:38 +08:00

3.8 KiB

Raw Blame History

Kubernetes Clusters

I'm running a Kubernetes cluster for testing and development.

I prefer to use k3s as the Kubernetes distribution, because it's lightweight, easy to install, and full featured(see what-have-k3s-removed-from-upstream-kubernetes for details).

VM Cluster

The VM cluster is running on physical machines; all my virtual machines run on these hosts, including the k3s-test-1 cluster.

K3s Clusters

The control plane runs as NixOS microVMs (microvm.nix) on the physical hosts. Each guest keeps its state in three sparse ext4 images under the host's /var/lib/microvms/<name>/:

etc.img (64M) — the ssh host key (also the agenix age identity) and machine-id
var.img (20G) — k3s state (/var/lib/rancher/k3s) and the NixOS uid/gid maps
home.img (4G) — the user's home

The guest's root is a tmpfs and /nix/store is the host's store shared read-only (virtiofs), so no image has to be built or uploaded; only the volumes above persist. The guest's tap interface is bridged onto br0, and its name is derived from the last octet of the guest IP by lib/genMicrovmGuestModule.nix (e.g. 192.168.5.114 -> vm114).

Updating a running guest

Deploy a guest with the repository's just recipe. It evaluates the guest locally, transfers the complete Nix closure to the physical host, installs it under /var/lib/microvms/<name>, and activates the guest:

just microvm-deploy <guest> <physical-host>

For example, just microvm-deploy k3s-test-1-master-3 shushou. The recipe copies the runner to the physical host, then restarts microvm@<guest> there; microvm.nix's SSH switch is not used because it cannot write the guest's read-only /nix/store (WA-026 in WORKAROUNDS.md). A later activation of the physical host re-points the guest at the runner baked into the host's system, so switch the host after installing a guest from a newer tree.

Use just build-microvm <guest> to build a runner without deploying it. Preserve the guest's etc.img, var.img, and home.img when updating it; they contain its identity and K3s state.

For a cluster rollout, deploy one guest at a time. Before each restart, record the active runner and verify that the new runner exists on the physical host. After each restart, wait for the guest Node to become Ready, then check the API VIP /readyz, Cilium, storage CSI, and any workload health gates. If a gate fails, restore the recorded runner through the same host access path and start the guest service before proceeding to the next node.

k3s-test-1-master-{1,2,3} — control plane, running as microVMs; tainted node-role.kubernetes.io/control-plane:NoSchedule
k3s-test-1-worker-{1,2,3} — workloads, running as microVMs

Placement: worker-1 (4 vCPU / 16 GiB) runs on shoryu; worker-2 (4 vCPU / 16 GiB) runs on youko; worker-3 (2 vCPU / 8 GiB) runs on shushou.

The master taint comes from kubelet's registerWithTaints, which only takes effect when the Node object is first created. These masters are long-lived and predate the config, so the taint was applied once by hand (kubectl taint nodes <name> node-role.kubernetes.io/control-plane=:NoSchedule); a node rebuilt from scratch gets it automatically. The workers carry no node-role.kubernetes.io/worker label: kubelet refuses to self-assign the reserved node-role.kubernetes.io/* labels via --node-label, and placement is enforced by the control-plane taint instead.

Kubernetes Resources

Kubernetes resources are deployed and managed separately through ryan4yin/k8s-gitops.

3.8 KiB Raw Blame History