diff --git a/AGENTS.md b/AGENTS.md index 369b842e..639f25e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,6 +68,10 @@ deployments. Keep repository guidance here; reusable global rules live in `agent - `just clean`, `just gc`, `just ggc`, and `just game` remove history or amend commits; they are not validation steps and require explicit authorization for their target and scope. - Do not use `just penvof` for process inspection: it can expose secret values. +- MicroVM guests (`k3s-test-1-master-*`) keep their state in + `/var/lib/microvms//{etc,var,home}.img` on their host. Deleting or recreating an image loses + the node's identity and cluster state; to resize one, stop the guest and grow the image in place + (`truncate` + `e2fsck -f` + `resize2fs`) rather than re-creating it. ## Further Context diff --git a/Virtual-Machine.md b/Virtual-Machine.md index 8bae5e46..c0d31c1c 100644 --- a/Virtual-Machine.md +++ b/Virtual-Machine.md @@ -1,5 +1,9 @@ ## How to create & managage KubeVirt's Virtual Machine from this flake? +> [!NOTE] KubeVirt is being retired. aquamarine runs natively on `kubevirt-youko` and the k3s-test +> nodes run as [microVMs](./hosts/k8s/README.md); this document only covers the KubeVirt VMs that +> are left to move (currently the Windows VM). + Use `aquamarine` as an example, first build and upload the virtual machine's qcow2 image to the file server: @@ -19,7 +23,6 @@ commands: ```shell just col aquamarine just col kubevirt-shoryu -just col k3s-test-1-master-1 # Set a configuration for the next boot instead of switching immediately just col aquamarine boot diff --git a/hosts/k8s/README.md b/hosts/k8s/README.md index 6dfe3089..64e26d1d 100644 --- a/hosts/k8s/README.md +++ b/hosts/k8s/README.md @@ -17,26 +17,34 @@ cluster, including other Kubernetes clusters. ## K3s Clusters -Clusters running as virtual machines on the KubeVirt cluster, for testing and development. +The control plane runs as NixOS **microVMs** ([microvm.nix]) on the physical hosts, not as KubeVirt +VMs. Each guest keeps its state in three sparse ext4 images under the host's +`/var/lib/microvms//`: -![](_img/2024-02-18_k8s-nodes-overview.webp) +- `etc.img` (64M) — the ssh host key (also the agenix age identity) and `machine-id` +- `var.img` (20G) — k3s state (`/var/lib/rancher/k3s`) and the NixOS uid/gid maps +- `home.img` (4G) — the user's home -1. `k3s-test-1-master-1` -1. `k3s-test-1-master-2` -1. `k3s-test-1-master-3` -1. `k3s-test-1-worker-1` -1. `k3s-test-1-worker-2` -1. `k3s-test-1-worker-3` +The guest's root is a tmpfs and `/nix/store` is the host's store shared read-only (virtiofs), so no +image has to be built or uploaded; only the volumes above persist. The guest's tap interface is +bridged onto `br0`, and its name is derived from the guest IP (e.g. `192.168.5.114` -> `vm114`, as +`IFNAMSIZ` caps interface names at 15 characters). + +To move an existing KubeVirt VM over in place, copy the host key and `machine-id` from its disk into +`etc.img`, and `/var/lib/rancher/k3s/server` into `var.img`, **before the first boot** — then it +keeps its identity and etcd membership instead of bootstraping a new cluster. + +1. `k3s-test-1-master-{1,2,3}` — running as microVMs +1. `k3s-test-1-worker-{1,2,3}` — to be re-added as microVMs ## TODO / Known issues +- **Retire KubeVirt.** aquamarine now runs natively on `kubevirt-youko`, and the k3s-test masters + run as microVMs. Remove KubeVirt/CDI/multus from the cluster (k8s-gitops) once nothing else needs + them, and the Windows VM is the only KubeVirt VM left to move. - **The USB HDD bridge is flaky.** aquamarine's two HDDs sit behind a JMicron JMS567 USB-SATA bridge that keeps resetting (`dmesg` on `kubevirt-youko`). Find out how often it resets and how much it matters before putting anything critical (e.g. an NFS export) on it. -- **Move aquamarine out of KubeVirt.** It is a "core" machine, so it should not depend on the - k8s/KubeVirt control plane; run it as a NixOS microVM on `kubevirt-youko` (`microvm.nix`) instead, - reusing its current disk. Consider consolidating the NFS server onto it afterwards (with a fast - and a slow tier). ## Kubernetes Resources @@ -44,5 +52,6 @@ Kubernetes resources are deployed and managed separately through [ryan4yin/k8s-gitops](https://github.com/ryan4yin/k8s-gitops). [k3s]: https://github.com/k3s-io/k3s/ +[microvm.nix]: https://github.com/microvm-nix/microvm.nix [what-have-k3s-removed-from-upstream-kubernetes]: https://github.com/k3s-io/k3s/?tab=readme-ov-file#what-have-you-removed-from-upstream-kubernetes