mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:46 +01:00
Ryan Yin
4470b3d014
Move standalone repository docs from the root into docs/ and add a documentation index that links the moved docs and the code-local runbooks. Update Markdown links, source comments, and the root entry points to the new locations. Keep README.md and AGENTS.md at the root as required entry points. Assisted-by: DeepSeek Harness (deepseek-flash)
77 lines
3.8 KiB
Markdown
77 lines
3.8 KiB
Markdown
# Kubernetes Clusters
|
|
|
|
I'm running a Kubernetes cluster for testing and development.
|
|
|
|
I prefer to use [k3s] as the Kubernetes distribution, because it's lightweight, easy to install, and
|
|
full featured(see [what-have-k3s-removed-from-upstream-kubernetes] for details).
|
|
|
|
## VM Cluster
|
|
|
|
The VM cluster is running on physical machines; all my virtual machines run on these hosts,
|
|
including the `k3s-test-1` cluster.
|
|
|
|
## K3s Clusters
|
|
|
|
The control plane runs as NixOS **microVMs** ([microvm.nix]) on the physical hosts. Each guest keeps
|
|
its state in three sparse ext4 images under the host's `/var/lib/microvms/<name>/`:
|
|
|
|
- `etc.img` (64M) — the ssh host key (also the agenix age identity) and `machine-id`
|
|
- `var.img` (20G) — k3s state (`/var/lib/rancher/k3s`) and the NixOS uid/gid maps
|
|
- `home.img` (4G) — the user's home
|
|
|
|
The guest's root is a tmpfs and `/nix/store` is the host's store shared read-only (virtiofs), so no
|
|
image has to be built or uploaded; only the volumes above persist. The guest's tap interface is
|
|
bridged onto `br0`, and its name is derived from the last octet of the guest IP by
|
|
`lib/genMicrovmGuestModule.nix` (e.g. `192.168.5.114` -> `vm114`).
|
|
|
|
### Updating a running guest
|
|
|
|
Deploy a guest with the repository's `just` recipe. It evaluates the guest locally, transfers the
|
|
complete Nix closure to the physical host, installs it under `/var/lib/microvms/<name>`, and
|
|
activates the guest:
|
|
|
|
```sh
|
|
just microvm-deploy <guest> <physical-host>
|
|
```
|
|
|
|
For example, `just microvm-deploy k3s-test-1-master-3 shushou`. The recipe copies the runner to the
|
|
physical host, then restarts `microvm@<guest>` there; microvm.nix's SSH switch is not used because
|
|
it cannot write the guest's read-only `/nix/store` (WA-026 in
|
|
[WORKAROUNDS.md](../../docs/WORKAROUNDS.md)). A later activation of the physical host re-points the
|
|
guest at the runner baked into the host's system, so switch the host after installing a guest from a
|
|
newer tree.
|
|
|
|
Use `just build-microvm <guest>` to build a runner without deploying it. Preserve the guest's
|
|
`etc.img`, `var.img`, and `home.img` when updating it; they contain its identity and K3s state.
|
|
|
|
For a cluster rollout, deploy one guest at a time. Before each restart, record the active runner and
|
|
verify that the new runner exists on the physical host. After each restart, wait for the guest Node
|
|
to become `Ready`, then check the API VIP `/readyz`, Cilium, storage CSI, and any workload health
|
|
gates. If a gate fails, restore the recorded runner through the same host access path and start the
|
|
guest service before proceeding to the next node.
|
|
|
|
1. `k3s-test-1-master-{1,2,3}` — control plane, running as microVMs; tainted
|
|
`node-role.kubernetes.io/control-plane:NoSchedule`
|
|
1. `k3s-test-1-worker-{1,2,3}` — workloads, running as microVMs
|
|
|
|
Placement: `worker-1` (4 vCPU / 16 GiB) runs on `shoryu`; `worker-2` (4 vCPU / 16 GiB) runs on
|
|
`youko`; `worker-3` (2 vCPU / 8 GiB) runs on `shushou`.
|
|
|
|
The master taint comes from kubelet's `registerWithTaints`, which only takes effect when the Node
|
|
object is first created. These masters are long-lived and predate the config, so the taint was
|
|
applied once by hand
|
|
(`kubectl taint nodes <name> node-role.kubernetes.io/control-plane=:NoSchedule`); a node rebuilt
|
|
from scratch gets it automatically. The workers carry no `node-role.kubernetes.io/worker` label:
|
|
kubelet refuses to self-assign the reserved `node-role.kubernetes.io/*` labels via `--node-label`,
|
|
and placement is enforced by the control-plane taint instead.
|
|
|
|
## Kubernetes Resources
|
|
|
|
Kubernetes resources are deployed and managed separately through
|
|
[ryan4yin/k8s-gitops](https://github.com/ryan4yin/k8s-gitops).
|
|
|
|
[k3s]: https://github.com/k3s-io/k3s/
|
|
[microvm.nix]: https://github.com/microvm-nix/microvm.nix
|
|
[what-have-k3s-removed-from-upstream-kubernetes]:
|
|
https://github.com/k3s-io/k3s/?tab=readme-ov-file#what-have-you-removed-from-upstream-kubernetes
|