* refactor(youko): let preservation own the service state dirs
The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:
- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind
Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.
* refactor(youko): drop caddy's redundant tmpfiles rules
The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.
* refactor(youko): use a podman named volume for uptime-kuma
podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
The services now run directly on the host, so their state does not belong
on the HDD behind the flaky USB-SATA bridge: /data/apps was small-file
heavy and latency/consistency sensitive. Point every dataDir at
/persistent/apps (the LUKS nvme) instead.
The HDD keeps the bulk data that suits it: /data/fileshare (173G) and
/data/backups (24G). The HDD's old @apps subvolume and its data are left
in place for now.
-B 255 turns APM off and it stuck, but the disks still reach standby, so
the bridge firmware is likely doing the spindown. -J 0 additionally asks
the drive to stop parking its heads (sdb has ~1M load cycles).
pcie_aspm=off was the least-justified mitigation (the flakiness is on the
USB/SATA side) and costs power. Also log the disks' APM level/state and
re-apply the settings hourly, since the bridge or the disk firmware can
drop them (the disks were still reaching standby).
The two 4TB HDDs sit behind a JMicron JMS567 that resets and throws
UDMA-CRC / I/O errors (the disks themselves are healthy). Best-effort
mitigations while the enclosure stays in use:
- kernel: usbcore.autosuspend=-1, usb-storage.delay_use=10, pcie_aspm=off
- udev: keep the bridge's power/control=on (never autosuspend)
- disable APM/standby on the HDDs (spindown is a common reset trigger)
- enable smartd (watch the CRC count and disk health)
The kernel parameters need a reboot to take effect.
It documented the retired KubeVirt workflow (upload a qcow2, create the vm
in k8s-gitops, let Flux create it), which no longer exists. The general
deployment notes live in the README and the per-platform docs
(hosts/k8s/README.md for the microVMs).
Update the user-facing bits that still pointed at KubeVirt: the homepage
dashboard group and entries (and drop the dead LongHorn one), the docs,
and remove the now-unneeded virtctl package.
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:
- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
restic backup path must stay as it is)
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).
- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
their DNS name changes with the hostname and is not resolvable until
the router learns it
This is not a Python package, so drop the packaging/build/pyinstaller/
framework/type-checker/pyenv/poetry/pdm/pixi/celery/rabbitmq rules and the
unused Marimo and Streamlit sections. Keep the bits occasional Python
scripts do need: byte-compiled files, virtualenvs, local secrets and the
common tool caches.
The Python section of the .gitignore template has a bare 'lib/' (and
'lib64/'), which also matched the Nix helper modules in lib/. The
'!lib/*.nix' line added earlier could not help, since a file cannot be
re-included while its parent directory is excluded.
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:
- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
virtctl package
- remove lib/genKubeVirtHostModule.nix
The file is not KubeVirt-specific: it is the hardware configuration for
any QEMU guest that boots from a disk (KubeVirt or libvirt). All it does
is pull in qemu-guest.nix and set fileSystems."/" from the disk labeled
nixos.
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.
- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works
lib/genMicrovmGuestModule.nix: qemu guest with a shared read-only store
(virtiofs), persistent /etc, /var and /home volumes, a tap on br0 (its
name derived from the guest IP, as IFNAMSIZ caps it at 15 chars), and a
deterministic MAC. Also keep lib/*.nix tracked.
The service modules moved two levels deeper, so their certs/ paths
resolved relative to the wrong directory. Use mylib.relativeToRoot so
they no longer depend on the module's depth.
Drop the retired host's name from the filenames:
aquamarine-services.nix -> homelab-services.nix and the aquamarine/
module dir -> homelab-services/. The "aqua" name now stays only in the
docs/vars, where it is kept for a possible later test microvm/qemu-vm on
idols-ai.
Aquamarine's services now run directly on kubevirt-youko, so the guest
definition is no longer needed:
- move its service modules to hosts/k8s/kubevirt-youko/aquamarine/ and
drop the explicit import list, leaving aquamarine-services.nix with
just its IP and secret groups
- delete the guest base, journald override, output entry and home config
- repoint the eval tests that read the guest's config to kubevirt-youko
- drop the now-dead 'aqua' colmena recipe and its image upload
The aqua KubeVirt VM itself is deleted separately; the name is kept in
the docs for a possible later test microvm/qemu-vm on idols-ai.
The VMs now declare their root disk as a DataVolume with `source: upload`, so
upload straight into `<vm>-disk` (CDI only accepts an upload into a DataVolume)
instead of a shared golden PVC. Takes the disk size as an argument.
* feat(k3s-test-1): add three worker nodes and taint the control plane
- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too
* chore(k3s-test-1): move the workers to .111-.113
The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.
- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
* feat(kubevirt-youko): export a shared golden-image store over NFS
* feat(vm-images): upload golden images into the cluster
Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
services.earlyoom.freeMemThresholdPercent does not exist in this nixpkgs; the module exposes freeMemThreshold/freeSwapThreshold. Just enable earlyoom with its defaults (trigger at 10% free mem/swap).
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
ryan4yin | 二花
ryan4yin | 二花