The ISP retires delegated prefixes, so a guest that registers its WAN prefix
loses it and Cilium never refreshes the node address. Derive the identity from
the IPv4 octets inside fd05:5::/64 and trust that ULA in the guest firewall.
* fix(nfs): restrict the golden export to its real clients
* fix(libvirt): bind VNC consoles to loopback
* fix(firewall): drop VNC ports on every host
* docs(security): record the NFS and VNC boundaries
* fix(k3s): deny credential-bearing host ports to pod sources
* test: assert the k3s pod firewall drops sensitive ports
* docs(security): record the pod firewall boundary
The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
* fix(colmena): expose colmenaHive for the new evaluator
nixpkgs now ships the nix-community Colmena rewrite, whose default
evaluator reads an evaluated colmenaHive flake output instead of the raw
colmena hive. Without it every colmena apply recipe failed with
"does not provide attribute colmenaHive".
Add a colmena input pinned to the same v0.5.0 as nixpkgs and expose
colmenaHive = colmena.lib.makeHive <raw hive>, with the raw hive kept as
an internal binding rather than a flake output of its own.
* chore: clean up stale colmena and aquamarine references
- point the Colmena links at nix-community/colmena after the move
- rename the security-aquamarine-metrics test to youko-metrics (it
already tested youko) and update the outputs README
- drop the commented aquamarine remote-builder example
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.
- base: nftables backend; deny inbound by default; trust the home LAN by
source and the tailnet by interface (restrict the tailnet with ACLs);
suppress per-service `openFirewall` global ports; allow node_exporter only
from the monitoring host. The local podman bridge, the tailnet and the pod
CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
(caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
redundant desktop firewall module; aarch64/riscv64 import the firewall
explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
the LAN CIDR rule and no global ports.
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
cilium writes /etc/cni/net.d while istio-cni reads k3s's own dir, so on the new workers the chained plugin never found a network config. Add the symlinks the server module already has.
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:
- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
restic backup path must stay as it is)
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).
- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
their DNS name changes with the hostname and is not resolvable until
the router learns it
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:
- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
virtctl package
- remove lib/genKubeVirtHostModule.nix
lib/genMicrovmGuestModule.nix: qemu guest with a shared read-only store
(virtiofs), persistent /etc, /var and /home volumes, a tap on br0 (its
name derived from the guest IP, as IFNAMSIZ caps it at 15 chars), and a
deterministic MAC. Also keep lib/*.nix tracked.
services.earlyoom.freeMemThresholdPercent does not exist in this nixpkgs; the module exposes freeMemThreshold/freeSwapThreshold. Just enable earlyoom with its defaults (trigger at 10% free mem/swap).
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
ryan4yin | 二花
Ryan Yin