myNurPkgs evaluated a second nixpkgs just for four packages. Apply the
NUR repo's own overlay instead, so they come from the host's pkgs.
Darwin's pkgs now gets nixpkgsConfig too: it set nixpkgs.pkgs, which
ignores nixpkgs.config, so only myNurPkgs carried the npm mirror there.
Assisted-by: Claude Code (claude-opus-5-5)
Apply hdparm -B 255 -S 0 whenever a disk appears, which also covers the
re-add after a bridge reset, instead of a boot oneshot plus an hourly
timer. Drop -J 0: hdparm has always aborted it without
--please-destroy-my-drive, and wdidle3 already reads disabled on both
disks (it is persistent firmware state).
This was the last user of mylib.systemdHardening.privilegedOneshot, so
remove the preset and its test group.
Assisted-by: Claude Code (claude-opus-5-5)
The homelab hosts accept root SSH with long-lived file keys, run several
exposed services with almost no systemd sandboxing, and keep no audit trail of
privileged actions. Close those three gaps, weighing each measure against its
cost instead of adopting a policy wholesale.
SSH
- Turn off KbdInteractiveAuthentication: nixpkgs defaults it to true and PAM
then still accepts the Unix password despite PasswordAuthentication = false.
- authorizedKeysInHomedir = false: only keys declared in Nix take effect;
youko keeps Gitea's forced-command file through `Match User gitea`.
- Pin aquamarine's passphrase-free remote-builder keys to
`restrict,command="nix-daemon --stdio --store daemon"`, so builds run in the
system daemon instead of an audited root login session.
systemd sandboxing
- Add the mylib.systemdHardening preset and apply it to the exposed, weakly
sandboxed services: caddy 7.8 -> 1.5, vmalert 8.2 -> 1.4, rustfs 6.6 -> 1.4,
whisper-asr 7.7 -> 1.6. An eval test pins the key restrictions.
- Rejected: a global all-keys policy (<=0.2 score gains on services nixpkgs
already sandboxes) and per-service confinement (bind paths to maintain).
auditd on all four hosts, reusing nixpkgs' loader and unit
- Audit login sessions only: a first `-a never,task -F auid=unset` rule keeps
services out of the per-syscall audit path. A compromised service is
therefore unrecorded until it opens a session.
- Record root commands, credential and host-key access, writes to SSH, account,
PAM, Nix-trust and audit configuration, module loads and session denials.
- Rejected: global denial logging, service-UID watches and
--loginuid-immutable (the kernel already records loginuid changes).
Verified on all four hosts after deployment: zero failed units, k3s 6/6 Ready,
and the exposed services healthy.
Assisted-by: Claude Code (claude-opus-5-5)
The etcd client API listens on the masters' static LAN address, and the
pod-source deny lists omitted it. Add 2379 to both nft drop lines and
assert it in the security-k3s-pod-firewall eval test. 2381 (etcd
metrics, scraped by in-cluster monitoring) stays reachable by design.
Pairs with the matching Cilium egressDeny port in k8s-gitops.
Assisted-by: DeepSeek Harness (gpt-6-astra)
Colmena deployments now ship the same toplevel as nixosConfigurations: nodes gain the nixpkgs
registry, nixPath and dated version, and akane no longer gets the x86_64 dsh.
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (claude-opus-5-5)
The ISP retires delegated prefixes, so a guest that registers its WAN prefix
loses it and Cilium never refreshes the node address. Derive the identity from
the IPv4 octets inside fd05:5::/64 and trust that ULA in the guest firewall.
* fix(nfs): restrict the golden export to its real clients
* fix(libvirt): bind VNC consoles to loopback
* fix(firewall): drop VNC ports on every host
* docs(security): record the NFS and VNC boundaries
* fix(k3s): deny credential-bearing host ports to pod sources
* test: assert the k3s pod firewall drops sensitive ports
* docs(security): record the pod firewall boundary
The Telegram channel was dominated by alerts that can never be actionable:
- DriveTemperatureHigh/Critical did not filter temperature_type, so
max(smartctl_device_temperature) picked the drive's rated
op_limit_max/critical_limit_max instead of the actual reading. ai reported
95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
hwmon sensor labels itself Tctl, so the CPU control temperature (up to
~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
six k3s guests. Disable automatic GC in the guest; the host's nix.gc
reclaims the shared store.
The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
* fix(colmena): expose colmenaHive for the new evaluator
nixpkgs now ships the nix-community Colmena rewrite, whose default
evaluator reads an evaluated colmenaHive flake output instead of the raw
colmena hive. Without it every colmena apply recipe failed with
"does not provide attribute colmenaHive".
Add a colmena input pinned to the same v0.5.0 as nixpkgs and expose
colmenaHive = colmena.lib.makeHive <raw hive>, with the raw hive kept as
an internal binding rather than a flake output of its own.
* chore: clean up stale colmena and aquamarine references
- point the Colmena links at nix-community/colmena after the move
- rename the security-aquamarine-metrics test to youko-metrics (it
already tested youko) and update the outputs README
- drop the commented aquamarine remote-builder example
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.
- base: nftables backend; deny inbound by default; trust the home LAN by
source and the tailnet by interface (restrict the tailnet with ACLs);
suppress per-service `openFirewall` global ports; allow node_exporter only
from the monitoring host. The local podman bridge, the tailnet and the pod
CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
(caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
redundant desktop firewall module; aarch64/riscv64 import the firewall
explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
the LAN CIDR rule and no global ports.
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
cilium writes /etc/cni/net.d while istio-cni reads k3s's own dir, so on the new workers the chained plugin never found a network config. Add the symlinks the server module already has.
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:
- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
restic backup path must stay as it is)
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).
- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
their DNS name changes with the hostname and is not resolvable until
the router learns it
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:
- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
virtctl package
- remove lib/genKubeVirtHostModule.nix
lib/genMicrovmGuestModule.nix: qemu guest with a shared read-only store
(virtiofs), persistent /etc, /var and /home volumes, a tap on br0 (its
name derived from the guest IP, as IFNAMSIZ caps it at 15 chars), and a
deterministic MAC. Also keep lib/*.nix tracked.
services.earlyoom.freeMemThresholdPercent does not exist in this nixpkgs; the module exposes freeMemThreshold/freeSwapThreshold. Just enable earlyoom with its defaults (trigger at 10% free mem/swap).
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
ryan4yin | 二花
Ryan Yin