nix-config/docs/nixos-modernization-options.md

NixOS Modernization — Options and Decision Record

Purpose: let the maintainer decide, per candidate, whether it is worth adopting — by comparing it against what this repository does today. A plan derived from this file is only approved work once its card here says adopt. The plans are not on main: they exist only on the local, unpushed git tag modernization-plans-wip, under docs/plans/: 2026-10-07-boot-chain-limine-uki-repart.md, 2026-10-07-host-level-modernization.md and 2026-10-07-lab-experimental-etc-init.md (git show modernization-plans-wip:docs/plans/<file>).

Researched against nixpkgs nixos-unstable on 2026-10-07. Option names and upstream wording are quoted from the module sources cited in each card. The Current: sections were re-checked against this repository on 2026-10-21; the upstream research was not redone.

How to use this file

1. Read the card, not the plan.
2. Fill in Decision with one of:
   • adopt — replace the current mechanism; the matching plan task becomes approved work.
   • watch — not now; the card's Trigger to adopt says what would change the answer.
   • reject — decide against, with a one-line reason, so it is not re-litigated.
3. Recommendations below are mine; they are inputs, not conclusions.

Ordering rule used throughout: a candidate earns a plan task only if it replaces a mechanism this repository already runs or unlocks a concrete need. "It exists and is newer" is not enough.

---

A. Host services and networking

A1. Tailscale Serve instead of caddy vhosts

Current: one caddy instance on youko fronts 17 vhosts on *.writefor.fun by Host/SNI, with two TLS chains — a private ecc-ca (only repo machines trust it) and a lego/Cloudflare DNS-01 wildcard for phone/TV clients — plus file_server, header_up Host (SigV4), a wildcard 404 fallback, and a root-readable Cloudflare token in agenix.

Proposed: services.tailscale.serve for the services whose only clients are tailnet devices.

Pros: certificates become automatic and every tailnet device trusts them (today a phone without ecc-ca gets a certificate error); removes the Cloudflare token and the DNS-01 propagation workaround from those services; access control moves from "IP trust zone" to tailnet identity/ACL, which is exactly the limitation docs/SECURITY.md records; --tcp=<port> can also expose raw TCP.

Cons: Serve routes by mount path on one name, or (via the nixpkgs module) as Tailscale Services with tcp:<port> endpoints only — there is no Host/SNI vhost equivalent, so path-sensitive apps (gitea, rustfs) and base-url assumptions are a problem; rustfs S3 is SigV4-signed on the Host header and cannot move; file_server and the wildcard 404 have no equivalent; Android TV/Jellyfin/Immich native apps do not run Tailscale; LAN-local traffic would detour through WireGuard/DERP.

Reversibility: high — caddy config stays until each vhost is deleted deliberately.

Trigger to adopt: only for admin-only UIs (grafana, prometheus, alertmanager, vmalert, uptime-kuma, s3-console) and for LAN-only services whose clients are all repo machines.

Recommendation: partial adopt — migrate admin UIs only; keep caddy for immich, jellyfin, rustfs, file server, and the 404 fallback. The bigger win is retiring the private-CA chain if every LAN-only client can run Tailscale.

Decision: pending

A2. systemd-resolved

Current: services.resolved.enable evaluates true on 14 of the 15 NixOS hosts (all but akane): nixpkgs enables it by default with systemd-networkd (nixos/modules/system/boot/networkd.nix), and shoukei, which runs NetworkManager, enables it through modules.networking.mihomo.resolvedIntegration (asserted by the eval test outputs/aarch64-linux/tests/mihomo-dns). On ai, a mihomo lifecycle hook points the link's resolved DNS at mihomo (hosts/idols-ai/default.nix). DNSOverTLS and DNSSEC evaluate false. The services.resolved.settings.Resolve.MulticastDNS = false in modules/nixos/base/networking/mdns.nix is therefore live; mDNS itself is provided by avahi.

Proposed: DNSOverTLS/DNSSEC and caching settings on the resolved instance the hosts already run (services.resolved.enable = true would still be new on akane).

Pros: DoT/DNSSEC available; the caching resolver and a consistent stub across hosts already come with the resolved instance in use, and the mDNS caveat is already handled by the existing setting.

Cons: the LAN gateway suzi is a mihomo transparent proxy that hijacks DNS to fake IPs that are only routable through the proxy — DoT/DoH on a host would bypass that hijack and break proxied domains (the same reason transmission.nix deliberately resolves through real resolvers instead of the host resolver). Hosts running mihomo (idols-ai, shoukei) are the sensitive ones. Also interacts with the transmission network namespace and container DNS.

Reversibility: high (one option, one service restart).

Trigger to adopt: a host that has no dependency on suzi's DNS hijack, or after the proxy setup changes.

Recommendation: watch — the fake-IP dependency makes this riskier than it looks.

Decision: pending

A3. systemd-networkd instead of dhcpcd

Current: dhcpcd is not used anywhere in this repo. networking.useNetworkd = true is set on 13 of the 15 hosts: every VM host (lib/genVmHostModule.nix) and MicroVM guest (lib/genMicrovmGuestModule.nix), plus ai, aquamarine, kana and ruby; only shoukei and akane run NetworkManager. VM hosts and MicroVM guests use static config with useDHCP = false. The br0 bridge on VM hosts is declared through systemd.network.netdevs/networks already.

Proposed: systemd-networkd on the remaining NetworkManager hosts, through the options the other hosts already use: networking.useNetworkd and systemd.network.{networks,netdevs}. The networking.networkd.enable / networking.networks / networkdOptions names first recorded here do not exist in the locked nixpkgs.

Pros: declarative per-interface config, one implementation for stage 1 and stage 2, no dhcpcd anywhere in the path (nothing in this repo uses it), and VM hosts already speak systemd.network for the bridge.

Cons: bigger blast radius than any other item — a mistake on a br0 VM host cuts the guests and possibly remote reachability; migration touches every DHCP host; behaviour differences around lease/renew and DNS handling.

Reversibility: medium — reversible in config, but only if you can still reach the host.

Trigger to adopt: guest-only first, then a non-VM host with console access.

Recommendation: watch, guest-only experiment (lab-experimental-etc-init.md, Task 5). The corrected facts make that experiment moot — every MicroVM guest already runs networkd, and the plan task uses the non-existent option names above — so the open call is whether shoukei and akane move off NetworkManager.

Decision: pending

A4. Native journald shipping instead of per-host scraping only

Current: metrics are centralised (node_exporter on the LAN IP, smartctl exporter, VictoriaMetrics on youko); logs stay on each host.

Proposed: services.journald.upload on hosts → services.journald.remote on youko (both supported; services.journald.gateway exposes a read API).

Pros: first-class systemd path, no promtail/filebeat/GELF sidecar, structured fields survive, per-host logs searchable in one place.

Cons: new listening port and its firewall rule plus eval test; youko becomes the retention owner (disk, rotation); upload failures are silent unless monitored; TLS configuration is on you to get right.

Reversibility: high.

Trigger to adopt: when needing to correlate logs across hosts, or when the first host with a missing-log incident shows up.

Recommendation: adopt (low risk, clear operational win).

Decision: pending

A5. zram-generator instead of zramSwap

Current: a local module modules/nixos/base/zram.nix wraps zramSwap and adds sysctl tunings (swappiness, readahead), disabled on idols-ai (disk swapfile) and aquamarine, memoryPercent-tuned on shoukei, and not imported on akane. In the locked nixpkgs zramSwap is itself implemented on zram-generator (nixos/modules/config/zram.nix sets services.zram-generator.enable, which youko evaluates true).

Proposed: services.zram-generator.enable with a zram0 definition.

Pros: systemd-native unit generator, config expressed as data, upstream-maintained, no NixOS-specific script — all but writing the zram0 settings directly already come with zramSwap (see Current).

Cons: your module couples zram presence to sysctl tunings — the migration must keep that logic working when zram is off; two zram implementations are easy to have half-enabled during the change; the gain over a working zramSwap is mostly cosmetic.

Reversibility: high.

Trigger to adopt: whenever the zram module is next touched anyway.

Recommendation: watch — no current pain; fold into the next zram change.

Decision: pending

A6. nixos-facter instead of hand-written hardware-configuration.nix

Current: 4 hosts carry hand-written hardware-configuration.nix (shoryu, shoukei, ai, aquamarine; shushou and youko import shoryu's, and the qemu guests use the shared modules/nixos/server/qemu-guest-hardware-configuration.nix), plus deliberate per-host quirks such as the NVMe APST workaround WA-008 and USB-SATA mitigations WA-009.

Proposed: hardware.facter from nixpkgs (nixos-facter-modules upstreamed), report generated on the machine.

Pros: hardware changes (disk swap, new NIC, RAM) no longer require regenerating a file by hand; consistent with nixos-anywhere provisioning.

Cons: the report must live somewhere (repo or machine) and be regenerated on hardware change; defaults become less transparent than a file you can read in the diff; your deliberate deviations (kernel params, module options) still live in host config, so only part of the file is replaced.

Reversibility: high.

Trigger to adopt: on a new host build, or when a hardware change forces a regeneration anyway.

Recommendation: watch — adopt on the next new host rather than retrofitting four.

Decision: pending

A7. attic self-hosted binary cache

Current: substituters are the two Chinese mirrors plus numtide's cache.

Proposed: services.networking.atticd on youko.

Cons: another service to run, back up, and monitor on the host that already carries the most state; keys and tokens become a security surface (modules/base/nix.nix treats substituter trust as a boundary); it solves a problem (cache misses / CI egress) that is not currently reported as pain. Pros: control over retention, faster cross-host reuse, no third-party dependency for private artifacts.

Reversibility: high.

Trigger to adopt: measurably slow rebuilds across hosts, or a need to cache artifacts that cannot go to a public cache.

Recommendation: watch.

Decision: pending

A8. nix-servd and the non-root daemon features

Current: static nix.buildMachines (modules/nixos/base/remote-builders.nix); the plain user is deliberately not a trusted user on the clients, though the builder host trusts its builder login; Nix 2.35 via nixVersions.latest (WA-023). auto-allocate-uids (with cgroups) is already enabled in modules/nixos/base/nix.nix for nixpkgs' nspawn-based tests, and the locked nixpkgs has no nix-servd module or package.

Proposed: nix-servd for scheduling, plus the local-overlay-store experimental feature (auto-allocate-uids is already on, see Current).

Pros: dynamic builder selection and health checks instead of a hand-maintained list; the experimental features move toward running the daemon without root, consistent with the no-trusted-user policy; local-overlay-store can cut store duplication for remote builds.

Cons: experimental features in a repo whose value is reproducibility; nix-servd is young; misconfiguration shows up as flaky builds rather than a clean failure; benefit is unclear while the current remote build setup works.

Reversibility: medium (builds can be re-pointed, but half-migrated builders are confusing).

Trigger to adopt: when the static builder list starts causing real friction.

Recommendation: watch.

Decision: pending

A9. sched_ext (scx-loader)

Current: default kernel scheduler (EEVDF); desktop latency is not currently reported as a problem.

Proposed: services.scx-loader.enable with an scx scheduler (kernel 6.12+; the desktop runs a recent kernel).

Pros: a scheduler that can be swapped per workload, userspace-recoverable if it misbehaves, active upstream development.

Cons: another daemon; scheduler changes can produce subtle latency/throughput regressions that are hard to attribute; no measured problem to solve yet.

Reversibility: high (stop the service).

Trigger to adopt: a benchmark showing a win on the desktop, or a workload that needs it.

Recommendation: watch, easy to trial.

Decision: pending

A10. oo7, then gopass, behind the Secret portal

Current: gnome-keyring serves the Secret Service and the Secret portal Chromium uses for its v12 cookie key; its portal can only serve its own daemon.

Proposed: services.oo7.enable, whose oo7-portal serves any Secret Service daemon; then replace oo7-daemon with gopass secret-service, so gh's token and Chrome's cookie key need the CanoKey.

Pros: oo7 is a memory-safe, actively developed replacement on its own; with gopass, hardware-gated secrets from upstream components only.

Cons: migration deletes the gnome-keyring files. On 2026-11-09, oo7 0.6.0 hid gh's token (fixed in 0.7.0) and gave Chrome a new key, losing its v12 cookies (fixed after 0.7.0 by oo7#631, unreleased). gopass 1.17.4 skips undecryptable items, so a refused touch would mint a new Chrome key the same way.

Reversibility: low; needs a backup of ~/.local/share/keyrings under /persistent.

Trigger to adopt: nixpkgs ships oo7 newer than 0.7.0 with the portal app-ID fix (oo7#631; 0.7.0 fixes only gh). For gopass: a release after 1.17.4 whose SearchItems stops skipping undecryptable items (no upstream issue yet).

Recommendation: watch.

Decision: watch (2026-11-09)

---

B. Boot, images, VMs

B1. limine instead of systemd-boot + lanzaboote

Current: systemd-boot on akane, shoukei and the VM hosts (shoryu, shushou, youko); GRUB on /dev/vda on the libvirt guests kana and ruby; no bootloader on the MicroVM guests; on idols-ai and aquamarine, lanzaboote provides Secure Boot with a /etc/secureboot (lzpt) bundle. The lanzaboote flake input is maintained separately for Secure Boot.

Proposed: boot.loader.limine — in nixpkgs, with secureBoot, validateChecksums, panicOnChecksumMismatch, biosSupport, maxGenerations, graphical menu.

Pros: one bootloader for UEFI and BIOS (covers the qemu guests' GRUB case); Secure Boot without the lanzaboote flake input; checksum validation of kernel and initrd; nicer menu.

Cons: different trust model — it signs limine itself and validates artifacts by checksum, rather than shipping a firmware-verified UKI, so docs/SECURITY.md and the threat model must be updated; key enrollment is a manual firmware dance (Setup Mode + sbctl enroll-keys) and is the one change in this repository that can leave a machine unbootable; the existing lzpt bundle may or may not be reusable; less NixOS mileage than systemd-boot.

Reversibility: low on the physical desktop (firmware + ESP state), high on a VM.

Trigger to adopt: wanting one loader across BIOS/UEFI targets, or the lanzaboote pin becoming actual maintenance pain.

Recommendation: watch for the desktop, adopt on a disposable guest first — the value is real but it is the highest-risk item on this list.

Decision: pending

B2. boot.uki / system.build.uki

Current: lanzaboote already produces and signs a UKI on idols-ai and aquamarine; other hosts boot kernel+initrd.

Proposed: boot.uki.

Pros: standard UKI with ukify settings, single signed artifact, clear story for measured boot.

Cons: largely redundant where lanzaboote exists; adopting it interacts with B1 (who signs, who validates); no benefit for hosts without Secure Boot.

Reversibility: medium.

Recommendation: reject for now — revisit together with B1.

Decision: pending

B3. Declarative images with systemd-repart

Current: libvirt guest images are one-time reflink copies of the VMs' former KubeVirt disks (hosts/12kingdoms-shoryu/libvirtd.nix); k8s guests are MicroVMs booting declared state images against a read-only host store, with no OS image built at all.

Proposed: image.repart.enable + system.build.images.{qemu,raw-efi,raw,sd-card}.

Pros: images become a declared output of the system rather than a separate generator; GPT layout in repart.d; upstream direction; pairs with nixos-anywhere and vmspawn testing.

Cons: new subsystem with less mileage; the artifacts are consumed by ~/codes/k8s-gitops, so every change is a two-repo change; must prove boot and state-image compatibility before replacing anything.

Reversibility: high (additive while the old pipeline stays).

Trigger to adopt: next time the image pipeline needs a change anyway.

Recommendation: adopt as an additive experiment, not a replacement.

Decision: pending

B4. systemd-repart at boot (partition growth)

Current: manual partition sizing; no automatic growth.

Proposed: boot.initrd.systemd.repart.enable.

Pros: images and SD-card installs grow themselves; GPT-only and declarative.

Cons: another initrd-stage moving part; only useful once B3 or a disk-image workflow exists.

Reversibility: high.

Recommendation: watch — depends on B3.

Decision: pending

B5. systemd-sysupdate (A/B updates)

Current: generation switching through nixos-rebuild + bootloader entries — already atomic and rollbackable.

Proposed: systemd.sysupdate.enable with transfer definitions and a timer.

Pros: A/B image updates with signature verification.

Cons: duplicates functionality the generation model already provides; adds a second update path and its own state; unclear what problem it solves here.

Recommendation: reject.

Decision: pending

B6. Measured boot phases / FIDO2 credentials

Current: Secure Boot via lanzaboote on ai and aquamarine, whose LUKS volumes are enrolled for TPM2 unlock bound to PCRs 0+7, with the passphrase slot as fallback (hosts/idols-ai/README.md, hosts/idols-aquamarine/README.md). boot.initrd.systemd.tpm2.enable evaluates true on every host except the MicroVM guests, and on ai and aquamarine systemd.tpm2.enable and boot.initrd.systemd.fido2.enable are true as well (nixpkgs defaults from the systemd package). Not adopted: systemd.tpm2.pcrphases.enable is false, and the READMEs document no FIDO2 credential, only TPM2 and passphrase slots.

Proposed: systemd.tpm2.enable, systemd.tpm2.pcrphases.enable, boot.initrd.systemd.tpm2.enable, boot.initrd.systemd.fido2.enable.

Pros: PCR phases make boot-phase measurement usable for sealing policies; FIDO2 gives a physical second factor for LUKS.

Cons: PCR policies must be re-sealed whenever the boot chain changes (kernel updates, bootloader change B1), which converts an update into a maintenance event; CPU microcode/firmware updates can invalidate sealed state.

Reversibility: medium.

Recommendation: watch — revisit after B1 is settled.

Decision: pending

B7. systemd-vmspawn

Current: libvirt domains (generated XML) for qemu guests; MicroVMs for k8s guests.

Proposed: hand-written systemd.services calling systemd-vmspawn (--network-bridge=br0, --image-disk-type=, --coco=, vTPM).

Pros: throwaway VMs in one command; boots the B3 images directly; confidential-computing options; no libvirt stack.

Cons: no nixpkgs module and no .vmspawn unit type, so it is hand-written unit code; no declarative state-image management (MicroVM keeps {etc,var,home}.img and MAC-pinned NICs); libvirt stays for existing guests anyway.

Reversibility: high (delete the unit).

Recommendation: adopt narrowly — for testing B3 images only.

Decision: pending

---

C. Repository tooling (low risk, no service impact)

C1. Literal lint settings

Current: no lint-url-literals. Proposed: nix.settings.lint-url-literals = "warn" (plus the short/absolute-path variants). Pros: catches deprecated literal syntax before it becomes an error. Cons: warning noise on first run. Recommendation: adopt. Decision: pending

C2. nix-unit

Current: hand-rolled expr.nix/expected.nix pairs run by just test. Proposed: nix-unit alongside them. Pros: assertions on options, warnings, and thrown errors; clearer failures. Cons: a second test runner and a migration that can silently lose coverage if pairs are deleted early. Recommendation: adopt additively. Decision: pending

C3. nix-fast-build

Current: CI (.github/workflows/flake_evaltests.yml) runs one nix eval .#evalTests and one VM test (checks.x86_64-linux.security-exporters) on GitHub-hosted ubuntu-latest runners. Proposed: nix-fast-build over the eval tests and the three host systems; the remote builder (aquamarine, over SSH from repo machines) serves only a local Justfile run, not the hosted CI runners. Pros: parallel eval/build, no new CI infra. Cons: one more tool in the Justfile; CI already gates merges. Recommendation: adopt. Decision: pending

C5. jj

Personal workflow change only; no repository impact. Recommendation: out of scope. Decision: pending

---

D. Lab-only experimental group

The whole group shares one characteristic: upstream labels it experimental, and a mistake can prevent boot or login. Recommendation for the group: evaluate in a disposable guest (lab-experimental-etc-init.md on the plans tag), adopt individually only afterwards.

#	Candidate	Current	Main pro	Main con	Recommendation
D1	system.etc.overlay.enable	/etc generated by the perl activation script	/etc becomes an EROFS + overlay mount; foundation for an immutable /etc	upstream: "This is currently experimental. Only enable this option if you are confident that you can recover your system if it breaks."; every /etc writer must be audited	lab only
D2	systemd.sysusers.enable / services.userborn.enable	update-users-groups.pl	removes the perl script; prerequisite for an immutable /etc	experimental; uid/gid migration risk	lab only
D3	system.etc.overlay.mutable = false	writable /etc	truly immutable /etc	blocked: agenix writes /etc/agenix (see the environment.etc copies in secrets/nixos.nix), so secrets must move to /run or to sops-nix first	lab only, after secrets move
D4	system.nixos-init.enable	activationScripts	bashless init	makes activationScripts a silent no-op — youko's data-ro-backing, installCaddyWallpapers and installHomepageDashboardConfig would stop running without an error	lab only
D5	security.pam.services.<name>.rules	per-service booleans	freeform PAM config	upstream marks it subject to breaking changes; a mistake locks you out	lab only
D6	security.shadow.enable = false + greetd	shadow suite provides login/su	fewer setuid programs	assertion couples the two; changes VT login semantics	lab only
D7	security.apparmor.policies	policies.sudo already in use (hardening/apparmor/default.nix); the old profiles option is removed upstream	policy set is now data	writing profiles is real work and breakage-prone; docs/SECURITY.md explicitly warns against untested profiles	lab only

---

E. Considered and rejected (kept so they are not re-litigated)

Candidate	What it is	Why not here
systemd-sysext / confext	overlay read-only images onto /usr//etc of an immutable OS	solves the ostree/Flatcar problem NixOS does not have: generations are already atomic and rollbackable. Layering via sysext means something escaped the rebuild.
Portable services	attach/detach a service image at runtime	aimed at distributing services to foreign systemd distros; generation switching is strictly stronger here.
Quadlet	podman generator turning .container files into units	NixOS already has virtualisation.oci-containers (in use on servers); Quadlet would move unit definitions out of Nix evaluation. A downgrade, not a modernization.
Landlock	unprivileged per-process LSM (network rules since 6.12)	already an active default LSM (security/lsm includes it); the missing piece is per-application policy, and this repo's answer to that is nixpak/bwrap. Watch trigger: nixpak/bwrap gain network-restriction policy.
Incus	LXD fork, container + VM management plane	no bridge option in the module (it creates incusbr0; attaching to br0 needs preseed); it is a management plane, not a hypervisor replacement, and does not remove libvirt or MicroVM. If adopted, it needs its own disposable host.
Gateway API	k8s ingress successor	cross-repo: it belongs to ~/codes/k8s-gitops with a matching branch here only if host networking changes. ingress-nginx is EOL, so this is real work — just not this repo's plan.
bcachefs	copy-on-write filesystem, out of experimental upstream	current storage (btrfs + btrbk + restic) works; migration cost is high and unrelated to any pain. Watch only.

---

F. Decision summary

Fill this in as cards are decided; the plan documents (on the modernization-plans-wip tag, see the header) should then be trimmed to adopt items only.

Card	Candidate	Decision	Date	Note
A1	Tailscale Serve (partial)
A2	systemd-resolved
A3	systemd-networkd
A4	journald upload/remote
A5	zram-generator
A6	nixos-facter
A7	attic
A8	nix-servd + daemon features
A9	sched_ext
A10	oo7, then gopass	watch	2026-11-09	oo7 > 0.7.0 with oo7#631; gopass > 1.17.4 with a SearchItems fix
B1	limine + Secure Boot
B2	boot.uki
B3	systemd-repart images
B4	initrd repart
B5	sysupdate
B6	measured boot / FIDO2
B7	systemd-vmspawn
C1–C3, C5	tooling + cleanups
D1–D7	lab-only group
E	rejected set

References

• nixpkgs modules: system/etc/etc.nix, system/activation/nixos-init.nix, system/boot/loader/limine/limine.nix, system/boot/systemd/{repart,sysupdate,tpm2,fido2,journald-*}.nix, image/{images,repart}.nix, services/networking/{tailscale-serve,dhcpcd,nat-nftables}.nix, services/system/{userborn,nix-daemon,zram-generator}.nix, services/monitoring/prometheus/exporters.nix.
• NixOS 26.05 release notes; Nix manual, Experimental Features.
• Repo context: docs/SECURITY.md (trust zones and control limits), docs/WORKAROUNDS.md (WA-001, WA-008, WA-009, WA-023), modules/nixos/base/networking/mdns.nix, hosts/12kingdoms-youko/homelab-services/caddy.nix, hosts/12kingdoms-youko/homelab-services/README.md.