30 KiB
NixOS Modernization — Options and Decision Record
Purpose: let the maintainer decide, per candidate, whether it is worth adopting — by comparing it against what this repository does today. A plan derived from this file is only approved work once its card here says adopt. The plans are not on
main: they exist only on the local, unpushed git tagmodernization-plans-wip, underdocs/plans/:2026-10-07-boot-chain-limine-uki-repart.md,2026-10-07-host-level-modernization.mdand2026-10-07-lab-experimental-etc-init.md(git show modernization-plans-wip:docs/plans/<file>).Researched against nixpkgs
nixos-unstableon 2026-10-07. Option names and upstream wording are quoted from the module sources cited in each card. The Current: sections were re-checked against this repository on 2026-10-21; the upstream research was not redone.
How to use this file
- Read the card, not the plan.
- Fill in Decision with one of:
- adopt — replace the current mechanism; the matching plan task becomes approved work.
- watch — not now; the card's Trigger to adopt says what would change the answer.
- reject — decide against, with a one-line reason, so it is not re-litigated.
- Recommendations below are mine; they are inputs, not conclusions.
Ordering rule used throughout: a candidate earns a plan task only if it replaces a mechanism this repository already runs or unlocks a concrete need. "It exists and is newer" is not enough.
A. Host services and networking
A1. Tailscale Serve instead of caddy vhosts
Current: one caddy instance on youko fronts 17 vhosts on *.writefor.fun by Host/SNI, with
two TLS chains — a private ecc-ca (only repo machines trust it) and a lego/Cloudflare DNS-01
wildcard for phone/TV clients — plus file_server, header_up Host (SigV4), a wildcard 404
fallback, and a root-readable Cloudflare token in agenix.
Proposed: services.tailscale.serve for the services whose only clients are tailnet devices.
Pros: certificates become automatic and every tailnet device trusts them (today a phone without
ecc-ca gets a certificate error); removes the Cloudflare token and the DNS-01 propagation
workaround from those services; access control moves from "IP trust zone" to tailnet identity/ACL,
which is exactly the limitation SECURITY.md records; --tcp=<port> can also expose raw TCP.
Cons: Serve routes by mount path on one name, or (via the nixpkgs module) as Tailscale Services
with tcp:<port> endpoints only — there is no Host/SNI vhost equivalent, so path-sensitive apps
(gitea, rustfs) and base-url assumptions are a problem; rustfs S3 is SigV4-signed on the Host header
and cannot move; file_server and the wildcard 404 have no equivalent; Android TV/Jellyfin/Immich
native apps do not run Tailscale; LAN-local traffic would detour through WireGuard/DERP.
Reversibility: high — caddy config stays until each vhost is deleted deliberately.
Trigger to adopt: only for admin-only UIs (grafana, prometheus, alertmanager, vmalert, uptime-kuma, s3-console) and for LAN-only services whose clients are all repo machines.
Recommendation: partial adopt — migrate admin UIs only; keep caddy for immich, jellyfin, rustfs, file server, and the 404 fallback. The bigger win is retiring the private-CA chain if every LAN-only client can run Tailscale.
Decision: pending
A2. systemd-resolved
Current: services.resolved.enable evaluates true on 14 of the 15 NixOS hosts (all but
akane): nixpkgs enables it by default with systemd-networkd
(nixos/modules/system/boot/networkd.nix), and shoukei, which runs NetworkManager, enables it
through modules.networking.mihomo.resolvedIntegration (asserted by the eval test
outputs/aarch64-linux/tests/mihomo-dns). On ai, a mihomo lifecycle hook points the link's
resolved DNS at mihomo (hosts/idols-ai/default.nix). DNSOverTLS and DNSSEC evaluate false.
The services.resolved.settings.Resolve.MulticastDNS = false in
modules/nixos/base/networking/mdns.nix is therefore live; mDNS itself is provided by avahi.
Proposed: DNSOverTLS/DNSSEC and caching settings on the resolved instance the hosts already
run (services.resolved.enable = true would still be new on akane).
Pros: DoT/DNSSEC available; the caching resolver and a consistent stub across hosts already come with the resolved instance in use, and the mDNS caveat is already handled by the existing setting.
Cons: the LAN gateway suzi is a mihomo transparent proxy that hijacks DNS to fake IPs that
are only routable through the proxy — DoT/DoH on a host would bypass that hijack and break proxied
domains (the same reason transmission.nix deliberately resolves through real resolvers instead of
the host resolver). Hosts running mihomo (idols-ai, shoukei) are the sensitive ones. Also interacts
with the transmission network namespace and container DNS.
Reversibility: high (one option, one service restart).
Trigger to adopt: a host that has no dependency on suzi's DNS hijack, or after the proxy setup changes.
Recommendation: watch — the fake-IP dependency makes this riskier than it looks.
Decision: pending
A3. systemd-networkd instead of dhcpcd
Current: dhcpcd is not used anywhere in this repo. networking.useNetworkd = true is set on 13
of the 15 hosts: every VM host (lib/genVmHostModule.nix) and MicroVM guest
(lib/genMicrovmGuestModule.nix), plus ai, aquamarine, kana and ruby; only shoukei and
akane run NetworkManager. VM hosts and MicroVM guests use static config with useDHCP = false.
The br0 bridge on VM hosts is declared through systemd.network.netdevs/networks already.
Proposed: systemd-networkd on the remaining NetworkManager hosts, through the options the other
hosts already use: networking.useNetworkd and systemd.network.{networks,netdevs}. The
networking.networkd.enable / networking.networks / networkdOptions names first recorded here
do not exist in the locked nixpkgs.
Pros: declarative per-interface config, one implementation for stage 1 and stage 2, no dhcpcd
anywhere in the path (nothing in this repo uses it), and VM hosts already speak systemd.network
for the bridge.
Cons: bigger blast radius than any other item — a mistake on a br0 VM host cuts the guests and
possibly remote reachability; migration touches every DHCP host; behaviour differences around
lease/renew and DNS handling.
Reversibility: medium — reversible in config, but only if you can still reach the host.
Trigger to adopt: guest-only first, then a non-VM host with console access.
Recommendation: watch, guest-only experiment (lab-experimental-etc-init.md, Task 5). The
corrected facts make that experiment moot — every MicroVM guest already runs networkd, and the plan
task uses the non-existent option names above — so the open call is whether shoukei and akane
move off NetworkManager.
Decision: pending
A4. Native journald shipping instead of per-host scraping only
Current: metrics are centralised (node_exporter on the LAN IP, smartctl exporter, VictoriaMetrics on youko); logs stay on each host.
Proposed: services.journald.upload on hosts → services.journald.remote on youko (both
supported; services.journald.gateway exposes a read API).
Pros: first-class systemd path, no promtail/filebeat/GELF sidecar, structured fields survive, per-host logs searchable in one place.
Cons: new listening port and its firewall rule plus eval test; youko becomes the retention owner (disk, rotation); upload failures are silent unless monitored; TLS configuration is on you to get right.
Reversibility: high.
Trigger to adopt: when needing to correlate logs across hosts, or when the first host with a missing-log incident shows up.
Recommendation: adopt (low risk, clear operational win).
Decision: pending
A5. zram-generator instead of zramSwap
Current: a local module modules/nixos/base/zram.nix wraps zramSwap and adds sysctl tunings
(swappiness, readahead), disabled on idols-ai (disk swapfile) and aquamarine, memoryPercent-tuned on
shoukei, and not imported on akane. In the locked nixpkgs zramSwap is itself implemented on
zram-generator (nixos/modules/config/zram.nix sets services.zram-generator.enable, which youko
evaluates true).
Proposed: services.zram-generator.enable with a zram0 definition.
Pros: systemd-native unit generator, config expressed as data, upstream-maintained, no
NixOS-specific script — all but writing the zram0 settings directly already come with zramSwap
(see Current).
Cons: your module couples zram presence to sysctl tunings — the migration must keep that logic
working when zram is off; two zram implementations are easy to have half-enabled during the change;
the gain over a working zramSwap is mostly cosmetic.
Reversibility: high.
Trigger to adopt: whenever the zram module is next touched anyway.
Recommendation: watch — no current pain; fold into the next zram change.
Decision: pending
A6. nixos-facter instead of hand-written hardware-configuration.nix
Current: 4 hosts carry hand-written hardware-configuration.nix (shoryu, shoukei, ai,
aquamarine; shushou and youko import shoryu's, and the qemu guests use the shared
modules/nixos/server/qemu-guest-hardware-configuration.nix), plus deliberate per-host quirks such
as the NVMe APST workaround WA-008 and USB-SATA mitigations WA-009.
Proposed: hardware.facter from nixpkgs (nixos-facter-modules upstreamed), report generated on
the machine.
Pros: hardware changes (disk swap, new NIC, RAM) no longer require regenerating a file by hand;
consistent with nixos-anywhere provisioning.
Cons: the report must live somewhere (repo or machine) and be regenerated on hardware change; defaults become less transparent than a file you can read in the diff; your deliberate deviations (kernel params, module options) still live in host config, so only part of the file is replaced.
Reversibility: high.
Trigger to adopt: on a new host build, or when a hardware change forces a regeneration anyway.
Recommendation: watch — adopt on the next new host rather than retrofitting four.
Decision: pending
A7. attic self-hosted binary cache
Current: substituters are the two Chinese mirrors plus numtide's cache.
Proposed: services.networking.atticd on youko.
Cons: another service to run, back up, and monitor on the host that already carries the most
state; keys and tokens become a security surface (modules/base/nix.nix treats substituter trust as
a boundary); it solves a problem (cache misses / CI egress) that is not currently reported as pain.
Pros: control over retention, faster cross-host reuse, no third-party dependency for private
artifacts.
Reversibility: high.
Trigger to adopt: measurably slow rebuilds across hosts, or a need to cache artifacts that cannot go to a public cache.
Recommendation: watch.
Decision: pending
A8. nix-servd and the non-root daemon features
Current: static nix.buildMachines (modules/nixos/base/remote-builders.nix); the plain user
is deliberately not a trusted user on the clients, though the builder host trusts its builder login;
Nix 2.35 via nixVersions.latest (WA-023). auto-allocate-uids (with cgroups) is already enabled
in modules/nixos/base/nix.nix for nixpkgs' nspawn-based tests, and the locked nixpkgs has no
nix-servd module or package.
Proposed: nix-servd for scheduling, plus the local-overlay-store experimental feature
(auto-allocate-uids is already on, see Current).
Pros: dynamic builder selection and health checks instead of a hand-maintained list; the
experimental features move toward running the daemon without root, consistent with the
no-trusted-user policy; local-overlay-store can cut store duplication for remote builds.
Cons: experimental features in a repo whose value is reproducibility; nix-servd is young;
misconfiguration shows up as flaky builds rather than a clean failure; benefit is unclear while the
current three-arch remote build setup works.
Reversibility: medium (builds can be re-pointed, but half-migrated builders are confusing).
Trigger to adopt: when the static builder list starts causing real friction.
Recommendation: watch.
Decision: pending
A9. sched_ext (scx-loader)
Current: default kernel scheduler (EEVDF); desktop latency is not currently reported as a problem.
Proposed: services.scx-loader.enable with an scx scheduler (kernel 6.12+; the desktop runs a
recent kernel).
Pros: a scheduler that can be swapped per workload, userspace-recoverable if it misbehaves, active upstream development.
Cons: another daemon; scheduler changes can produce subtle latency/throughput regressions that are hard to attribute; no measured problem to solve yet.
Reversibility: high (stop the service).
Trigger to adopt: a benchmark showing a win on the desktop, or a workload that needs it.
Recommendation: watch, easy to trial.
Decision: pending
B. Boot, images, VMs
B1. limine instead of systemd-boot + lanzaboote
Current: systemd-boot on akane, shoukei and the VM hosts (shoryu, shushou, youko); GRUB on
/dev/vda on the libvirt guests kana and ruby; no bootloader on the MicroVM guests; on idols-ai and
aquamarine, lanzaboote provides Secure Boot with a /etc/secureboot (lzpt) bundle. The lanzaboote
flake input is maintained separately for Secure Boot.
Proposed: boot.loader.limine — in nixpkgs, with secureBoot, validateChecksums,
panicOnChecksumMismatch, biosSupport, maxGenerations, graphical menu.
Pros: one bootloader for UEFI and BIOS (covers the qemu guests' GRUB case); Secure Boot without the lanzaboote flake input; checksum validation of kernel and initrd; nicer menu.
Cons: different trust model — it signs limine itself and validates artifacts by checksum,
rather than shipping a firmware-verified UKI, so SECURITY.md and the threat model must be updated;
key enrollment is a manual firmware dance (Setup Mode + sbctl enroll-keys) and is the one change
in this repository that can leave a machine unbootable; the existing lzpt bundle may or may not be
reusable; less NixOS mileage than systemd-boot.
Reversibility: low on the physical desktop (firmware + ESP state), high on a VM.
Trigger to adopt: wanting one loader across BIOS/UEFI targets, or the lanzaboote pin becoming actual maintenance pain.
Recommendation: watch for the desktop, adopt on a disposable guest first — the value is real but it is the highest-risk item on this list.
Decision: pending
B2. boot.uki / system.build.uki
Current: lanzaboote already produces and signs a UKI on idols-ai and aquamarine; other hosts boot kernel+initrd.
Proposed: boot.uki.
Pros: standard UKI with ukify settings, single signed artifact, clear story for measured boot.
Cons: largely redundant where lanzaboote exists; adopting it interacts with B1 (who signs, who validates); no benefit for hosts without Secure Boot.
Reversibility: medium.
Recommendation: reject for now — revisit together with B1.
Decision: pending
B3. Declarative images with systemd-repart
Current: libvirt guest images are one-time reflink copies of the VMs' former KubeVirt disks
(hosts/12kingdoms-shoryu/libvirtd.nix); k8s guests are MicroVMs booting declared state images
against a read-only host store, with no OS image built at all.
Proposed: image.repart.enable + system.build.images.{qemu,raw-efi,raw,sd-card}.
Pros: images become a declared output of the system rather than a separate generator; GPT layout
in repart.d; upstream direction; pairs with nixos-anywhere and vmspawn testing.
Cons: new subsystem with less mileage; the artifacts are consumed by ~/codes/k8s-gitops, so
every change is a two-repo change; must prove boot and state-image compatibility before replacing
anything.
Reversibility: high (additive while the old pipeline stays).
Trigger to adopt: next time the image pipeline needs a change anyway.
Recommendation: adopt as an additive experiment, not a replacement.
Decision: pending
B4. systemd-repart at boot (partition growth)
Current: manual partition sizing; no automatic growth.
Proposed: boot.initrd.systemd.repart.enable.
Pros: images and SD-card installs grow themselves; GPT-only and declarative.
Cons: another initrd-stage moving part; only useful once B3 or a disk-image workflow exists.
Reversibility: high.
Recommendation: watch — depends on B3.
Decision: pending
B5. systemd-sysupdate (A/B updates)
Current: generation switching through nixos-rebuild + bootloader entries — already atomic and
rollbackable.
Proposed: systemd.sysupdate.enable with transfer definitions and a timer.
Pros: A/B image updates with signature verification.
Cons: duplicates functionality the generation model already provides; adds a second update path and its own state; unclear what problem it solves here.
Recommendation: reject.
Decision: pending
B6. Measured boot phases / FIDO2 credentials
Current: Secure Boot via lanzaboote on ai and aquamarine, whose LUKS volumes are enrolled for
TPM2 unlock bound to PCRs 0+7, with the passphrase slot as fallback (hosts/idols-ai/README.md,
hosts/idols-aquamarine/README.md). boot.initrd.systemd.tpm2.enable evaluates true on every
host except the MicroVM guests, and on ai and aquamarine systemd.tpm2.enable and
boot.initrd.systemd.fido2.enable are true as well (nixpkgs defaults from the systemd package).
Not adopted: systemd.tpm2.pcrphases.enable is false, and the READMEs document no FIDO2
credential, only TPM2 and passphrase slots.
Proposed: systemd.tpm2.enable, systemd.tpm2.pcrphases.enable,
boot.initrd.systemd.tpm2.enable, boot.initrd.systemd.fido2.enable.
Pros: PCR phases make boot-phase measurement usable for sealing policies; FIDO2 gives a physical second factor for LUKS.
Cons: PCR policies must be re-sealed whenever the boot chain changes (kernel updates, bootloader change B1), which converts an update into a maintenance event; CPU microcode/firmware updates can invalidate sealed state.
Reversibility: medium.
Recommendation: watch — revisit after B1 is settled.
Decision: pending
B7. systemd-vmspawn
Current: libvirt domains (generated XML) for qemu guests; MicroVMs for k8s guests.
Proposed: hand-written systemd.services calling systemd-vmspawn (--network-bridge=br0,
--image-disk-type=, --coco=, vTPM).
Pros: throwaway VMs in one command; boots the B3 images directly; confidential-computing options; no libvirt stack.
Cons: no nixpkgs module and no .vmspawn unit type, so it is hand-written unit code; no
declarative state-image management (MicroVM keeps {etc,var,home}.img and MAC-pinned NICs); libvirt
stays for existing guests anyway.
Reversibility: high (delete the unit).
Recommendation: adopt narrowly — for testing B3 images only.
Decision: pending
C. Repository tooling (low risk, no service impact)
C1. Literal lint settings
Current: no lint-url-literals. Proposed: nix.settings.lint-url-literals = "warn" (plus
the short/absolute-path variants). Pros: catches deprecated literal syntax before it becomes an
error. Cons: warning noise on first run. Recommendation: adopt. Decision: pending
C2. nix-unit
Current: hand-rolled expr.nix/expected.nix pairs run by just test. Proposed:
nix-unit alongside them. Pros: assertions on options, warnings, and thrown errors; clearer
failures. Cons: a second test runner and a migration that can silently lose coverage if pairs
are deleted early. Recommendation: adopt additively. Decision: pending
C3. nix-fast-build
Current: CI (.github/workflows/flake_evaltests.yml) runs one nix eval .#evalTests and one VM
test (checks.x86_64-linux.security-exporters) on GitHub-hosted ubuntu-latest runners.
Proposed: nix-fast-build over the eval tests and the three host systems; the remote builder
(aquamarine, over SSH from repo machines) serves only a local Justfile run, not the hosted CI
runners. Pros: parallel eval/build, no new CI infra. Cons: one more tool in the Justfile; CI
already gates merges. Recommendation: adopt. Decision: pending
C5. jj
Personal workflow change only; no repository impact. Recommendation: out of scope. Decision: pending
D. Lab-only experimental group
The whole group shares one characteristic: upstream labels it experimental, and a mistake can
prevent boot or login. Recommendation for the group: evaluate in a disposable guest
(lab-experimental-etc-init.md on the plans tag), adopt individually only afterwards.
| # | Candidate | Current | Main pro | Main con | Recommendation |
|---|---|---|---|---|---|
| D1 | system.etc.overlay.enable |
/etc generated by the perl activation script | /etc becomes an EROFS + overlay mount; foundation for an immutable /etc | upstream: "This is currently experimental. Only enable this option if you are confident that you can recover your system if it breaks."; every /etc writer must be audited | lab only |
| D2 | systemd.sysusers.enable / services.userborn.enable |
update-users-groups.pl |
removes the perl script; prerequisite for an immutable /etc | experimental; uid/gid migration risk | lab only |
| D3 | system.etc.overlay.mutable = false |
writable /etc | truly immutable /etc | blocked: agenix writes /etc/agenix (see the environment.etc copies in secrets/nixos.nix), so secrets must move to /run or to sops-nix first |
lab only, after secrets move |
| D4 | system.nixos-init.enable |
activationScripts |
bashless init | makes activationScripts a silent no-op — youko's data-ro-backing, installCaddyWallpapers and installHomepageDashboardConfig would stop running without an error |
lab only |
| D5 | security.pam.services.<name>.rules |
per-service booleans | freeform PAM config | upstream marks it subject to breaking changes; a mistake locks you out | lab only |
| D6 | security.shadow.enable = false + greetd |
shadow suite provides login/su | fewer setuid programs | assertion couples the two; changes VT login semantics | lab only |
| D7 | security.apparmor.policies |
policies.sudo already in use (hardening/apparmor/default.nix); the old profiles option is removed upstream |
policy set is now data | writing profiles is real work and breakage-prone; SECURITY.md explicitly warns against untested profiles |
lab only |
E. Considered and rejected (kept so they are not re-litigated)
| Candidate | What it is | Why not here |
|---|---|---|
| systemd-sysext / confext | overlay read-only images onto /usr//etc of an immutable OS |
solves the ostree/Flatcar problem NixOS does not have: generations are already atomic and rollbackable. Layering via sysext means something escaped the rebuild. |
| Portable services | attach/detach a service image at runtime | aimed at distributing services to foreign systemd distros; generation switching is strictly stronger here. |
| Quadlet | podman generator turning .container files into units |
NixOS already has virtualisation.oci-containers (in use on servers); Quadlet would move unit definitions out of Nix evaluation. A downgrade, not a modernization. |
| Landlock | unprivileged per-process LSM (network rules since 6.12) | already an active default LSM (security/lsm includes it); the missing piece is per-application policy, and this repo's answer to that is nixpak/bwrap. Watch trigger: nixpak/bwrap gain network-restriction policy. |
| Incus | LXD fork, container + VM management plane | no bridge option in the module (it creates incusbr0; attaching to br0 needs preseed); it is a management plane, not a hypervisor replacement, and does not remove libvirt or MicroVM. If adopted, it needs its own disposable host. |
| Gateway API | k8s ingress successor | cross-repo: it belongs to ~/codes/k8s-gitops with a matching branch here only if host networking changes. ingress-nginx is EOL, so this is real work — just not this repo's plan. |
| bcachefs | copy-on-write filesystem, out of experimental upstream | current storage (btrfs + btrbk + restic) works; migration cost is high and unrelated to any pain. Watch only. |
F. Decision summary
Fill this in as cards are decided; the plan documents (on the modernization-plans-wip tag, see the
header) should then be trimmed to adopt items only.
| Card | Candidate | Decision | Date | Note |
|---|---|---|---|---|
| A1 | Tailscale Serve (partial) | |||
| A2 | systemd-resolved | |||
| A3 | systemd-networkd | |||
| A4 | journald upload/remote | |||
| A5 | zram-generator | |||
| A6 | nixos-facter | |||
| A7 | attic | |||
| A8 | nix-servd + daemon features | |||
| A9 | sched_ext | |||
| B1 | limine + Secure Boot | |||
| B2 | boot.uki | |||
| B3 | systemd-repart images | |||
| B4 | initrd repart | |||
| B5 | sysupdate | |||
| B6 | measured boot / FIDO2 | |||
| B7 | systemd-vmspawn | |||
| C1–C3, C5 | tooling + cleanups | |||
| D1–D7 | lab-only group | |||
| E | rejected set |
References
- nixpkgs modules:
system/etc/etc.nix,system/activation/nixos-init.nix,system/boot/loader/limine/limine.nix,system/boot/systemd/{repart,sysupdate,tpm2,fido2,journald-*}.nix,image/{images,repart}.nix,services/networking/{tailscale-serve,dhcpcd,nat-nftables}.nix,services/system/{userborn,nix-daemon,zram-generator}.nix,services/monitoring/prometheus/exporters.nix. - NixOS 26.05 release notes; Nix manual, Experimental Features.
- Repo context:
SECURITY.md(trust zones and control limits),WORKAROUNDS.md(WA-001, WA-008, WA-009, WA-023),modules/nixos/base/networking/mdns.nix,hosts/12kingdoms-youko/homelab-services/caddy.nix,hosts/12kingdoms-youko/homelab-services/README.md.