Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.
- base: nftables backend; deny inbound by default; trust the home LAN by
source and the tailnet by interface (restrict the tailnet with ACLs);
suppress per-service `openFirewall` global ports; allow node_exporter only
from the monitoring host. The local podman bridge, the tailnet and the pod
CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
(caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
redundant desktop firewall module; aarch64/riscv64 import the firewall
explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
the LAN CIDR rule and no global ports.
* feat(youko): add Jellyfin media server
Jellyfin reads the existing Transmission download tree on the public HDD
(via the shared `fileshare` group) and transcodes with the AMD Barceló
iGPU through VA-API. Encoding settings are declared in Nix
(`forceEncodingConfig`), so the web UI cannot silently drift.
Replace the per-service public ACME certificates with a single
`*.writefor.fun` wildcard (plus the apex) shared by every public vhost,
so adding a service needs no new ACME entry; a wildcard also keeps
individual hostnames out of public CT logs.
Pin the jellyfin uid/gid to avoid the dynamic-id drift fixed for the
other services in #320.
* fix(youko): enable graphics for Jellyfin VA-API
VA-API needs Mesa's radeonsi VA driver under /run/opengl-driver, but
youko is headless so hardware.graphics was off. Without it Jellyfin's
ffmpeg fails with 'va_openDriver() returns -1'. Enable it so the iGPU
can be used for hardware transcoding.
* fix(youko): use VA-API hardware encoding for Jellyfin
Enabling hardware decoding alone left encoding on the CPU (libx264), which
pins ~3 cores per 720p stream. Set transcoding.enableHardwareEncoding so
the iGPU (Vega VCN) does the H.264 encode too.
* feat(youko): harden Jellyfin proxy trust, groups, and HEVC encode
Add jellyfin to the render/video groups for robust VA-API access (was
relying on the world-writable render node), enable HEVC hardware encoding
(Vega's VCN supports it; AV1 does not), and patch network.xml on start to
trust Caddy as KnownProxies and advertise the public URL.
* fix(youko): run Jellyfin network.xml patch every start; allow LAN discovery
The module's preStart exits early once encoding.xml matches, so the
mkAfter network.xml patch only ran when encoding settings changed. Run it
with mkBefore instead. Also allow Jellyfin app auto-discovery (UDP 7359 /
TCP 8096) from the home network only.
Service users/groups get a dynamically allocated uid/gid, which shifts when the user set changes: adding `immich` moved gitea 993:985 -> 996:993 and rustfs 987:977 -> 988:983. Files on the persistent volumes keep the old numeric owner, so gitea and rustfs lost access to their data and failed to start after a restart. Pin every dynamically allocated service id on youko to its current value. The ids are host-specific, so this is a host file.
timerConfig was set with lowercase onCalendar/randomizedDelaySec, which systemd treats as unknown directives. The generated timer had no schedule, so `systemctl list-timers` showed NEXT=- and no backup ever ran. Use the real keys OnCalendar/RandomizedDelaySec.
Add Immich via the native nixpkgs services.immich module on youko, reusing the host PostgreSQL 16 (pgvector + vectorchord) and a shared Valkey instance for Redis-compatible caching and BullMQ queues (Dragonfly rejected BullMQ's dynamic Lua keys). Media lives on the encrypted HDD at /data/apps/immich; Caddy publishes immich.writefor.fun with a publicly-trusted Let's Encrypt cert (security.acme + Cloudflare DNS-01) so the mobile app connects. OTEL metrics are scraped by VictoriaMetrics, and /var/cache is now preserved on the tmpfs-root hosts so the ML model cache survives reboots.
The s3-console Caddy vhost redirected every request to / to /rustfs/console/. The RustFS console frontend signs its own S3/STS requests to / (POST / for STS, GET /?x-id=ListBuckets), so the 302 broke the console session and login failed server-side with SignatureDoesNotMatch ("missing signed header: content-type"). Restrict the redirect to unsigned browser GET /; signed requests now reach RustFS unchanged.
- run two autocutsel instances so the X PRIMARY/CLIPBOARD selections stay in
sync, making copy/paste work in the session and through VNC
- drop --disable-blink-features=AutomationControlled: nothing drives the
browser over CDP, so navigator.webdriver is already false and the flag only
marked the setup as automated
- document why the driver stays on real X11/AT-SPI input and is never granted
a CDP attach, and note the SSH-tunnel path for manual VNC access
- extend the computer-use eval test
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
The script pointed the macOS nix-daemon at the homelab v2ray HTTP proxy (192.168.5.101:7890), but that proxy was never running. Remove it together with the just darwin-set-proxy recipe and the README step that referenced it.
Reserve 192.168.5.192/27 as a dedicated kube-vip block, outside the router's DHCP pool (.2-.99): one /29 per cluster, k3s-test-1 gets .192/29 with the API VIP at .193.
Advertise only youko itself (.183/32) and that block over Tailscale, instead of the whole /24. The /24 route shadowed hosts that are already reachable directly, e.g. idols-ai at 192.168.5.100.
Deploy prerequisites (not in this repo): approve the new Tailscale routes; point kube-vip at the new VIP; move the *.writefor.fun wildcard from .101 to .183 and update scripts/darwin_set_proxy.py, since .101 is no longer advertised.
Only three secrets are readable as the login user (nix-access-tokens,
ssh-key-romantic and the nushell one); everything else is root-only and
the filesystem already keeps agents out.
* feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
* docs(backup): say 'cloud copy' rather than 'offsite'
Clearer for a reader who does not live in backup jargon; the requirement
(not in the homelab, immutable) is spelled out in the text.
* docs(secrets): state the recipient invariant
A recipient set must answer two questions: who may read it, and how it is
recovered once those hosts are gone. Dropping recovery_key while narrowing
a secret is easy and silent.
* docs(secrets): state that every secret must be decryptable by the desktops
The desktops hold the trusted admin keys and are where secrets are edited and
rekeyed from; narrowing a set to the servers alone breaks agenix -r and drops
the offline recovery key.
* docs(secrets): desktops and recovery_key have identical access
recovery_key is a member of desktop_keys, so every group written as
desktop_keys ++ ... covers both.
* feat(monitoring): scrape the restic rest-server
Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.
* docs(backup): record what the metrics actually expose
rest-server only ships the go/promhttp defaults, so repository growth is
charted from node-exporter's filesystem metrics instead.
* feat(hosts): per-host restic backups to the SMB remote
Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:
- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
(rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
excludes for the regenerable/bulky paths (podman overlay layers, microvm
and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
and dumps all postgres databases first (a consistent SQL artifact beside
the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token
Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.
* refactor(hosts): only restic does off-host; btrbk stays local
btrbk works at the subvolume level, so a would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.
So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.
Update the btrbk module's comment to say so.
* fix(restic): never back up keys; drop the implicit /etc/agenix
restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).
* refactor(restic): make the backup a proper module
Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.
* refactor(restic): exclude the whole podman storage, trim retention
Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.
Retention: daily 3, weekly 2, monthly 2 - no yearly.
* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump
A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.
* feat(secrets): restic-password for modules.restic-backup
With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.
* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix
* fix(restic): take the password path from agenix, not a hardcoded one
config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
* refactor(youko): let preservation own the service state dirs
The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:
- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind
Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.
* refactor(youko): drop caddy's redundant tmpfiles rules
The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.
* refactor(youko): use a podman named volume for uptime-kuma
podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
The VMs now declare their root disk as a DataVolume with `source: upload`, so
upload straight into `<vm>-disk` (CDI only accepts an upload into a DataVolume)
instead of a shared golden PVC. Takes the disk size as an argument.
* feat(k3s-test-1): add three worker nodes and taint the control plane
- hosts/k8s/k3s-test-1-worker-{1,2,3} (k3s agents) + flake outputs
- vars/networking: .117/.118/.119 (the .111-.113 range is still held by the
never-deployed k3s-prod-1 workers)
- k3s-test-1-master-{1,2,3}: nodeTaints=node-role.kubernetes.io/control-plane:NoSchedule
- Justfile: upload the worker images too
* chore(k3s-test-1): move the workers to .111-.113
The k3s-prod-1 entries that held those addresses are gone (nix-config#284).
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.
- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
* feat(kubevirt-youko): export a shared golden-image store over NFS
* feat(vm-images): upload golden images into the cluster
Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
services.earlyoom.freeMemThresholdPercent does not exist in this nixpkgs; the module exposes freeMemThreshold/freeSwapThreshold. Just enable earlyoom with its defaults (trigger at 10% free mem/swap).
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
* feat(btrbk): make snapshots explicit and guarded
Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.
Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.
Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.
* feat(monitoring): alert on btrbk snapshot failure
Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
* docs(agents): bind authorization to the change boundary
* docs(agents): require target identity confirmation
* docs(agents): classify operations by impact, not command form
* docs(agents): require reviewed previews for high-risk IaC
* docs(agents): verify production changes beyond exit codes
* docs(agents): sync new-script eval with current rules
* docs(agents): split global rules evals into smoke and extended
* docs(agents): refine change guardrails wording
- scope the guardrails to any environment, not only production
- preview high-risk changes with plan, diff, or dry-run
- trim redundant examples and simplify target identity confirmation
- confirm unclear targets with the user instead of stopping
- render Niri on the Intel iGPU while retaining NVIDIA-connected physical outputs
- encode Sunshine streams with Intel VAAPI and wait for a Niri output before startup
- load VKMS persistently for headless sessions and configure `Virtual-1` at 2560x1600@60
- let Moonlight request the stream bitrate instead of enforcing a 20 Mbps host cap
- remove unsupported Dynamic Boost and global NVIDIA session overrides
- grant Sunshine access to virtual input devices
- add a Nushell recovery helper that starts a temporary greetd/Niri session from SSH
- identify and safely replace normal, standalone, or previous transient Niri sessions
- launch the transient service in the background and verify greetd, Niri, and Sunshine before returning
- ignore local `.worktrees` directories and keep eval regression coverage focused on structural behavior
* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
* feat(nushell): add trash command using the home trash can
On a tmpfs root with persistent dirs bind-mounted in, `rm --trash`
scatters items into per-mount .Trash-$uid dirs that file managers never
show, and gio refuses to trash on those internal mounts.
Add a `trash` command implementing the freedesktop trash spec
'failsafe' mode: items are moved to ~/.local/share/Trash with
spec-compliant .trashinfo entries, so Thunar can list and restore them.
Also preserve ~/.local/share/Trash on idols-ai so the trash survives
reboots.
* feat(nixos): manage scattered trash dirs with trash-cli retention timer
Replace the hand-written nushell trash command with the established
trash-cli tooling. On a tmpfs root with persistent dirs bind-mounted
in, the trash crate (nushell rm --trash) scatters items into per-mount
.Trash-$uid dirs that file managers never show and nothing cleans up.
- add trash-cli to system packages: trash-list scans/trash-restore
handles every mount point's trash dir
- daily systemd timer runs 'trash-empty 30 -f', purging items older
than 30 days across the home trash and all mount points
- drop the hand-written trash.nu module
- note in preservation.nix: do NOT persist ~/.local/share/Trash; a
bind-mounted trash dir breaks the trash crate's home-topdir match
for files straight under $HOME (it would try /.Trash-$uid, EACCES)
* fix(nixos): order trash-empty after preservation.target
The scattered .Trash-$uid dirs live inside the preservation bind
mounts, which use DefaultDependencies=no and are therefore NOT ordered
after local-fs.target. Add preservation.target to After= so trash-empty
only runs once all those mounts are up.
Explicitly set programs.nixvim.nixpkgs.source to suppress the follows
warning, and migrate all programs.ssh.matchBlocks to programs.ssh.settings
using upstream PascalCase directive names.
* chore(home): drop AstroNvim bundle and centralize editors
Remove the bundled Neovim plugin tree to shrink supply-chain exposure; keep a
minimal programs.neovim backup. Daily editing stays Helix-first with Yazi and
Zellij.
- Add Helix/Neovim docs, glossary, and cheatsheets under core/editors/.
- Default EDITOR/VISUAL to hx; use SUDO_EDITOR nvim --clean for sudoedit and
other sensitive edits; adjust Nushell buffer_editor accordingly.
- Apply Helix Home Manager settings (keys, editor UI) in core/editors/helix.
- Keep heavy language-server tooling only in home/base/tui/editors/packages.nix;
wire it via an explicit default.nix import so core stays lightweight.
- Extend modules/base packages.nix where that profile sets global editor env.
* docs: link editors, VS Code, and agents from README
Document tui/editors versus core/editors; add root README pointers to GUI
vscode (Home Manager) and agents/. Cross-link heavy packages README from
core/editors.
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
* feat: comment out nixpkgs-unstable, we use unstable by default
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
* fix: typo...
---------
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
* feat: replace impermanence with preservation
* fix: This module cannot be used with scripted initrd.
* fix: permission for /var/lib/private
* chore: boot.initrd.systemd.enable
* docs: boot.initrd.luks.devices.<name>.fallbackToPassword is implied by systemd stage 1
* fix: typo
ryan4yin | 二花