* fix(youko): add ROCm OpenCL so Jellyfin can tone-map HDR
Jellyfin's HDR/DoVi transcodes use ffmpeg's `tonemap_opencl`, but only
Mesa was installed via `hardware.graphics.enable`, so no OpenCL ICD was
registered. ffmpeg then failed to create an OpenCL device ("Failed to
get number of OpenCL platforms: -1001") and aborted with exit code 237,
leaving HDR titles unplayable.
Add `rocmPackages.clr.icd` to `hardware.graphics.extraPackages`; it
ships the ICD for the Barceló iGPU (reported as `gfx90c`) under
/run/opengl-driver/etc/OpenCL/vendors, where `ocl-icd` (built with
`--enable-custom-vendordir`) finds it.
The Barceló iGPU only manages ~0.8x realtime for a 4K HDR -> 1080p
tone-mapped transcode, so playback can still buffer; this only makes
such titles playable at all.
* fix(youko): allow /dev/kfd so Jellyfin OpenCL can reach the GPU
The jellyfin module's `DeviceAllow` whitelist only permits the VA-API
render node (`/dev/dri/renderD128`). ROCm's OpenCL runtime opens
`/dev/kfd`, so even with the ICD installed OpenCL device creation failed
("No matching devices found") and HDR transcodes still aborted with exit
code 237.
Add `/dev/kfd rw` to the whitelist; list definitions merge with the
module's entry. Verified under the unit's sandbox (PrivateUsers,
PrivateTmp) that OpenCL init then succeeds as the jellyfin user.
davfs2 caches the whole opened file regardless of cache_size, so opening media on /mnt/fileshare downloads it fully and blocks the mount. The share is no longer used.
* fix(youko): stop homepage from probing the authenticated transmission RPC
The siteMonitor hit /transmission/rpc unauthenticated, so every probe returned 401 and counted against transmission's anti-brute-force threshold.
* feat(youko): tune transmission and firewall its network namespace
Raise the upload limit and queues, relax required encryption to preferred, and add a default-drop nftables input chain inside the netns so the peer port stays open while everything else is closed. Allowlist the nftables 'iif' keyword for typos.
* feat(youko): export transmission metrics and add a Grafana dashboard
nixpkgs has no transmission exporter, so add a small stdlib Python exporter that queries the authenticated netns RPC on loopback, scrape it from VictoriaMetrics, and ship a Transmission dashboard.
* fix(youko): cap transmission upload at 2500 kB/s
30 MB/s ignored the 50 Mbps uplink; keep seeding around 20 Mbps.
* Revert "feat(youko): export transmission metrics and add a Grafana dashboard"
This reverts commit 9df7a26526.
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.
- base: nftables backend; deny inbound by default; trust the home LAN by
source and the tailnet by interface (restrict the tailnet with ACLs);
suppress per-service `openFirewall` global ports; allow node_exporter only
from the monitoring host. The local podman bridge, the tailnet and the pod
CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
(caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
redundant desktop firewall module; aarch64/riscv64 import the firewall
explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
the LAN CIDR rule and no global ports.
* feat(youko): add Jellyfin media server
Jellyfin reads the existing Transmission download tree on the public HDD
(via the shared `fileshare` group) and transcodes with the AMD Barceló
iGPU through VA-API. Encoding settings are declared in Nix
(`forceEncodingConfig`), so the web UI cannot silently drift.
Replace the per-service public ACME certificates with a single
`*.writefor.fun` wildcard (plus the apex) shared by every public vhost,
so adding a service needs no new ACME entry; a wildcard also keeps
individual hostnames out of public CT logs.
Pin the jellyfin uid/gid to avoid the dynamic-id drift fixed for the
other services in #320.
* fix(youko): enable graphics for Jellyfin VA-API
VA-API needs Mesa's radeonsi VA driver under /run/opengl-driver, but
youko is headless so hardware.graphics was off. Without it Jellyfin's
ffmpeg fails with 'va_openDriver() returns -1'. Enable it so the iGPU
can be used for hardware transcoding.
* fix(youko): use VA-API hardware encoding for Jellyfin
Enabling hardware decoding alone left encoding on the CPU (libx264), which
pins ~3 cores per 720p stream. Set transcoding.enableHardwareEncoding so
the iGPU (Vega VCN) does the H.264 encode too.
* feat(youko): harden Jellyfin proxy trust, groups, and HEVC encode
Add jellyfin to the render/video groups for robust VA-API access (was
relying on the world-writable render node), enable HEVC hardware encoding
(Vega's VCN supports it; AV1 does not), and patch network.xml on start to
trust Caddy as KnownProxies and advertise the public URL.
* fix(youko): run Jellyfin network.xml patch every start; allow LAN discovery
The module's preStart exits early once encoding.xml matches, so the
mkAfter network.xml patch only ran when encoding settings changed. Run it
with mkBefore instead. Also allow Jellyfin app auto-discovery (UDP 7359 /
TCP 8096) from the home network only.
Service users/groups get a dynamically allocated uid/gid, which shifts when the user set changes: adding `immich` moved gitea 993:985 -> 996:993 and rustfs 987:977 -> 988:983. Files on the persistent volumes keep the old numeric owner, so gitea and rustfs lost access to their data and failed to start after a restart. Pin every dynamically allocated service id on youko to its current value. The ids are host-specific, so this is a host file.
Add Immich via the native nixpkgs services.immich module on youko, reusing the host PostgreSQL 16 (pgvector + vectorchord) and a shared Valkey instance for Redis-compatible caching and BullMQ queues (Dragonfly rejected BullMQ's dynamic Lua keys). Media lives on the encrypted HDD at /data/apps/immich; Caddy publishes immich.writefor.fun with a publicly-trusted Let's Encrypt cert (security.acme + Cloudflare DNS-01) so the mobile app connects. OTEL metrics are scraped by VictoriaMetrics, and /var/cache is now preserved on the tmpfs-root hosts so the ML model cache survives reboots.
The s3-console Caddy vhost redirected every request to / to /rustfs/console/. The RustFS console frontend signs its own S3/STS requests to / (POST / for STS, GET /?x-id=ListBuckets), so the 302 broke the console session and login failed server-side with SignatureDoesNotMatch ("missing signed header: content-type"). Restrict the redirect to unsigned browser GET /; signed requests now reach RustFS unchanged.
Deploy RustFS in place of the retired MinIO and move the S3 consumers over.
- flake.nix / flake.lock / outputs/default.nix: add a nixpkgs-small
(nixos-unstable-small) input for the rustfs 1.0.0 package (with the web
console), instantiated as pkgs-small in genSpecialArgs alongside the other
nixpkgs instances; drop the rustfs flake input. The rest of the system stays
on the default nixpkgs.
- hosts/12kingdoms-youko/homelab-services/rustfs.nix: use nixpkgs' own
services.rustfs module, with package = pkgs-small.rustfs and
environmentFile = the agenix rustfs.env secret.
- caddy.nix: s3.writefor.fun (S3 API) and s3-console.writefor.fun (console,
redirects / to /rustfs/console/).
- secrets/nixos.nix: declare a single rustfs.env (dotenv) secret holding
RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY, replacing the separate key files.
- infra/minio -> infra/rustfs: buckets are managed with the hashicorp/aws
provider against the RustFS S3 endpoint; IAM/ILM are applied with rc.
Crush hardcodes banned bash commands, blocking many operations. Drop the package, its AGENTS.md/CRUSH.md install target, env vars, and preservation entries.
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
kubelet refuses to self-assign the reserved node-role.kubernetes.io/* labels, so the worker --node-label made the agent crash-loop. Drop it and rely on the masters' control-plane taint for placement; document that the taint only lands via registerWithTaints on first registration, so it was applied by hand here.
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
Reserve 192.168.5.192/27 as a dedicated kube-vip block, outside the router's DHCP pool (.2-.99): one /29 per cluster, k3s-test-1 gets .192/29 with the API VIP at .193.
Advertise only youko itself (.183/32) and that block over Tailscale, instead of the whole /24. The /24 route shadowed hosts that are already reachable directly, e.g. idols-ai at 192.168.5.100.
Deploy prerequisites (not in this repo): approve the new Tailscale routes; point kube-vip at the new VIP; move the *.writefor.fun wildcard from .101 to .183 and update scripts/darwin_set_proxy.py, since .101 is no longer advertised.
Give the idols-ruby and idols-kana VMs a headless computer-use
environment: Xvfb plus i3, driven by cua-driver (and computer-use-linux
as a fallback). Includes the system module (AT-SPI, fonts, Mesa, Clash
Verge proxy, proxy-region timezone), the home i3 module (Xvfb/i3/x11vnc
services, browser flags), host wiring and an eval test.
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
A desktop holds both the plaintext and the repository password, so blocking
deletion on the server protects little while preventing the automatic
retention the client needs. Immutability belongs in the offsite copy.
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
Split the repository password into desktop and homelab groups, add the
environmentFile/pruneOpts module options, and point the desktop at youko's
REST endpoint.
Allowlist of the irreplaceable directories (Documents, work, Pictures,
nix-config, .local/state); project checkouts under ~/codes live in git and
are not backed up. Kept the excludes generic, since this repo is public.
Also move the restic-password secret to a desktop.enable || server.storage
gate so desktops get it without enabling a server group.
* feat(hosts): per-host restic backups to the SMB remote
Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:
- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
(rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
excludes for the regenerable/bulky paths (podman overlay layers, microvm
and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
and dumps all postgres databases first (a consistent SQL artifact beside
the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token
Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.
* refactor(hosts): only restic does off-host; btrbk stays local
btrbk works at the subvolume level, so a would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.
So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.
Update the btrbk module's comment to say so.
* fix(restic): never back up keys; drop the implicit /etc/agenix
restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).
* refactor(restic): make the backup a proper module
Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.
* refactor(restic): exclude the whole podman storage, trim retention
Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.
Retention: daily 3, weekly 2, monthly 2 - no yearly.
* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump
A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.
* feat(secrets): restic-password for modules.restic-backup
With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.
* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix
* fix(restic): take the password path from agenix, not a hardcoded one
config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
* refactor(youko): let preservation own the service state dirs
The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:
- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind
Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.
* refactor(youko): drop caddy's redundant tmpfiles rules
The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.
* refactor(youko): use a podman named volume for uptime-kuma
podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
The services now run directly on the host, so their state does not belong
on the HDD behind the flaky USB-SATA bridge: /data/apps was small-file
heavy and latency/consistency sensitive. Point every dataDir at
/persistent/apps (the LUKS nvme) instead.
The HDD keeps the bulk data that suits it: /data/fileshare (173G) and
/data/backups (24G). The HDD's old @apps subvolume and its data are left
in place for now.
-B 255 turns APM off and it stuck, but the disks still reach standby, so
the bridge firmware is likely doing the spindown. -J 0 additionally asks
the drive to stop parking its heads (sdb has ~1M load cycles).
pcie_aspm=off was the least-justified mitigation (the flakiness is on the
USB/SATA side) and costs power. Also log the disks' APM level/state and
re-apply the settings hourly, since the bridge or the disk firmware can
drop them (the disks were still reaching standby).
The two 4TB HDDs sit behind a JMicron JMS567 that resets and throws
UDMA-CRC / I/O errors (the disks themselves are healthy). Best-effort
mitigations while the enclosure stays in use:
- kernel: usbcore.autosuspend=-1, usb-storage.delay_use=10, pcie_aspm=off
- udev: keep the bridge's power/control=on (never autosuspend)
- disable APM/standby on the HDDs (spindown is a common reset trigger)
- enable smartd (watch the CRC count and disk health)
The kernel parameters need a reboot to take effect.
Update the user-facing bits that still pointed at KubeVirt: the homepage
dashboard group and entries (and drop the dead LongHorn one), the docs,
and remove the now-unneeded virtctl package.
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:
- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
restic backup path must stay as it is)
Drop the now-meaningless kubevirt- prefix and move the hosts out of
hosts/k8s/, following the existing 12kingdoms-shoukei layout: the
directory and output carry the theme prefix while the configuration,
hostname and colmena target are the bare name (shoryu, shushou, youko).
- hosts/k8s/kubevirt-* -> hosts/12kingdoms-*; the shared disko config
becomes host-disko-fs.nix; the now-different relative imports are fixed
- outputs, vars/networking.nix keys, the Justfile recipes/tags, the eval
tests and the docs follow
- colmena reaches these hosts by IP (a new targetHost override), since
their DNS name changes with the hostname and is not resolvable until
the router learns it
The physical hosts no longer need k3s or KubeVirt: aquamarine runs
natively, the k3s-test masters run as microVMs and kana/ruby under
libvirt. So drop the host cluster and split the shared parts out of the
KubeVirt host module:
- lib/genVmHostModule.nix: the host bits the VMs still need (br0 + the
physical NIC as a bridge port, KVM, sysctls, earlyoom, libvirt,
journald, stateVersion)
- the three hosts use it and drop genK3sServerModule; also drop the
longhorn workarounds (openiscsi + the /usr/local/bin tmpfile) and the
virtctl package
- remove lib/genKubeVirtHostModule.nix
hosts/k8s/README.md: how the microVMs keep state, the tap/br0 naming, how
to move a KubeVirt VM over in place, and the refreshed TODOs. Also note
the migration in Virtual-Machine.md and flag the volume images in
AGENTS.md.
Replace the KubeVirt/etcd members with microVMs on the three physical
hosts, keeping the existing cluster: the guests inherit their host key
and k3s state from the old disks, so etcd recovers in place instead of
bootstrapping a new cluster.
- convert hosts/k8s/k3s-test-1-master-{1,2,3} to genMicrovmGuestModule
and switch their outputs to the microvm module (keep home-modules)
- declare the vms on shoryu/youko/shushou and bridge their taps to br0
- each guest config is self-contained (its secret group and the
nixpkgs.config override live in the guest, not the host module)
- guest specialArgs carry agenix/mysecrets so secrets/nixos.nix works