Commit Graph
614 Commits
Author SHA1 Message Date
ryan4yin | 二花 d056aece28 chore(workarounds): re-evaluate stale hardware/kernel fixes (#360)
* chore(workarounds): re-evaluate stale hardware/kernel fixes

Prune workarounds whose root cause is gone, fix one vendor typo, and
document the ones we keep:

- drop `transparent_hugepage=madvise` on shoryu/shushou/youko: it was tied to
  the old fixed 1G hugetlb pool, and the VMs now use ordinary memory
- shoryu: `hardware.cpu.intel.updateMicrocode` -> `amd` (it is an AMD APU)
- ai: drop `i915.enable_guc=3` (GuC/HuC submission is the default on modern
  i915/Xe)
- kernel-hardening: name the Dirty Frag CVEs (CVE-2026-43284 / CVE-2026-43500)
  and the re-evaluation condition
- document re-eval conditions for the Insta360 resume fix, the darwin
  auto-optimise-store workaround, and the youko USB-SATA bridge mitigations

* chore(workarounds): keep the Dirty Frag module blacklist as defence in depth

* chore(ai): correct the i915.enable_guc rationale (verified against the kernel)

* chore(darwin): drop the auto-optimise-store workaround (nix#7273 fixed)
2026-10-03 11:25:34 +08:00
ryan4yin | 二花 dc3ec8c159 feat(storage): relax ai NVMe workarounds and add drive SMART monitoring (#359)
* fix(ai): relax NVMe power-management workarounds and enable smartd

Logs show no NVMe errors, so allow APST again (capped at 100 ms), restore
the default io_timeout, and drop pcie_aspm=off. Add smartd for drive
health visibility.

* feat(storage): add drive/SSD SMART monitoring on physical hosts

node_exporter's nvme collector only exposes node_nvme_info and namespace
capacity, not SMART health, so SSD wear/lifespan and drive temperature were
not observable anywhere.

- enable services.prometheus.exporters.smartctl on physical hosts, gated on
  modules.hardwareTools (off on QEMU/MicroVM guests), bound to the host's LAN
  address on port 9633
- firewall: drop 9633 from anything but the monitoring host
- youko: scrape smartctl-<host> for the physical hosts
- alert rules (storage.yml): SMART health, NVMe wear, available spare, media
  errors, critical warning, and drive temperature (warn 70 / crit 80 C)
- Grafana dashboard "Drive SMART (smartctl)"
2026-10-03 11:13:46 +08:00
ryan4yin | 二花 2bacb5c138 refactor: drop redundant package declarations (#353)
Remove no-op overrides and packages already provided elsewhere in the
same profile.
2026-10-03 10:11:30 +08:00
Ryan Yin d9208f492b fix(preservation): keep logrotate state in a persisted directory 2026-10-03 09:42:14 +08:00
Ryan Yin efdf153f0a fix(shoukei): stop latched nscd failures on resolvconf restarts 2026-10-03 09:42:13 +08:00
ryan4yin | 二花 9254b63a18 chore(idols-ai): move tmp state off RAM onto btrfs and harden tmp mounts (#348)
* feat(idols-ai): persist /var/tmp and small state dirs, wipe /var/tmp at boot

Move the strata model staging area (~3G) off the root tmpfs onto the
persistent btrfs volume to free RAM, preserve btrbk/cups/logrotate
state, and wipe /var/tmp at boot so it only accumulates between
reboots.

* chore(idols-ai): add nosuid,nodev to root tmpfs and tmp btrfs mounts

Hardens /, /tmp and /snapshots; safe because /dev, /run, /nix and
/run/setuid-programs are separate mounts, so nothing there needs suid
bits or device nodes.

* chore(idols-ai): add noatime to the @tmp btrfs mount

Temp files do not need access times; matches the @nix subvolume style
and saves btrfs metadata writes.

* chore(idols-ai): harden the /var/tmp bind mount with nosuid,nodev

World-writable staging data should not honor suid bits or device
nodes, same as /tmp and /snapshots.

* fix(idols-ai): preserve logrotate state as a file, fix persisted dir ownership

- /var/lib/logrotate.status is a regular file, not a directory: move it from
  preservation directories to files. A missing state entry makes logrotate
  treat every log as freshly rotated, so it must survive reboots.
- keep /var/lib/btrbk owned by btrbk:btrbk 0750 so preservation does not
  override the btrbk module's rule with root:root.
- set /var/tmp mode 1777 explicitly instead of relying on systemd's tmp.conf
  ordering.
- rewrite comments to describe the paths generically.

* chore(idols-ai): cap journal, wipe coredumps at boot, persist upower/noctalia

- services.journald.settings.Journal: SystemMaxUse=10G, RuntimeMaxUse=256M,
  matching the cap already set on idols-kana/idols-ruby.
- wipe /var/lib/systemd/coredump at boot: it lives in the persisted
  /var/lib/systemd and crash dumps can contain secrets copied from memory.
- persist /var/lib/upower (device battery history) and
  ~/.local/state/noctalia (shell settings, notification/clipboard history).
2026-10-02 21:58:21 +08:00
Ryan Yin a26ab3be63 chore(disko): grow swapfile from 20G to 96G
Running strata with qwen3.8-flash-next keeps ~53G of model weights
resident, leaving little RAM for the desktop side and pushing ~13G of
anonymous pages into the 20G swapfile. With swap that full, a burst
risks OOM-killing processes. Disk space is not a constraint (1.9T
btrfs, ~1.4T free), so grow the swapfile to the classic 1x-RAM size.

Keep zram off and zswap at 10%: zram's compressed pool would steal page
cache that the mmap'd weights rely on.
2026-10-02 19:45:08 +08:00
Ryan Yin ef61c458e6 chore(go-musicfox): declare config.toml and reset it on rebuild
Manage ~/.config/go-musicfox/config.toml declaratively: the upstream default plus the mpv engine, ~/Music download dir, 2 GiB cache and cover art.

The app rewrites the file with a temp file plus rename, which would replace a plain symlink, so use force = true like fcitx5's profile: the repo copy is restored on every rebuild and runtime edits are ephemeral. Runtime data and state stay on the persistent volume.
2026-10-02 14:39:19 +08:00
Ryan Yin 25f059a49b chore: add go-musicfox 2026-10-02 14:12:05 +08:00
ryan4yin | 二花 969a20ac7b docs: refresh repository images (#343) 2026-10-02 12:30:53 +08:00
ryan4yin | 二花 3a0afe5499 chore(youko): drop stale KubeVirt references (#341)
KubeVirt is gone: remove its Grafana dashboard and the commented
Homepage service entry.
2026-10-02 11:16:45 +08:00
ryan4yin | 二花 2b0f01d99c chore(agents): workspace layout, related repositories, and ~/src checkouts (#340)
* docs(skills): point nix-config-secrets at the private repo under ~/codes

Record where the private nix-secrets checkout lives and that agents should
do the mechanical secret work themselves, handing back only the
interactive/sudo agenix operations and pushes.

* docs(agents): add related repositories to nix-config

* chore(agents): adopt ~/src for source checkouts

Move third-party/source checkouts from ~/codes/src to ~/src so the home
layout reads as three siblings: ~/codes (personal), ~/work (work),
~/src (checkouts). Preserve and back up the new path, and update the
global rule and the two nixpkgs skill references.

* docs(agents): tighten the code layout rule

* docs(agents): fold the work-confidentiality note into the layout line

* docs(agents): base the confidentiality rule on repo visibility

* docs(agents): drop the stale KubeVirt mention
2026-10-02 11:16:24 +08:00
ryan4yin | 二花 68ea04f173 fix(docs): correct stale and mismatched documentation (#338) 2026-10-02 10:21:55 +08:00
ryan4yin | 二花 b2ec56db66 docs(skills): add repo-scoped task skills and give each fact one home (#337) 2026-10-02 10:13:03 +08:00
ryan4yin | 二花 d690f45183 feat: add CanoKey hardware security key support (#334) 2026-10-01 20:50:13 +08:00
Ryan Yin 14db49660b chore(preservation): drop persist entries for removed apps 2026-10-01 20:10:08 +08:00
ryan4yin | 二花 ab2fd0fee9 feat: add plocate filesystem search and cap podman log size (#333) 2026-10-01 20:00:53 +08:00
ryan4yin | 二花 8f9fdda2bb refactor: trim unused virtualisation config and packages (#332) 2026-10-01 19:42:20 +08:00
Ryan Yin 01ed0aa52f chore(agents): upgrade to opencode v2 2026-10-01 09:39:05 +08:00
ryan4yin | 二花 9ab5eca60c feat(monitoring): GPU-aware btop variants and GPU metrics (#330) 2026-09-30 00:16:50 +08:00
Ryan Yin c7d8db2bf3 fix(k8s): rebalance k3s-test-1 VMs after the youko/shushou swap 2026-09-29 21:46:09 +08:00
Ryan Yin 2e5f539a47 fix(hosts): move UPS/nut metrics to youko after the swap 2026-09-29 21:36:47 +08:00
Ryan Yin 2be0337979 docs(hosts): record the youko/shushou swap and its rationale 2026-09-29 21:36:47 +08:00
ryan4yin | 二花 8ef52e2cf7 fix(youko): add ROCm OpenCL so Jellyfin can tone-map HDR (#328)
* fix(youko): add ROCm OpenCL so Jellyfin can tone-map HDR

Jellyfin's HDR/DoVi transcodes use ffmpeg's `tonemap_opencl`, but only
Mesa was installed via `hardware.graphics.enable`, so no OpenCL ICD was
registered. ffmpeg then failed to create an OpenCL device ("Failed to
get number of OpenCL platforms: -1001") and aborted with exit code 237,
leaving HDR titles unplayable.

Add `rocmPackages.clr.icd` to `hardware.graphics.extraPackages`; it
ships the ICD for the Barceló iGPU (reported as `gfx90c`) under
/run/opengl-driver/etc/OpenCL/vendors, where `ocl-icd` (built with
`--enable-custom-vendordir`) finds it.

The Barceló iGPU only manages ~0.8x realtime for a 4K HDR -> 1080p
tone-mapped transcode, so playback can still buffer; this only makes
such titles playable at all.

* fix(youko): allow /dev/kfd so Jellyfin OpenCL can reach the GPU

The jellyfin module's `DeviceAllow` whitelist only permits the VA-API
render node (`/dev/dri/renderD128`). ROCm's OpenCL runtime opens
`/dev/kfd`, so even with the ICD installed OpenCL device creation failed
("No matching devices found") and HDR transcodes still aborted with exit
code 237.

Add `/dev/kfd rw` to the whitelist; list definitions merge with the
module's entry. Verified under the unit's sandbox (PrivateUsers,
PrivateTmp) that OpenCL init then succeeds as the jellyfin user.
2026-09-29 19:52:27 +08:00
Ryan Yin 470006ed3a chore(rustfs): drop nixpkgs-small input, use nixpkgs rustfs 1.0.0 2026-09-28 10:42:51 +08:00
Ryan Yin 853747425f chore(idols-ai): drop the davfs WebDAV fileshare mount
davfs2 caches the whole opened file regardless of cache_size, so opening media on /mnt/fileshare downloads it fully and blocks the mount. The share is no longer used.
2026-09-26 23:47:29 +08:00
ryan4yin | 二花 1256570a84 feat(youko): firewall and tune transmission, stop homepage RPC probes (#326)
* fix(youko): stop homepage from probing the authenticated transmission RPC

The siteMonitor hit /transmission/rpc unauthenticated, so every probe returned 401 and counted against transmission's anti-brute-force threshold.

* feat(youko): tune transmission and firewall its network namespace

Raise the upload limit and queues, relax required encryption to preferred, and add a default-drop nftables input chain inside the netns so the peer port stays open while everything else is closed. Allowlist the nftables 'iif' keyword for typos.

* feat(youko): export transmission metrics and add a Grafana dashboard

nixpkgs has no transmission exporter, so add a small stdlib Python exporter that queries the authenticated netns RPC on loopback, scrape it from VictoriaMetrics, and ship a Transmission dashboard.

* fix(youko): cap transmission upload at 2500 kB/s

30 MB/s ignored the 50 Mbps uplink; keep seeding around 20 Mbps.

* Revert "feat(youko): export transmission metrics and add a Grafana dashboard"

This reverts commit 9df7a26526.
2026-09-26 22:40:02 +08:00
ryan4yin | 二花 e5b95affa6 feat(nixos): default-deny host inbound, trust the LAN/tailnet (#325)
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.

- base: nftables backend; deny inbound by default; trust the home LAN by
  source and the tailnet by interface (restrict the tailnet with ACLs);
  suppress per-service `openFirewall` global ports; allow node_exporter only
  from the monitoring host. The local podman bridge, the tailnet and the pod
  CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
  (caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
  redundant desktop firewall module; aarch64/riscv64 import the firewall
  explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
  flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
  LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
  proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
  the LAN CIDR rule and no global ports.
2026-09-26 14:02:03 +08:00
ryan4yin | 二花 2314754655 fix(youko): run transmission in its own network namespace (#324) 2026-09-26 12:48:51 +08:00
Ryan Yin 2a2933a889 docs(youko): note the transmission inbound-port TODO
youko's default gateway is suzi (mihomo transparent proxy, no UPnP/NAT-PMP),
so transmission cannot map its peer port; record the options for later.
2026-09-26 00:23:54 +08:00
ryan4yin | 二花 9fe6166fd1 feat(homepage): add Jellyfin, alerting, files and k3s-test-1 services (#322) 2026-09-26 00:15:06 +08:00
ryan4yin | 二花 a111676ede feat(youko): add Jellyfin media server (#321)
* feat(youko): add Jellyfin media server

Jellyfin reads the existing Transmission download tree on the public HDD
(via the shared `fileshare` group) and transcodes with the AMD Barceló
iGPU through VA-API. Encoding settings are declared in Nix
(`forceEncodingConfig`), so the web UI cannot silently drift.

Replace the per-service public ACME certificates with a single
`*.writefor.fun` wildcard (plus the apex) shared by every public vhost,
so adding a service needs no new ACME entry; a wildcard also keeps
individual hostnames out of public CT logs.

Pin the jellyfin uid/gid to avoid the dynamic-id drift fixed for the
other services in #320.

* fix(youko): enable graphics for Jellyfin VA-API

VA-API needs Mesa's radeonsi VA driver under /run/opengl-driver, but
youko is headless so hardware.graphics was off. Without it Jellyfin's
ffmpeg fails with 'va_openDriver() returns -1'. Enable it so the iGPU
can be used for hardware transcoding.

* fix(youko): use VA-API hardware encoding for Jellyfin

Enabling hardware decoding alone left encoding on the CPU (libx264), which
pins ~3 cores per 720p stream. Set transcoding.enableHardwareEncoding so
the iGPU (Vega VCN) does the H.264 encode too.

* feat(youko): harden Jellyfin proxy trust, groups, and HEVC encode

Add jellyfin to the render/video groups for robust VA-API access (was
relying on the world-writable render node), enable HEVC hardware encoding
(Vega's VCN supports it; AV1 does not), and patch network.xml on start to
trust Caddy as KnownProxies and advertise the public URL.

* fix(youko): run Jellyfin network.xml patch every start; allow LAN discovery

The module's preStart exits early once encoding.xml matches, so the
mkAfter network.xml patch only ran when encoding settings changed. Run it
with mkBefore instead. Also allow Jellyfin app auto-discovery (UDP 7359 /
TCP 8096) from the home network only.
2026-09-25 23:22:04 +08:00
ryan4yin | 二花 9187e4d920 fix(youko): pin dynamically-allocated service uid/gid (#320)
Service users/groups get a dynamically allocated uid/gid, which shifts when the user set changes: adding `immich` moved gitea 993:985 -> 996:993 and rustfs 987:977 -> 988:983. Files on the persistent volumes keep the old numeric owner, so gitea and rustfs lost access to their data and failed to start after a restart. Pin every dynamically allocated service id on youko to its current value. The ids are host-specific, so this is a host file.
2026-09-25 20:55:02 +08:00
ryan4yin | 二花 793b636fa5 feat(youko): add Immich photo platform (#317)
Add Immich via the native nixpkgs services.immich module on youko, reusing the host PostgreSQL 16 (pgvector + vectorchord) and a shared Valkey instance for Redis-compatible caching and BullMQ queues (Dragonfly rejected BullMQ's dynamic Lua keys). Media lives on the encrypted HDD at /data/apps/immich; Caddy publishes immich.writefor.fun with a publicly-trusted Let's Encrypt cert (security.acme + Cloudflare DNS-01) so the mobile app connects. OTEL metrics are scraped by VictoriaMetrics, and /var/cache is now preserved on the tmpfs-root hosts so the ML model cache survives reboots.
2026-09-25 20:30:01 +08:00
ryan4yin | 二花 6a6950e7eb fix(youko): don't redirect rustfs console STS/S3 calls (#316)
The s3-console Caddy vhost redirected every request to / to /rustfs/console/. The RustFS console frontend signs its own S3/STS requests to / (POST / for STS, GET /?x-id=ListBuckets), so the 302 broke the console session and login failed server-side with SignatureDoesNotMatch ("missing signed header: content-type"). Restrict the redirect to unsigned browser GET /; signed requests now reach RustFS unchanged.
2026-09-25 16:21:12 +08:00
ryan4yin | 二花 7d8d6ec8ad chore(youko): drop redundant loki-gateway Caddy vhost (#315) 2026-09-25 16:09:44 +08:00
Ryan Yin 7e0e4aa566 feat(youko): replace MinIO with RustFS S3
Deploy RustFS in place of the retired MinIO and move the S3 consumers over.

- flake.nix / flake.lock / outputs/default.nix: add a nixpkgs-small
  (nixos-unstable-small) input for the rustfs 1.0.0 package (with the web
  console), instantiated as pkgs-small in genSpecialArgs alongside the other
  nixpkgs instances; drop the rustfs flake input. The rest of the system stays
  on the default nixpkgs.
- hosts/12kingdoms-youko/homelab-services/rustfs.nix: use nixpkgs' own
  services.rustfs module, with package = pkgs-small.rustfs and
  environmentFile = the agenix rustfs.env secret.
- caddy.nix: s3.writefor.fun (S3 API) and s3-console.writefor.fun (console,
  redirects / to /rustfs/console/).
- secrets/nixos.nix: declare a single rustfs.env (dotenv) secret holding
  RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY, replacing the separate key files.
- infra/minio -> infra/rustfs: buckets are managed with the hashicorp/aws
  provider against the RustFS S3 endpoint; IAM/ILM are applied with rc.
2026-09-25 15:23:50 +08:00
Ryan Yin e401a75ca5 chore(agents): remove crush
Crush hardcodes banned bash commands, blocking many operations. Drop the package, its AGENTS.md/CRUSH.md install target, env vars, and preservation entries.
2026-09-25 00:30:12 +08:00
Ryan Yin 9a6923b4e9 fix(k3s): drop Home Manager from test masters
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
2026-09-24 13:35:33 +08:00
ryan4yin | 二花 069e081fec fix(k3s): bind control-plane metrics to node addresses (#311)
* fix(k3s): bind control-plane metrics to node addresses

* docs(microvm): document remote deployment workflow

* fix(microvm): deploy runner before guest switch

* docs(microvm): standardize on just deployment commands

* docs(k3s): explain control-plane metrics address

* docs(hosts): move generic build commands to host guide

* docs: simplify command operation guidance
2026-09-24 11:13:59 +08:00
Ryan Yin 64cad91401 fix(k8s): unblock the k3s-test-1 workers
kubelet refuses to self-assign the reserved node-role.kubernetes.io/* labels, so the worker --node-label made the agent crash-loop. Drop it and rely on the masters' control-plane taint for placement; document that the taint only lands via registerWithTaints on first registration, so it was applied by hand here.
2026-09-23 12:07:28 +08:00
ryan4yin | 二花 b19818ef44 feat(k8s): add the k3s-test-1 worker nodes (#307)
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
2026-09-21 23:09:13 +08:00
ryan4yin | 二花 242c607ce6 feat(k8s): reserve a VIP block and narrow the tailscale routes (#305)
Reserve 192.168.5.192/27 as a dedicated kube-vip block, outside the router's DHCP pool (.2-.99): one /29 per cluster, k3s-test-1 gets .192/29 with the API VIP at .193.

Advertise only youko itself (.183/32) and that block over Tailscale, instead of the whole /24. The /24 route shadowed hosts that are already reachable directly, e.g. idols-ai at 192.168.5.100.

Deploy prerequisites (not in this repo): approve the new Tailscale routes; point kube-vip at the new VIP; move the *.writefor.fun wildcard from .101 to .183 and update scripts/darwin_set_proxy.py, since .101 is no longer advertised.
2026-09-21 22:28:36 +08:00
ryan4yin | 二花 97c2d1325d docs: refresh stale docs after the VM and backup refactors 2026-09-21 22:03:21 +08:00
Ryan Yin 00aab5f562 feat(computer-use): add a headless X11/i3 environment
Give the idols-ruby and idols-kana VMs a headless computer-use
environment: Xvfb plus i3, driven by cua-driver (and computer-use-linux
as a fallback). Includes the system module (AT-SPI, fonts, Mesa, Clash
Verge proxy, proxy-region timezone), the home i3 module (Xvfb/i3/x11vnc
services, browser flags), host wiring and an eval test.
2026-09-21 21:31:27 +08:00
Ryan Yin 842cd26f64 feat(monitoring): scrape the restic rest-server
Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.
2026-09-21 20:56:01 +08:00
Ryan Yin 096607a5c5 feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
2026-09-21 20:42:39 +08:00
Ryan Yin c3f0dbf8bd revert(backup): drop append-only for the desktop endpoint
A desktop holds both the plaintext and the repository password, so blocking
deletion on the server protects little while preventing the automatic
retention the client needs. Immutability belongs in the offsite copy.
2026-09-21 20:38:08 +08:00
Ryan Yin 5a017666f6 refactor(backup): name the REST endpoint restic.writefor.fun
'backup' did not say what the service is.
2026-09-21 20:38:08 +08:00
Ryan Yin d322f4d2b6 docs: add BACKUP.md, and stagger the restic schedule per host
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
2026-09-21 20:38:08 +08:00