Commit Graph
130 Commits
Author SHA1 Message Date
ryan4yin | 二花 d6450d48a2 feat(remote-builders): build ai and shoukei on aquamarine (#467)
Aquamarine builds for ai and shoukei across the three platforms, with both
daemons logging in as root. They authenticate with their passphrase-free SSH
host keys, and ssh_known_hosts now carries each builder's address.

Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
2026-10-17 00:49:22 +08:00
ryan4yin | 二花 9c38819517 docs: review and correct comments and documentation (#436) 2026-10-12 10:02:38 +08:00
ryan4yin | 二花 835b7f6df1 feat(shoukei): pin home Wi-Fi to a static IPv4 & adjust networking.nix (#407) 2026-10-07 21:27:34 +08:00
ryan4yin | 二花 8f065f24bc feat(nix): use Nix 2.35 via nixVersions.latest (#405)
Nix 2.35 brings lazy flake source copying, mimalloc (5-12% faster
evaluation), lower nix-daemon memory use, HTTP/3 support, and jittered
transfer retries with Retry-After.
2026-10-07 18:32:09 +08:00
ryan4yin | 二花 f5a111c638 feat(backup): hourly read-only snapshots and per-backup integrity checks (#402)
* feat(restic): verify the repository after each backup

* feat(youko): snapshot the restic store hourly

* docs(backup): document the snapshot and check layers
2026-10-07 15:26:57 +08:00
ryan4yin | 二花 3acb5d4ee5 fix(nfs,libvirt): restrict the golden export and bind VNC to loopback (#400)
* fix(nfs): restrict the golden export to its real clients

* fix(libvirt): bind VNC consoles to loopback

* fix(firewall): drop VNC ports on every host

* docs(security): record the NFS and VNC boundaries
2026-10-07 15:10:52 +08:00
ryan4yin | 二花 dc65254511 fix(ssh): make sandboxed ssh work on Linux (#390) 2026-10-06 15:20:53 +08:00
ryan4yin | 二花 6259359945 security(monitoring): bind exporters to static LAN addresses (#373)
node_exporter listened on 0.0.0.0, so access control rested entirely on
the nftables allowlist. Bind node/smartctl exporters to the host static
LAN IPv4 (the same address VictoriaMetrics scrapes), keeping the socket
off tailscale/bridge interfaces even if the firewall regresses.

Hosts without a static LAN address (shoukei, akane) now run no exporters
at all instead of loopback-bound ones nobody scrapes; akane, a UTM VM,
previously ran a loopback smartctl exporter with no physical disks.

LAN-bound exporters also get the boot-race treatment from #372
(network-online ordering plus patient restart); loopback-bound hosts
must not pull in network-online because NM wait-online would stall
their offline boots.
2026-10-03 19:07:21 +08:00
ryan4yin | 二花 1b31730db4 fix(monitoring): order smartctl exporter after network-online.target (#372)
The exporter binds to the host LAN address, which systemd-networkd
assigns during boot. Without ordering, the service starts before the
address exists, `bind: cannot assign requested address` trips the
start limit, and the unit stays failed until restarted manually.

Order the unit after network-online.target, and since wait-online is
best-effort (120s timeout), also retry patiently so the unit self-heals
when the link comes up late.
2026-10-03 18:54:08 +08:00
ryan4yin | 二花 37d79fbb08 chore(hardening): consolidate system hardening and fix stale docs (#368) 2026-10-03 17:11:51 +08:00
ryan4yin | 二花 f03907c6a4 feat(security): add balanced kernel and exporter baseline (#365) 2026-10-03 16:33:50 +08:00
ryan4yin | 二花 d056aece28 chore(workarounds): re-evaluate stale hardware/kernel fixes (#360)
* chore(workarounds): re-evaluate stale hardware/kernel fixes

Prune workarounds whose root cause is gone, fix one vendor typo, and
document the ones we keep:

- drop `transparent_hugepage=madvise` on shoryu/shushou/youko: it was tied to
  the old fixed 1G hugetlb pool, and the VMs now use ordinary memory
- shoryu: `hardware.cpu.intel.updateMicrocode` -> `amd` (it is an AMD APU)
- ai: drop `i915.enable_guc=3` (GuC/HuC submission is the default on modern
  i915/Xe)
- kernel-hardening: name the Dirty Frag CVEs (CVE-2026-43284 / CVE-2026-43500)
  and the re-evaluation condition
- document re-eval conditions for the Insta360 resume fix, the darwin
  auto-optimise-store workaround, and the youko USB-SATA bridge mitigations

* chore(workarounds): keep the Dirty Frag module blacklist as defence in depth

* chore(ai): correct the i915.enable_guc rationale (verified against the kernel)

* chore(darwin): drop the auto-optimise-store workaround (nix#7273 fixed)
2026-10-03 11:25:34 +08:00
ryan4yin | 二花 dc3ec8c159 feat(storage): relax ai NVMe workarounds and add drive SMART monitoring (#359)
* fix(ai): relax NVMe power-management workarounds and enable smartd

Logs show no NVMe errors, so allow APST again (capped at 100 ms), restore
the default io_timeout, and drop pcie_aspm=off. Add smartd for drive
health visibility.

* feat(storage): add drive/SSD SMART monitoring on physical hosts

node_exporter's nvme collector only exposes node_nvme_info and namespace
capacity, not SMART health, so SSD wear/lifespan and drive temperature were
not observable anywhere.

- enable services.prometheus.exporters.smartctl on physical hosts, gated on
  modules.hardwareTools (off on QEMU/MicroVM guests), bound to the host's LAN
  address on port 9633
- firewall: drop 9633 from anything but the monitoring host
- youko: scrape smartctl-<host> for the physical hosts
- alert rules (storage.yml): SMART health, NVMe wear, available spare, media
  errors, critical warning, and drive temperature (warn 70 / crit 80 C)
- Grafana dashboard "Drive SMART (smartctl)"
2026-10-03 11:13:46 +08:00
ryan4yin | 二花 51261645ca chore: small audit cleanups (#357)
Remove a dead .gitattributes rule, bump the mirror workflow's checkout,
document the daemon test settings, drop unused JACK, and align the
installer's disko version.
2026-10-03 10:32:08 +08:00
ryan4yin | 二花 885dabb49d refactor: scope host tools, locate and mDNS by purpose (#356) 2026-10-03 10:23:21 +08:00
ryan4yin | 二花 ce416240b2 fix: avoid mkForce for allowUnfree and boot timeout (#355)
Let hosts override these again and drop the forced priorities.
2026-10-03 10:14:50 +08:00
ryan4yin | 二花 42641e2fdf feat!: replace zellij with tuios, default to ghostty + kitty (#349)
* feat!: replace zellij with tuios and switch to ghostty + kitty

Terminal environment overhaul:

- add the tuios flake input (upstream 0.8.5; nixpkgs has 0.8.1) and link
  the checked-in config out-of-store; auto-start tuios in nushell instead
  of zellij
- remove zellij entirely: both modules, config, README and nushell
  completions
- remove foot and alacritty, which lack the Kitty graphics protocol;
2026-10-03 02:34:31 +08:00
Ryan Yin facf33615e fix(nixos): install terminfo only for terminals in use
environment.enableAllTerminfo installs every terminal's terminfo, including rxvt-unicode, which no longer builds with GCC 16 (NixOS/nixpkgs#568896). Install just the terminals we use (alacritty, foot, ghostty, kitty) instead. This also makes the aarch64 jasper workaround unnecessary.
2026-10-03 00:14:00 +08:00
ryan4yin | 二花 4909f6357b security: scope privileges and stop a world-readable secret copy (#335) 2026-10-01 21:18:34 +08:00
ryan4yin | 二花 ab2fd0fee9 feat: add plocate filesystem search and cap podman log size (#333) 2026-10-01 20:00:53 +08:00
ryan4yin | 二花 8f9fdda2bb refactor: trim unused virtualisation config and packages (#332) 2026-10-01 19:42:20 +08:00
ryan4yin | 二花 9ab5eca60c feat(monitoring): GPU-aware btop variants and GPU metrics (#330) 2026-09-30 00:16:50 +08:00
ryan4yin | 二花 e5b95affa6 feat(nixos): default-deny host inbound, trust the LAN/tailnet (#325)
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.

- base: nftables backend; deny inbound by default; trust the home LAN by
  source and the tailnet by interface (restrict the tailnet with ACLs);
  suppress per-service `openFirewall` global ports; allow node_exporter only
  from the monitoring host. The local podman bridge, the tailnet and the pod
  CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
  (caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
  redundant desktop firewall module; aarch64/riscv64 import the firewall
  explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
  flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
  LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
  proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
  the LAN CIDR rule and no global ports.
2026-09-26 14:02:03 +08:00
ryan4yin | 二花 5ee8b728b5 fix(restic): use the correct systemd timer option names (#319)
timerConfig was set with lowercase onCalendar/randomizedDelaySec, which systemd treats as unknown directives. The generated timer had no schedule, so `systemctl list-timers` showed NEXT=- and no backup ever ran. Use the real keys OnCalendar/RandomizedDelaySec.
2026-09-25 20:39:59 +08:00
ryan4yin | 二花 97c2d1325d docs: refresh stale docs after the VM and backup refactors 2026-09-21 22:03:21 +08:00
ryan4yin | 二花 ef00eb31e0 docs: warn that user-readable decrypted secrets must not be read (#303)
Only three secrets are readable as the login user (nix-access-tokens,
ssh-key-romantic and the nushell one); everything else is root-only and
the filesystem already keeps agents out.
2026-09-21 21:18:06 +08:00
Ryan Yin 096607a5c5 feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
2026-09-21 20:42:39 +08:00
Ryan Yin d322f4d2b6 docs: add BACKUP.md, and stagger the restic schedule per host
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
2026-09-21 20:38:08 +08:00
Ryan Yin 2ff93c16ce feat(restic): move idols-ai's off-host backup to the REST server
Split the repository password into desktop and homelab groups, add the
environmentFile/pruneOpts module options, and point the desktop at youko's
REST endpoint.
2026-09-21 20:38:08 +08:00
ryan4yin | 二花 e37042f61c feat(youko): restic backup of the service state to the HDD (part 1) (#299)
* feat(hosts): per-host restic backups to the SMB remote

Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:

- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
  (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
  excludes for the regenerable/bulky paths (podman overlay layers, microvm
  and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
  and dumps all postgres databases first (a consistent SQL artifact beside
  the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token

Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.

* refactor(hosts): only restic does off-host; btrbk stays local

btrbk works at the subvolume level, so a  would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.

So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.

Update the btrbk module's comment to say so.

* fix(restic): never back up keys; drop the implicit /etc/agenix

restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).

* refactor(restic): make the backup a proper module

Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.

* refactor(restic): exclude the whole podman storage, trim retention

Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.

Retention: daily 3, weekly 2, monthly 2 - no yearly.

* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump

A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.

* feat(secrets): restic-password for modules.restic-backup

With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.

* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix

* fix(restic): take the password path from agenix, not a hardcoded one

config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
2026-09-21 19:00:35 +08:00
Ryan Yin 47bf47f41d chore(desktop): drop duplicate applets and tune the noctalia bar 2026-09-20 01:35:20 +08:00
Ryan Yin ccaed6beff chore(i18n): keep only China-specific locale overrides
English stays the default (LANG=en_US.UTF-8). Drop the Chinese overrides for
LC_TIME, LC_NUMERIC, LC_ADDRESS, LC_NAME, LC_TELEPHONE and LC_IDENTIFICATION:
LC_NUMERIC matches en_US, the others are effectively unused, and LC_TIME forced
Chinese date and weekday names.

Keep zh_CN only where the en_US default is wrong for China: metric units, A4
paper and CNY.
2026-09-19 16:15:35 +08:00
Ryan Yin 2dff4f062b fix(networking): disable systemd-resolved mDNS when avahi is enabled
networkd enables systemd-resolved by default, and resolved's own mDNS responder conflicts with avahi (duplicate answers, avahi logs 'Detected another mDNS stack running on this host'). Keep avahi for DNS-SD (cups-browsed depends on it) and turn resolved's mDNS off.
2026-09-19 13:54:19 +08:00
Ryan Yin a8ea040c32 fix(trash): persist the cleanup timer
trash-empty lacked Persistent=true, so a host that was off or suspended at the scheduled time never ran the purge and trash retention silently stopped.
2026-09-19 12:11:18 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin f766dcc24f fix(security): tighten desktop remote access 2026-09-01 20:20:51 +08:00
Ryan Yin 13436c0c31 fix(security): remove container daemon group access 2026-09-01 19:17:39 +08:00
Ryan Yin 4b7375c077 fix(nixos): gate zram kernel sysctl tunings behind modules.zram.enable
The zram module was auto-imported on all hosts via base scanPaths, and its
swappiness=180/page-cluster=0 tunings leaked onto hosts that disabled zram
(e.g. idols-ai with a disk swapfile, pushing cold anon pages to disk swap).
Wrap the whole module in modules.zram.enable (default true); idols-ai now
turns off zram and its tunings with a single switch.
2026-08-29 11:16:12 +08:00
ryan4yin | 二花 21d64e7add security: hardening pass 1 (AppArmor, firewall default, loopback metrics, k3s perms, SSH X11) (#266)
* security: make firewall secure-by-default, disable explicitly on servers

Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.

* test: fix security-firewall eval test to scalar per-host values

* security: enable AppArmor (complain) on all Linux hosts

* security: bind aquamarine metrics exporters to loopback

* security: restrict k3s kubeconfig file mode to 600

* security: disable SSH X11 forwarding on servers, keep on desktops

* security: add conservative systemd hardening to aquamarine services

* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)

nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.

* fix(aquamarine): scrape same-host exporters over loopback

The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.

* Revert "security: add conservative systemd hardening to aquamarine services"

This reverts commit f9cf99dadb.
2026-08-27 00:13:01 +08:00
ryan4yin | 二花 6e0d44c50d feat(nixos): manage scattered trash dirs with trash-cli retention timer (#265)
* feat(nushell): add trash command using the home trash can

On a tmpfs root with persistent dirs bind-mounted in, `rm --trash`
scatters items into per-mount .Trash-$uid dirs that file managers never
show, and gio refuses to trash on those internal mounts.

Add a `trash` command implementing the freedesktop trash spec
'failsafe' mode: items are moved to ~/.local/share/Trash with
spec-compliant .trashinfo entries, so Thunar can list and restore them.

Also preserve ~/.local/share/Trash on idols-ai so the trash survives
reboots.

* feat(nixos): manage scattered trash dirs with trash-cli retention timer

Replace the hand-written nushell trash command with the established
trash-cli tooling. On a tmpfs root with persistent dirs bind-mounted
in, the trash crate (nushell rm --trash) scatters items into per-mount
.Trash-$uid dirs that file managers never show and nothing cleans up.

- add trash-cli to system packages: trash-list scans/trash-restore
  handles every mount point's trash dir
- daily systemd timer runs 'trash-empty 30 -f', purging items older
  than 30 days across the home trash and all mount points
- drop the hand-written trash.nu module
- note in preservation.nix: do NOT persist ~/.local/share/Trash; a
  bind-mounted trash dir breaks the trash crate's home-topdir match
  for files straight under $HOME (it would try /.Trash-$uid, EACCES)

* fix(nixos): order trash-empty after preservation.target

The scattered .Trash-$uid dirs live inside the preservation bind
mounts, which use DefaultDependencies=no and are therefore NOT ordered
after local-fs.target. Add preservation.target to After= so trash-empty
only runs once all those mounts are up.
2026-08-23 23:40:53 +08:00
Ryan Yin 0fe12bef5a fix(nix): preserve default sandbox shell
Setting nix.settings.sandbox-paths replaces Nix compiled sandbox defaults. Keeping only /dev/net made local builds lose the sandbox shell that normally provides /bin/sh, which broke upstream scripts with /bin/sh shebangs in packages such as X11-fonts and openldap.

Use extra-sandbox-paths for /dev/net instead. This keeps Nix daemon defaults intact, including the configured sandbox shell, while still allowing builds that need /dev/net.

Add eval coverage for both Linux output sets so future changes keep /dev/net in extra-sandbox-paths and do not reintroduce an explicit sandbox-paths override.
2026-07-06 23:04:16 +08:00
Ryan Yin 8462d34ade feat: NixOS tests using systemd-nspawn containers 2026-06-22 09:02:06 +08:00
Ryan Yin 662316c18a Nixos 26.05 (#261)
* flake.lock: Update

Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/7aaa00e7cc9be6c316cb5f6617bd740dd435c59d?narHash=sha256-WacE23EbHTsBKvr8cu%2B1DFNbP6Rh1brHUH5SDUI0NQI%3D' (2026-04-30)
  → 'github:NixOS/nixpkgs/e9a7635a57597d9754eccebdfc7045e6c8600e6b?narHash=sha256-u6WU/yd/o8iYQrHX3RAwO1hYa3LkoSL%2BWNQD0rJfJZQ%3D' (2026-05-29)

* feat: upgrade to nixos 26.05
2026-06-01 10:02:49 +08:00
Ryan Yin 6a12b600a1 fix: mitigate Dirty Frag LPE vulnerabilities 2026-05-14 11:54:36 +08:00
Ryan Yin b143a89443 feat(ai): add webdav mount (#253)
feat(aquamarine): add group for filesharing, protect /data on subvolume mount failures
2026-03-19 22:25:48 +08:00
Ryan Yin f6aa45aa00 feat: idols-ai - add new data disk via disko (#250)
feat: luks automatic unlock via tpm2 chip for all physical hosts
feat: enable login manager
fix: Rename network interface config back following hardware change
2026-03-18 22:26:08 +08:00
Ryan Yin 850a7b2c43 feat: add nvme-cli 2026-03-16 09:25:28 +08:00
Ryan Yin 7e7f5a796e feat: add pv & smartctl 2026-03-16 02:21:18 +08:00
Ryan Yin 69b4d96915 feat: idols-akane - use docker instead of podman 2026-03-10 23:45:55 +08:00
Ryan Yin 4211d18ad1 fix: base config should not relys on agenix secrets 2026-03-08 00:25:03 +08:00