152 Commits
Author SHA1 Message Date
Ryan Yin ac3f775efa chore(nix): drop the nixVersions.latest override
Assisted-by: DeepSeek Harness (claude-opus-5-5)
2026-11-17 23:39:58 +08:00
Ryan Yin 6bd97ddf98 fix(monitoring): require exporter sockets from their services
Assisted-by: DeepSeek Harness (claude-opus-5-5)
2026-11-17 22:43:59 +08:00
ryan4yin | 二花 6b425fe036 feat(monitoring): close host and service monitoring gaps (#530)
Assisted-by: DeepSeek Harness (claude-opus-5-5)
2026-11-17 21:08:02 +08:00
ryan4yin | 二花 667fd76cac feat(nixos): Nix store hygiene, PostgreSQL dumps, OOM and backup quotas (#532)
Assisted-by: DeepSeek Harness (claude-opus-5-5)
2026-11-17 21:07:39 +08:00
ryan4yin | 二花 0dcbbb02e9 feat(canokey): enroll new CanoKeys and retire per-host SSH keys (#524)
Hosts now accept only the two CanoKeys' resident SSH keys for ryan, plus the romantic recovery key for root; per-host user keys stop working. Darwin builds work again: the shared base no longer sets nixpkgs.config next to nix-darwin's nixpkgs.pkgs.

Assisted-by: DeepSeek Harness (claude-opus-5-5)
Assisted-by: DeepSeek Harness (deepseek-flash)
2026-11-11 01:38:12 +08:00
Ryan Yin c4a00e507c fix(nixpak): pass the session TZ through to sandboxes
ICU takes the zone name from the /etc/localtime symlink target path. A bind
mount turns it into a plain file, which keeps the TZif data but loses the
path, so the only identifier left is the ambiguous "CST" abbreviation that
ICU reads from the file and maps to America/Chicago.

Export TZ from time.timeZone and pass TZ/TZDIR through the nixpak env
allowlist.

Assisted-by: DeepSeek Harness (deepseek-flash)
2026-11-04 21:38:18 +08:00
ryan4yin | 二花 13d95daa36 fix(remote-builders): serve x86_64 only, drop aquamarine binfmt (#517)
Assisted-by: DeepSeek Harness (deepseek-flash)
2026-11-04 17:28:30 +08:00
Ryan Yin 4470b3d014 docs: centralize documentation under docs/
Move standalone repository docs from the root into docs/ and add a documentation index that links the moved docs and the code-local runbooks.

Update Markdown links, source comments, and the root entry points to the new locations. Keep README.md and AGENTS.md at the root as required entry points.

Assisted-by: DeepSeek Harness (deepseek-flash)
2026-11-02 17:48:57 +08:00
Ryan Yin 3789a2a282 chore(nixos): guard the nix and xdg-desktop-portal-wlr pins at eval time
WA-023 warns once nixVersions.stable reaches 2.35; WA-046 fails the build
once nixpkgs' xdg-desktop-portal-wlr is no longer the broken 0.8.4, so both
pins get revisited instead of lingering.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 11:55:14 -06:00
Ryan Yin fe280d8fa2 refactor(ssh): drop no-op openFirewall
It is already the sshd module default, and the shared firewall forces
allowedTCPPorts to [ ] anyway; toplevel drvPaths are unchanged.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 11:55:14 -06:00
Ryan Yin ce49b03014 fix(monitoring): socket-activate LAN exporters with FreeBind
Exporters raced networkd for the LAN address at boot; drop the
network-online ordering and unlimited-restart hack and let a FreeBind
socket unit hold the port, passed in via --web.systemd-socket.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 11:55:14 -06:00
Ryan Yin f8ca25a03b refactor(power-monitoring): grant RAPL read access from a udev rule
The boot-time oneshot only fixed the zones present when it ran, and
node_exporter had to Require it. A powercap add rule covers every zone
whenever it appears; the collector reads at scrape time, so no ordering
is needed.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 11:42:57 -06:00
Ryan Yin 8b22aec1bb fix(restic): drop a stale btrfs snapshot before taking a new one
Cleanup runs as ExecStopPost, which a power loss mid-backup skips; the
leftover @restic-snapshot then failed every later run.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 10:51:51 -06:00
Ryan Yin e4c2548a93 refactor: replace redundant mkForce with plain values and mkDefault
Each of these overrode a value nothing else set at a higher priority:

- qemu-guest: btrbk, hardwareTools and cloud-init have no competing
  definition on the guests.
- mdns: nixpkgs' sunshine module sets avahi.publish.* with mkDefault.
- computer-use: i18n now sets the time zone with mkDefault.
- darwin terminal: the shared ghostty/kitty font sizes are now mkDefault.
- shoukei: sunshine is mkDefault false in remote-desktop.nix, zram's
  memoryPercent is mkDefault 50; supportedFilesystems now merges instead
  of discarding the fsType entries other modules add.
- youko: nothing enables docker.
- nixpak: pasta.enable and sockets.pipewire are mkDefault in the shared
  modules, sockets.pulse already was.

Every NixOS host's toplevel drvPath is unchanged; on fern and frieren the
terminal font sizes and time zone evaluate to the same values.

Assisted-by: Claude Code (claude-opus-5-5)
2026-11-01 10:14:27 -06:00
ryan4yin | 二花 0888f87075 feat(power): monitor homelab power and reduce aquamarine idle draw (#501)
Assisted-by: DeepSeek Harness (deepseek-flash)
2026-11-01 20:53:41 +08:00
Ryan Yin 7dc4e7a39f feat(security): harden homelab SSH, services and audit trail
The homelab hosts accept root SSH with long-lived file keys, run several
exposed services with almost no systemd sandboxing, and keep no audit trail of
privileged actions. Close those three gaps, weighing each measure against its
cost instead of adopting a policy wholesale.

SSH
- Turn off KbdInteractiveAuthentication: nixpkgs defaults it to true and PAM
  then still accepts the Unix password despite PasswordAuthentication = false.
- authorizedKeysInHomedir = false: only keys declared in Nix take effect;
  youko keeps Gitea's forced-command file through `Match User gitea`.
- Pin aquamarine's passphrase-free remote-builder keys to
  `restrict,command="nix-daemon --stdio --store daemon"`, so builds run in the
  system daemon instead of an audited root login session.

systemd sandboxing
- Add the mylib.systemdHardening preset and apply it to the exposed, weakly
  sandboxed services: caddy 7.8 -> 1.5, vmalert 8.2 -> 1.4, rustfs 6.6 -> 1.4,
  whisper-asr 7.7 -> 1.6. An eval test pins the key restrictions.
- Rejected: a global all-keys policy (<=0.2 score gains on services nixpkgs
  already sandboxes) and per-service confinement (bind paths to maintain).

auditd on all four hosts, reusing nixpkgs' loader and unit
- Audit login sessions only: a first `-a never,task -F auid=unset` rule keeps
  services out of the per-syscall audit path. A compromised service is
  therefore unrecorded until it opens a session.
- Record root commands, credential and host-key access, writes to SSH, account,
  PAM, Nix-trust and audit configuration, module loads and session denials.
- Rejected: global denial logging, service-UID watches and
  --loginuid-immutable (the kernel already records loginuid changes).

Verified on all four hosts after deployment: zero failed units, k3s 6/6 Ready,
and the exposed services healthy.

Assisted-by: Claude Code (claude-opus-5-5)
2026-10-31 01:09:15 +08:00
ryan4yin | 二花 e8e9691f2d chore(nix): update nixpkgs to e62746b and refresh flake inputs (#488)
* chore(nix): update nixpkgs to e62746b and fix 26.11 breakage

Bump nixpkgs to e62746b and preservation to 00be7f7, then adapt to the removals and renamed options in the new revision: drop the removed boot.initrd.systemd.enable settings, trim the stale emSign CA blacklist on Linux while keeping it for the older Darwin input, and migrate Avahi publishing to settings.publish. Work around sunshine still setting the legacy Avahi options and keep dsh-nu on its tested Nushell 0.115.1. Patch NUR dsh's pnpm call to pass nodejs = null, and update pyclipsync for the stdenv.hostPlatform.system rename.

* chore(nix): refresh flake inputs for 26.11

- update nixpkgs-darwin to the same e62746b revision as nixpkgs
- make dsh-nu follow main nixpkgs and patch its testedWith manifest for the resolved Nushell
- refresh floating inputs; update lanzaboote, tuios, and ponytail
- drop the nixpkgs-2505 and rust-overlay workarounds
- keep pyclipsync on main and simplify the cross-platform CA blacklist
- adopt home-manager's Nushell loadSessionVariables default

* fix(gitea): use ROOT_URL for nixpkgs 26.11

nixpkgs removed the legacy settings.server.DOMAIN option; the existing ROOT_URL already carries the public URL.

* chore(nix): update dsh-nu to 0.7.0

Assisted-by: DeepSeek Harness (deepseek-flash)
2026-10-30 00:39:01 +08:00
Ryan Yin 7e579d6aca fix(monitoring): order nvidia exporter after network-online
The nvidia exporter binds the host's static LAN IPv4, but unlike the node and smartctl exporters it was not ordered after network-online.target. On boot it lost the race with networkd, hit the systemd start limit, and stayed failed until manually restarted.
2026-10-29 21:28:08 +08:00
Ryan Yin 267ccd9d34 fix(firewall): note the one port that is open to the Internet
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-flash)
2026-10-22 00:54:27 +08:00
Ryan Yin 9b68cfdb8a feat(backup): alert when the last successful backup goes stale
The backup unit publishes `restic_backup_last_success_timestamp_seconds` for the node_exporter textfile collector, and a rejected alert now fails the notify unit instead of exiting 0.

Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-flash)
2026-10-22 00:54:27 +08:00
Ryan Yin a3c75250c4 fix(restic): catch up backups missed while a host was off
Assisted-by: DeepSeek Harness 0.2.1-rc.2 (claude-opus-5-5)
2026-10-21 20:59:07 +08:00
Ryan Yin 52573278bd feat(base): install python313 on every NixOS host
Hosts without Home Manager (aquamarine, the VM hosts, k3s guests) had no
interpreter. The system profile is separate from the per-user one, so desktops
keep their Home Manager copy in PATH.

Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-flash)
2026-10-20 00:20:12 +08:00
ryan4yin | 二花 d6450d48a2 feat(remote-builders): build ai and shoukei on aquamarine (#467)
Aquamarine builds for ai and shoukei across the three platforms, with both
daemons logging in as root. They authenticate with their passphrase-free SSH
host keys, and ssh_known_hosts now carries each builder's address.

Assisted-by: DeepSeek Harness 0.2.0-rc.2 (deepseek-flash)
2026-10-17 00:49:22 +08:00
ryan4yin | 二花 9c38819517 docs: review and correct comments and documentation (#436) 2026-10-12 10:02:38 +08:00
ryan4yin | 二花 835b7f6df1 feat(shoukei): pin home Wi-Fi to a static IPv4 & adjust networking.nix (#407) 2026-10-07 21:27:34 +08:00
ryan4yin | 二花 8f065f24bc feat(nix): use Nix 2.35 via nixVersions.latest (#405)
Nix 2.35 brings lazy flake source copying, mimalloc (5-12% faster
evaluation), lower nix-daemon memory use, HTTP/3 support, and jittered
transfer retries with Retry-After.
2026-10-07 18:32:09 +08:00
ryan4yin | 二花 f5a111c638 feat(backup): hourly read-only snapshots and per-backup integrity checks (#402)
* feat(restic): verify the repository after each backup

* feat(youko): snapshot the restic store hourly

* docs(backup): document the snapshot and check layers
2026-10-07 15:26:57 +08:00
ryan4yin | 二花 3acb5d4ee5 fix(nfs,libvirt): restrict the golden export and bind VNC to loopback (#400)
* fix(nfs): restrict the golden export to its real clients

* fix(libvirt): bind VNC consoles to loopback

* fix(firewall): drop VNC ports on every host

* docs(security): record the NFS and VNC boundaries
2026-10-07 15:10:52 +08:00
ryan4yin | 二花 dc65254511 fix(ssh): make sandboxed ssh work on Linux (#390) 2026-10-06 15:20:53 +08:00
ryan4yin | 二花 6259359945 security(monitoring): bind exporters to static LAN addresses (#373)
node_exporter listened on 0.0.0.0, so access control rested entirely on
the nftables allowlist. Bind node/smartctl exporters to the host static
LAN IPv4 (the same address VictoriaMetrics scrapes), keeping the socket
off tailscale/bridge interfaces even if the firewall regresses.

Hosts without a static LAN address (shoukei, akane) now run no exporters
at all instead of loopback-bound ones nobody scrapes; akane, a UTM VM,
previously ran a loopback smartctl exporter with no physical disks.

LAN-bound exporters also get the boot-race treatment from #372
(network-online ordering plus patient restart); loopback-bound hosts
must not pull in network-online because NM wait-online would stall
their offline boots.
2026-10-03 19:07:21 +08:00
ryan4yin | 二花 1b31730db4 fix(monitoring): order smartctl exporter after network-online.target (#372)
The exporter binds to the host LAN address, which systemd-networkd
assigns during boot. Without ordering, the service starts before the
address exists, `bind: cannot assign requested address` trips the
start limit, and the unit stays failed until restarted manually.

Order the unit after network-online.target, and since wait-online is
best-effort (120s timeout), also retry patiently so the unit self-heals
when the link comes up late.
2026-10-03 18:54:08 +08:00
ryan4yin | 二花 37d79fbb08 chore(hardening): consolidate system hardening and fix stale docs (#368) 2026-10-03 17:11:51 +08:00
ryan4yin | 二花 f03907c6a4 feat(security): add balanced kernel and exporter baseline (#365) 2026-10-03 16:33:50 +08:00
ryan4yin | 二花 d056aece28 chore(workarounds): re-evaluate stale hardware/kernel fixes (#360)
* chore(workarounds): re-evaluate stale hardware/kernel fixes

Prune workarounds whose root cause is gone, fix one vendor typo, and
document the ones we keep:

- drop `transparent_hugepage=madvise` on shoryu/shushou/youko: it was tied to
  the old fixed 1G hugetlb pool, and the VMs now use ordinary memory
- shoryu: `hardware.cpu.intel.updateMicrocode` -> `amd` (it is an AMD APU)
- ai: drop `i915.enable_guc=3` (GuC/HuC submission is the default on modern
  i915/Xe)
- kernel-hardening: name the Dirty Frag CVEs (CVE-2026-43284 / CVE-2026-43500)
  and the re-evaluation condition
- document re-eval conditions for the Insta360 resume fix, the darwin
  auto-optimise-store workaround, and the youko USB-SATA bridge mitigations

* chore(workarounds): keep the Dirty Frag module blacklist as defence in depth

* chore(ai): correct the i915.enable_guc rationale (verified against the kernel)

* chore(darwin): drop the auto-optimise-store workaround (nix#7273 fixed)
2026-10-03 11:25:34 +08:00
ryan4yin | 二花 dc3ec8c159 feat(storage): relax ai NVMe workarounds and add drive SMART monitoring (#359)
* fix(ai): relax NVMe power-management workarounds and enable smartd

Logs show no NVMe errors, so allow APST again (capped at 100 ms), restore
the default io_timeout, and drop pcie_aspm=off. Add smartd for drive
health visibility.

* feat(storage): add drive/SSD SMART monitoring on physical hosts

node_exporter's nvme collector only exposes node_nvme_info and namespace
capacity, not SMART health, so SSD wear/lifespan and drive temperature were
not observable anywhere.

- enable services.prometheus.exporters.smartctl on physical hosts, gated on
  modules.hardwareTools (off on QEMU/MicroVM guests), bound to the host's LAN
  address on port 9633
- firewall: drop 9633 from anything but the monitoring host
- youko: scrape smartctl-<host> for the physical hosts
- alert rules (storage.yml): SMART health, NVMe wear, available spare, media
  errors, critical warning, and drive temperature (warn 70 / crit 80 C)
- Grafana dashboard "Drive SMART (smartctl)"
2026-10-03 11:13:46 +08:00
ryan4yin | 二花 51261645ca chore: small audit cleanups (#357)
Remove a dead .gitattributes rule, bump the mirror workflow's checkout,
document the daemon test settings, drop unused JACK, and align the
installer's disko version.
2026-10-03 10:32:08 +08:00
ryan4yin | 二花 885dabb49d refactor: scope host tools, locate and mDNS by purpose (#356) 2026-10-03 10:23:21 +08:00
ryan4yin | 二花 ce416240b2 fix: avoid mkForce for allowUnfree and boot timeout (#355)
Let hosts override these again and drop the forced priorities.
2026-10-03 10:14:50 +08:00
ryan4yin | 二花 42641e2fdf feat!: replace zellij with tuios, default to ghostty + kitty (#349)
* feat!: replace zellij with tuios and switch to ghostty + kitty

Terminal environment overhaul:

- add the tuios flake input (upstream 0.8.5; nixpkgs has 0.8.1) and link
  the checked-in config out-of-store; auto-start tuios in nushell instead
  of zellij
- remove zellij entirely: both modules, config, README and nushell
  completions
- remove foot and alacritty, which lack the Kitty graphics protocol;
2026-10-03 02:34:31 +08:00
Ryan Yin facf33615e fix(nixos): install terminfo only for terminals in use
environment.enableAllTerminfo installs every terminal's terminfo, including rxvt-unicode, which no longer builds with GCC 16 (NixOS/nixpkgs#568896). Install just the terminals we use (alacritty, foot, ghostty, kitty) instead. This also makes the aarch64 jasper workaround unnecessary.
2026-10-03 00:14:00 +08:00
ryan4yin | 二花 4909f6357b security: scope privileges and stop a world-readable secret copy (#335) 2026-10-01 21:18:34 +08:00
ryan4yin | 二花 ab2fd0fee9 feat: add plocate filesystem search and cap podman log size (#333) 2026-10-01 20:00:53 +08:00
ryan4yin | 二花 8f9fdda2bb refactor: trim unused virtualisation config and packages (#332) 2026-10-01 19:42:20 +08:00
ryan4yin | 二花 9ab5eca60c feat(monitoring): GPU-aware btop variants and GPU metrics (#330) 2026-09-30 00:16:50 +08:00
ryan4yin | 二花 e5b95affa6 feat(nixos): default-deny host inbound, trust the LAN/tailnet (#325)
Hosts used to trust the router as the only boundary (servers ran with the
firewall off; service ports were opened globally), so any LAN device could
reach every host service. Move filtering to the hosts as defence in depth
behind the router, which keeps its firewall on and uses a port mapping for
the few ports we actually need.

- base: nftables backend; deny inbound by default; trust the home LAN by
  source and the tailnet by interface (restrict the tailnet with ACLs);
  suppress per-service `openFirewall` global ports; allow node_exporter only
  from the monitoring host. The local podman bridge, the tailnet and the pod
  CIDRs are accepted only after the node_exporter drop.
- servers: stop disabling the firewall; drop redundant global port opens
  (caddy 80/443, NFS 2049, Jellyfin, localsend/test ports); delete the
  redundant desktop firewall module; aarch64/riscv64 import the firewall
  explicitly (akane was missing it).
- k3s: Cilium pods (10.0.0.0/8, fd00::/8) may reach host services; drop the
  flannel-era cni0/flannel.1 trust.
- youko: postgres listens on localhost + the LAN with pg_hba limited to the
  LAN; the NFS export stays LAN-scoped; remove the unauthenticated v2ray
  proxy.
- tests: `security-firewall` asserts every host runs the shared firewall, has
  the LAN CIDR rule and no global ports.
2026-09-26 14:02:03 +08:00
ryan4yin | 二花 5ee8b728b5 fix(restic): use the correct systemd timer option names (#319)
timerConfig was set with lowercase onCalendar/randomizedDelaySec, which systemd treats as unknown directives. The generated timer had no schedule, so `systemctl list-timers` showed NEXT=- and no backup ever ran. Use the real keys OnCalendar/RandomizedDelaySec.
2026-09-25 20:39:59 +08:00
ryan4yin | 二花 97c2d1325d docs: refresh stale docs after the VM and backup refactors 2026-09-21 22:03:21 +08:00
ryan4yin | 二花 ef00eb31e0 docs: warn that user-readable decrypted secrets must not be read (#303)
Only three secrets are readable as the login user (nix-access-tokens,
ssh-key-romantic and the nushell one); everything else is root-only and
the filesystem already keeps agents out.
2026-09-21 21:18:06 +08:00
Ryan Yin 096607a5c5 feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
2026-09-21 20:42:39 +08:00
Ryan Yin d322f4d2b6 docs: add BACKUP.md, and stagger the restic schedule per host
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
2026-09-21 20:38:08 +08:00