Commit Graph
2472 Commits
Author SHA1 Message Date
ryan4yin | 二花 7d8d6ec8ad chore(youko): drop redundant loki-gateway Caddy vhost (#315) 2026-09-25 16:09:44 +08:00
ryan4yin | 二花 e8e7844051 Merge pull request #314 from ryan4yin/feat/rustfs-s3
feat(youko): replace MinIO with RustFS S3
2026-09-25 15:53:40 +08:00
Ryan Yin 7e0e4aa566 feat(youko): replace MinIO with RustFS S3
Deploy RustFS in place of the retired MinIO and move the S3 consumers over.

- flake.nix / flake.lock / outputs/default.nix: add a nixpkgs-small
  (nixos-unstable-small) input for the rustfs 1.0.0 package (with the web
  console), instantiated as pkgs-small in genSpecialArgs alongside the other
  nixpkgs instances; drop the rustfs flake input. The rest of the system stays
  on the default nixpkgs.
- hosts/12kingdoms-youko/homelab-services/rustfs.nix: use nixpkgs' own
  services.rustfs module, with package = pkgs-small.rustfs and
  environmentFile = the agenix rustfs.env secret.
- caddy.nix: s3.writefor.fun (S3 API) and s3-console.writefor.fun (console,
  redirects / to /rustfs/console/).
- secrets/nixos.nix: declare a single rustfs.env (dotenv) secret holding
  RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY, replacing the separate key files.
- infra/minio -> infra/rustfs: buckets are managed with the hashicorp/aws
  provider against the RustFS S3 endpoint; IAM/ILM are applied with rc.
2026-09-25 15:23:50 +08:00
Ryan Yin e401a75ca5 chore(agents): remove crush
Crush hardcodes banned bash commands, blocking many operations. Drop the package, its AGENTS.md/CRUSH.md install target, env vars, and preservation entries.
2026-09-25 00:30:12 +08:00
Ryan Yin 310d59b437 flake.lock: Update
Flake lock file updates:

• Updated input 'mysecrets':
    'git+ssh://git@github.com/ryan4yin/nix-secrets.git?ref=refs/heads/main&rev=8533d18ee50e6e5cddd3cd3decfa6ac54f116472&shallow=1' (2026-09-23)
  → 'git+ssh://git@github.com/ryan4yin/nix-secrets.git?ref=refs/heads/main&rev=7e75aad771e8748f666a83832678f3fbb6378a73&shallow=1' (2026-09-24)
2026-09-25 00:30:12 +08:00
ryan4yin | 二花 c3a8d99ceb Merge pull request #313 from ryan4yin/fix/k3s-test-1-master-config-cleanup
fix(k3s): drop Home Manager from test masters
2026-09-24 14:12:04 +08:00
Ryan Yin 9a6923b4e9 fix(k3s): drop Home Manager from test masters
Keep the six node configuration intact while removing the failing user activation from the three control-plane MicroVMs. Update the regression test and current K3s configuration notes.
2026-09-24 13:35:33 +08:00
ryan4yin | 二花 f5f4531fcb Merge pull request #312 from ryan4yin/fix/k3s-test-1-guest-dns
fix(k3s): stabilize guest networking and control-plane metrics
2026-09-24 11:36:41 +08:00
ryan4yin | 二花 882383f754 Merge pull request #309 from ryan4yin/fix/k3s-test-1-stable-ipv6
fix(k3s): disable temporary IPv6 addresses in test VMs
2026-09-24 11:29:23 +08:00
ryan4yin | 二花 069e081fec fix(k3s): bind control-plane metrics to node addresses (#311)
* fix(k3s): bind control-plane metrics to node addresses

* docs(microvm): document remote deployment workflow

* fix(microvm): deploy runner before guest switch

* docs(microvm): standardize on just deployment commands

* docs(k3s): explain control-plane metrics address

* docs(hosts): move generic build commands to host guide

* docs: simplify command operation guidance
2026-09-24 11:13:59 +08:00
Ryan Yin c752703488 docs(network): explain DHCPv6 DNS overrides 2026-09-24 10:43:49 +08:00
Ryan Yin 688a244ae0 fix(k3s): ignore advertised DNS on test VMs 2026-09-23 23:44:13 +08:00
Ryan Yin f357569d32 fix(k3s): disable IPv6 privacy on guest NICs 2026-09-23 23:05:53 +08:00
Ryan Yin fa0743a28b fix(k3s): disable temporary IPv6 addresses in test VMs 2026-09-23 22:28:35 +08:00
Ryan Yin d1dd35bd7c chore(codex): bump the override to 0.156.1
0.156.1 still lacks the codex-chatgpt recursion_limit attribute, so keep the postPatch that adds it. The vendored Cargo.lock is unchanged, so the cargoHash stays the same.
2026-09-23 12:07:28 +08:00
Ryan Yin 55aa4e3e10 fix(k8s): link the CNI dirs on the k3s agent nodes
cilium writes /etc/cni/net.d while istio-cni reads k3s's own dir, so on the new workers the chained plugin never found a network config. Add the symlinks the server module already has.
2026-09-23 12:07:28 +08:00
Ryan Yin 64cad91401 fix(k8s): unblock the k3s-test-1 workers
kubelet refuses to self-assign the reserved node-role.kubernetes.io/* labels, so the worker --node-label made the agent crash-loop. Drop it and rely on the masters' control-plane taint for placement; document that the taint only lands via registerWithTaints on first registration, so it was applied by hand here.
2026-09-23 12:07:28 +08:00
Ryan Yin a3efb42202 chore(k8s): update secrets 2026-09-23 12:07:28 +08:00
Ryan Yin a2604a0c32 chore(codex): override to 0.156.0 pending llm-agents bump 2026-09-23 10:37:52 +08:00
Ryan Yin f6e331dbf7 flake.lock: Update
Flake lock file updates:

• Updated input 'llm-agents':
    'github:numtide/llm-agents.nix/34081a176242cbccd62f19978f32abfff933813f?narHash=sha256-5QvQZFsM/xcZRVUGuv56QU3d0RQZcqR9axikxGf1KLE%3D' (2026-09-19)
  → 'github:numtide/llm-agents.nix/fcf80443a6e5363f30dc9b191bcb65cfcd788df2?narHash=sha256-uWgg4A%2BhrlViz1L0y9D8zMkIjQU8UzJqoyxaZ3xY24Q%3D' (2026-09-23)
• Updated input 'llm-agents/bun2nix':
    'github:Mic92/bun2nix/5765b0614591f75ee8ba5596e81ae85c167d1071?narHash=sha256-mEq2kU%2BljompTToZ44Afvm7d/rHJ5iMBl0tjZuyShJs%3D' (2026-08-29)
  → 'github:Mic92/bun2nix/07a5bfc8ac36c5370199343fa620b69f63186e33?narHash=sha256-1/SLBjSYBzDdDhvLQ83wrzBakyS/xNGgtxHGNuveOEQ%3D' (2026-09-21)
• Updated input 'llm-agents/nixpkgs':
    'github:NixOS/nixpkgs/c7def046b9a883d46974757852106483d741586f?narHash=sha256-6RSEDHIWQtesQKWSu5qRai8L2h4KgCgMEfJHstW99G4%3D' (2026-09-14)
  → 'github:NixOS/nixpkgs/b6c98e9e6633ee64753b594ff4a5febf0367fc00?narHash=sha256-QpXotokGh5eeMG1Njiuh27oP0rK6t5%2BcMp7ymXSs7mc%3D' (2026-09-22)
2026-09-23 09:50:10 +08:00
ryan4yin | 二花 3b13291216 feat(computer-use): sync the X clipboard and document the input-level design (#308)
- run two autocutsel instances so the X PRIMARY/CLIPBOARD selections stay in
  sync, making copy/paste work in the session and through VNC
- drop --disable-blink-features=AutomationControlled: nothing drives the
  browser over CDP, so navigator.webdriver is already false and the flag only
  marked the setup as automated
- document why the driver stays on real X11/AT-SPI input and is never granted
  a CDP attach, and note the SSH-tunnel path for manual VNC access
- extend the computer-use eval test
2026-09-22 20:45:09 +08:00
ryan4yin | 二花 b19818ef44 feat(k8s): add the k3s-test-1 worker nodes (#307)
Add worker-1 (4 vCPU / 16 GiB) on shoryu, and worker-2 (4 vCPU / 16 GiB) plus worker-3 (2 vCPU / 8 GiB) on shushou, as microVMs. They come with a genK3sAgentModule and the per-host/output/home-manager wiring, and are labelled node-role.kubernetes.io/worker=true. The masters keep their control-plane:NoSchedule taint, so workloads land on the workers.
2026-09-21 23:09:13 +08:00
ryan4yin | 二花 ca00d26642 chore: drop the unused darwin proxy set script (#306)
The script pointed the macOS nix-daemon at the homelab v2ray HTTP proxy (192.168.5.101:7890), but that proxy was never running. Remove it together with the just darwin-set-proxy recipe and the README step that referenced it.
2026-09-21 22:33:33 +08:00
ryan4yin | 二花 242c607ce6 feat(k8s): reserve a VIP block and narrow the tailscale routes (#305)
Reserve 192.168.5.192/27 as a dedicated kube-vip block, outside the router's DHCP pool (.2-.99): one /29 per cluster, k3s-test-1 gets .192/29 with the API VIP at .193.

Advertise only youko itself (.183/32) and that block over Tailscale, instead of the whole /24. The /24 route shadowed hosts that are already reachable directly, e.g. idols-ai at 192.168.5.100.

Deploy prerequisites (not in this repo): approve the new Tailscale routes; point kube-vip at the new VIP; move the *.writefor.fun wildcard from .101 to .183 and update scripts/darwin_set_proxy.py, since .101 is no longer advertised.
2026-09-21 22:28:36 +08:00
ryan4yin | 二花 97c2d1325d docs: refresh stale docs after the VM and backup refactors 2026-09-21 22:03:21 +08:00
ryan4yin | 二花 5d0055b024 Merge pull request #277 from ryan4yin/feat/cua-headless-sway
feat(computer-use): headless X11/i3 + cua-driver on ruby and kana
2026-09-21 21:43:45 +08:00
Ryan Yin 0185fd0f85 docs(computer-use): document the headless i3 session 2026-09-21 21:31:27 +08:00
Ryan Yin 00aab5f562 feat(computer-use): add a headless X11/i3 environment
Give the idols-ruby and idols-kana VMs a headless computer-use
environment: Xvfb plus i3, driven by cua-driver (and computer-use-linux
as a fallback). Includes the system module (AT-SPI, fonts, Mesa, Clash
Verge proxy, proxy-region timezone), the home i3 module (Xvfb/i3/x11vnc
services, browser flags), host wiring and an eval test.
2026-09-21 21:31:27 +08:00
Ryan Yin 90761a6b73 feat(computer-use): package computer-use-linux and cua-driver
Package the prebuilt releases (computer-use-linux v0.7.1 and
cua-driver-rs v0.24.0) under pkgs/, injected through thin callPackage
overlays so they are easy to relocate to the NUR repository later.
2026-09-21 21:31:27 +08:00
ryan4yin | 二花 ef00eb31e0 docs: warn that user-readable decrypted secrets must not be read (#303)
Only three secrets are readable as the login user (nix-access-tokens,
ssh-key-romantic and the nushell one); everything else is root-only and
the filesystem already keeps agents out.
2026-09-21 21:18:06 +08:00
ryan4yin | 二花 260da1ee20 chore: rename the nushell secret, and forbid reading decrypted secrets 2026-09-21 21:13:50 +08:00
ryan4yin | 二花 bc6748d021 feat(backup): notify on failure, and 404 unconfigured hostnames (#301)
* feat(backup): notify on failure, and 404 unconfigured hostnames

A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.

* docs(backup): say 'cloud copy' rather than 'offsite'

Clearer for a reader who does not live in backup jargon; the requirement
(not in the homelab, immutable) is spelled out in the text.

* docs(secrets): state the recipient invariant

A recipient set must answer two questions: who may read it, and how it is
recovered once those hosts are gone. Dropping recovery_key while narrowing
a secret is easy and silent.

* docs(secrets): state that every secret must be decryptable by the desktops

The desktops hold the trusted admin keys and are where secrets are edited and
rekeyed from; narrowing a set to the servers alone breaks agenix -r and drops
the offline recovery key.

* docs(secrets): desktops and recovery_key have identical access

recovery_key is a member of desktop_keys, so every group written as
desktop_keys ++ ... covers both.

* feat(monitoring): scrape the restic rest-server

Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.

* docs(backup): record what the metrics actually expose

rest-server only ships the go/promhttp defaults, so repository growth is
charted from node-exporter's filesystem metrics instead.
2026-09-21 21:01:18 +08:00
Ryan Yin 37f29154d1 docs(backup): record what the metrics actually expose
rest-server only ships the go/promhttp defaults, so repository growth is
charted from node-exporter's filesystem metrics instead.
2026-09-21 20:57:41 +08:00
Ryan Yin 842cd26f64 feat(monitoring): scrape the restic rest-server
Expose /metrics on the loopback-bound server, scrape it with
victoriametrics, and block /metrics on the public vhost.
2026-09-21 20:56:01 +08:00
Ryan Yin 1cbb52add1 docs(secrets): desktops and recovery_key have identical access
recovery_key is a member of desktop_keys, so every group written as
desktop_keys ++ ... covers both.
2026-09-21 20:51:13 +08:00
Ryan Yin 89e57b5b39 docs(secrets): state that every secret must be decryptable by the desktops
The desktops hold the trusted admin keys and are where secrets are edited and
rekeyed from; narrowing a set to the servers alone breaks agenix -r and drops
the offline recovery key.
2026-09-21 20:50:11 +08:00
Ryan Yin a43aa02a69 docs(secrets): state the recipient invariant
A recipient set must answer two questions: who may read it, and how it is
recovered once those hosts are gone. Dropping recovery_key while narrowing
a secret is easy and silent.
2026-09-21 20:45:47 +08:00
Ryan Yin 8a44c23d74 docs(backup): say 'cloud copy' rather than 'offsite'
Clearer for a reader who does not live in backup jargon; the requirement
(not in the homelab, immutable) is spelled out in the text.
2026-09-21 20:44:56 +08:00
Ryan Yin 096607a5c5 feat(backup): notify on failure, and 404 unconfigured hostnames
A failed backup was silent. Post to alertmanager (which routes above-info
to telegram), and have caddy answer 404 instead of its empty 200 for
*.writefor.fun names that are not services.
2026-09-21 20:42:39 +08:00
Ryan Yin c3f0dbf8bd revert(backup): drop append-only for the desktop endpoint
A desktop holds both the plaintext and the repository password, so blocking
deletion on the server protects little while preventing the automatic
retention the client needs. Immutability belongs in the offsite copy.
2026-09-21 20:38:08 +08:00
Ryan Yin 5a017666f6 refactor(backup): name the REST endpoint restic.writefor.fun
'backup' did not say what the service is.
2026-09-21 20:38:08 +08:00
Ryan Yin d322f4d2b6 docs: add BACKUP.md, and stagger the restic schedule per host
The design (layers, password groups, retention, restore steps) was only in
code comments. Also make the timer configurable: a whole hour of random
delay defeated any staggering, and the desktop's push, youko's own run and
btrbk all otherwise contend for the same disks.
2026-09-21 20:38:08 +08:00
Ryan Yin 2ff93c16ce feat(restic): move idols-ai's off-host backup to the REST server
Split the repository password into desktop and homelab groups, add the
environmentFile/pruneOpts module options, and point the desktop at youko's
REST endpoint.
2026-09-21 20:38:08 +08:00
Ryan Yin 1e2cef2550 feat(youko): serve restic backups over an append-only REST server
Run restic-rest-server behind caddy (append-only, private repos, htpasswd
from agenix) and register youko's host key centrally.
2026-09-21 20:38:08 +08:00
Ryan Yin 12ac5f45aa feat(idols-ai): restic backup to a local repository
Allowlist of the irreplaceable directories (Documents, work, Pictures,
nix-config, .local/state); project checkouts under ~/codes live in git and
are not backed up. Kept the excludes generic, since this repo is public.

Also move the restic-password secret to a desktop.enable || server.storage
gate so desktops get it without enabling a server group.
2026-09-21 19:04:58 +08:00
ryan4yin | 二花 e37042f61c feat(youko): restic backup of the service state to the HDD (part 1) (#299)
* feat(hosts): per-host restic backups to the SMB remote

Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:

- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
  (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
  excludes for the regenerable/bulky paths (podman overlay layers, microvm
  and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
  and dumps all postgres databases first (a consistent SQL artifact beside
  the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token

Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.

* refactor(hosts): only restic does off-host; btrbk stays local

btrbk works at the subvolume level, so a  would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.

So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.

Update the btrbk module's comment to say so.

* fix(restic): never back up keys; drop the implicit /etc/agenix

restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).

* refactor(restic): make the backup a proper module

Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.

* refactor(restic): exclude the whole podman storage, trim retention

Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.

Retention: daily 3, weekly 2, monthly 2 - no yearly.

* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump

A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.

* feat(secrets): restic-password for modules.restic-backup

With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.

* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix

* fix(restic): take the password path from agenix, not a hardcoded one

config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
2026-09-21 19:00:35 +08:00
ryan4yin | 二花 bb33cca83a refactor(youko): let preservation own the service state dirs (#298)
* refactor(youko): let preservation own the service state dirs

The services' data now sits on the SSD, so the hand-written dataDir +
systemd.tmpfiles rules are unnecessary: /var/lib is already preserved by
preservation, and the upstream modules create and own their default
state dirs (StateDirectory). Use the defaults:

- gitea /var/lib/gitea, grafana, sftpgo, caddy, victoriametrics,
  postgresql (/var/lib/postgresql/<version>), uptime-kuma, homepage
- drop the per-service tmpfiles rules and victoriametrics' self-bind

Data was moved in place with mv (same filesystem): /persistent/apps/<svc>
(== /var/lib/<svc> via the preservation bind) so nothing was copied.

* refactor(youko): drop caddy's redundant tmpfiles rules

The activation script already creates the fileserver tree with mkdir -p and
the module creates caddy's dataDir, so the rules - and the unused vms
directory they created - are redundant.

* refactor(youko): use a podman named volume for uptime-kuma

podman creates the volume and takes the ownership from the image, so the
kuma user, the tmpfiles rule and the host directory are no longer needed;
the volume lives under /var/lib/containers, which preservation keeps.
2026-09-21 18:06:40 +08:00
ryan4yin | 二花 9ef7ba7112 Merge pull request #297 from ryan4yin/feat/app-data-to-ssd
feat(youko): move the service data from the HDD to the SSD
2026-09-21 18:00:10 +08:00
Ryan Yin a755b6fa11 feat(youko): move the service data from the HDD to the SSD
The services now run directly on the host, so their state does not belong
on the HDD behind the flaky USB-SATA bridge: /data/apps was small-file
heavy and latency/consistency sensitive. Point every dataDir at
/persistent/apps (the LUKS nvme) instead.

The HDD keeps the bulk data that suits it: /data/fileshare (173G) and
/data/backups (24G). The HDD's old @apps subvolume and its data are left
in place for now.
2026-09-21 17:56:35 +08:00
ryan4yin | 二花 12966d326c Merge pull request #296 from ryan4yin/fix/usb-hdd-mitigations
fix(youko): mitigate the flaky USB-SATA HDD bridge
2026-09-21 17:56:08 +08:00