Files
nix-config/docs/BACKUP.md
T
2026-11-17 21:08:02 +08:00

11 KiB

Backups

How this fleet is backed up, where the data lives, and how to restore it.

What protects against what

Threat Defence
accidental deletion, bad edit btrbk: local btrfs snapshots
disk or host loss restic: encrypted copies on youko
a compromised backup server desktop repositories use a password youko never holds
a compromised host youko takes hourly read-only btrfs snapshots of the REST store, which a desktop cannot reach; the immutable cloud copy (planned) is still the offsite layer

Layers

  1. btrbk — local snapshots (modules/nixos/base/btrbk.nix). Same-filesystem btrfs snapshots of @persistent, run on Tue/Sat 03:45:20, kept for 7 days (minimum 2). Cheap rollback for mistakes; it does not survive disk loss.
  2. restic — encrypted copies on youko (modules/nixos/base/restic-backup.nix). Encryption happens on the client, so the server only ever stores ciphertext. youko runs restic-rest-server with --private-repos, authenticated by htpasswd, published behind caddy as restic.writefor.fun. Deliberately not append-only: a desktop holds both the plaintext and the repository password, so blocking deletion there protects little while denying the client the automatic retention it needs. A compromised desktop can still forget --prune its own repository through the REST API, so youko also snapshots /data/backups hourly into a read-only btrfs subvolume the desktop cannot reach (see hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix). The REST store has one 512 GiB quota (--max-size) shared by all repositories, so a runaway client cannot fill the HDD pool it shares with /data/apps and /data/fileshare; at the quota, uploads fail with HTTP 507 and the client's backup-failure alert fires. youko's own repository (/data/backups/restic) is written directly, outside the quota.
  3. cloud copy (planned). youko will copy the homelab repositories with restic copy (it holds that password). Desktops copy their own: youko does not hold the desktop password. This layer is what protects history against a compromised host, so the target should be immutable or versioned (object lock or equivalent) rather than a plain bucket.

Password model

Two independent repository passwords, deliberately not shared:

Group Secret Recipients Decrypts
desktop restic-password-desktop.age desktops desktop repositories
homelab restic-password-homelab.age homelab hosts homelab repositories
REST access restic-rest-credentials.age desktops nothing (auth only)
REST access restic-rest-htpasswd.age the REST server (youko) nothing (auth only)

Consequences worth remembering:

  • youko stores desktop repositories but cannot read or verify them. Any content check (restic snapshots, restic check) for a desktop repository must run on that desktop.
  • Keep both repository passwords offline (password manager or paper). If a host dies, its password is the only way to restore its data.
  • restic's own password is never stored inside a backed-up tree: etc/agenix and all key material are excluded (see below).
  • Recipients follow the secrets repository's rule: every secret is decryptable by the desktops (desktop_keys, which also carries the offline recovery_key), so a desktop can edit or rekey anything and no secret becomes unrecoverable when a host is lost. The desktop repository password is the one secret the servers do not get.

Schedule

Slots are assigned per host; the random delay only smooths load and must not be relied on to prevent collisions.

Time Job
01:30 + 15m desktops push to youko
02:30 + 15m youko dumps PostgreSQL, then backs itself up
hourly youko snapshots the REST store
03:45:20 Tue/Sat btrbk local snapshots
04:00-05:00, 1st btrfs scrub on the physical servers
06:00 (planned) youko copies homelab repos to cloud

Set per host via modules.restic-backup.onCalendar and randomizedDelaySec. The scrub row is a start window: reading youko's HDDs (about 2.7 TiB used) takes hours, past the 06:00 slot.

What is never backed up

  • Keys and credentials, excluded by the module on every host: etc/agenix, etc/ssh/ssh_host_*, **/.ssh, **/.gnupg, **/.aws, **/.config/gcloud. Backing these up would put repository and host credentials inside the repository itself.
  • Regenerable bulk, excluded per host: podman's storage tree, microVM/libvirt images, NFS and cache/log directories on youko; language build artefacts, caches, model weights and anything larger than 500M on desktops.

Prefer a specific paths allowlist plus a size cap over a long exclude list.

Operations

# run a backup now, and read its log
systemctl start restic-backups-homelab
journalctl -u restic-backups-homelab -n 50 --no-pager

# timer state
systemctl list-timers 'restic-backups*'

Inspect or restore a repository where its password lives (a desktop repository on a desktop, a homelab repository on a homelab host):

restic -r <repository> --password-file /run/agenix/restic-password snapshots
restic -r <repository> --password-file /run/agenix/restic-password check
restic -r <repository> --password-file /run/agenix/restic-password \
  restore latest --target /tmp/restore --include /home/ryan/Documents

Remote (REST) repositories also need the service credentials:

set -a; . /run/agenix/restic-rest-credentials; set +a
restic -r rest:https://restic.writefor.fun/idols-ai/ \
  --password-file /run/agenix/restic-password snapshots

Restoring a btrbk snapshot (offline; stop writers first):

  1. btrfs subvolume delete /btr_pool/@persistent
  2. btrfs subvolume snapshot /btr_pool/@snapshots/@persistent.<timestamp> /btr_pool/@persistent
  3. reboot, or remount /persistent, to pick up the restored subvolume.

Retention

  • btrbk: automatic, 7 days with a 2 day minimum.
  • youko's own restic repository: automatic, --keep-daily 3 --keep-weekly 2 --keep-monthly 2.
  • Desktop restic repositories: automatic, the module defaults (--keep-daily 3 --keep-weekly 2 --keep-monthly 2). Immutability is the cloud copy's job (planned), not the local server's.

Verifying

  • Units: systemctl status restic-backups-homelab, systemctl list-timers 'restic-backups*'.
  • Repository integrity (where the password is): restic check, which runs after every backup via modules.restic-backup.checkOpts (a 5% data read, 1/20, by default).
  • PostgreSQL on youko: postgresqlBackup.service runs before every restic-backups-homelab run and writes pg_dumpall output to /var/lib/postgresql-backup/all.sql.zstd (the previous dump is kept as all.prev.sql.zstd). Restore into an empty cluster with zstd -dc all.sql.zstd | sudo -u postgres psql -X postgres. A failed dump does not block the backup; it shows up as a failed unit.
  • REST store quota: restic-rest-server logs Quota initialized, currently using N GiB at start.
  • Server side, ciphertext only: repository directories under /data/backups/rest-server/<user>/, with no leftover files in locks/ and no stale .tmp files.
  • restic-rest-server's log must not print Invalid htpasswd entry: that means the client's REST password and the server's htpasswd disagree, and the credentials pair needs rebuilding from one password.
  • Metrics: the rest-server exposes the Go/promhttp defaults on loopback (scraped as the restic-rest-server job; /metrics is blocked on the public vhost). Repository growth is charted from node-exporter's filesystem metrics for youko's /data mount instead — rest-server itself has no per-repository size metrics.
  • Failures are not silent: a failed restic-backups-homelab starts restic-backup-notify, which posts to alertmanager and, above info severity, on to telegram. alertmanager clears it after resolve_timeout.

Planned

  • Restore drill: actually restore a file from a restic snapshot and from a btrbk snapshot, on a schedule. An unverified restore is not a backup.
  • Cloud copy: youko copies the homelab repositories with restic copy; desktops copy their own. The target must not live in the homelab, and should be immutable or versioned (object lock) — that is the layer which survives a compromised host.

Components

Path Role
modules/nixos/base/restic-backup.nix client module: repository, paths, excludes, schedule
modules/nixos/base/btrbk.nix local btrfs snapshots
hosts/12kingdoms-youko/default.nix the REST server and youko's own backup
hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix hourly read-only snapshots of the REST store
hosts/12kingdoms-youko/homelab-services/postgresql.nix the PostgreSQL dump taken before youko's backup
hosts/12kingdoms-youko/homelab-services/caddy.nix the restic.writefor.fun vhost
secrets/nixos.nix which secret is defined on which host
hosts/idols-ai/restic.nix, hosts/12kingdoms-youko/default.nix per-host backup configuration