feat(youko): restic backup of the service state to the HDD (part 1) (#299)

* feat(hosts): per-host restic backups to the SMB remote

Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:

- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
  (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
  excludes for the regenerable/bulky paths (podman overlay layers, microvm
  and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
  and dumps all postgres databases first (a consistent SQL artifact beside
  the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token

Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.

* refactor(hosts): only restic does off-host; btrbk stays local

btrbk works at the subvolume level, so a  would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.

So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.

Update the btrbk module's comment to say so.

* fix(restic): never back up keys; drop the implicit /etc/agenix

restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).

* refactor(restic): make the backup a proper module

Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.

* refactor(restic): exclude the whole podman storage, trim retention

Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.

Retention: daily 3, weekly 2, monthly 2 - no yearly.

* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump

A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.

* feat(secrets): restic-password for modules.restic-backup

With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.

* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix

* fix(restic): take the password path from agenix, not a hardcoded one

config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.

flake.lock

@@ -503,10 +503,10 @@
     "mysecrets": {
       "flake": false,
       "locked": {
-        "lastModified": 1789975872,
-        "narHash": "sha256-DQrX1au7BWEzXJqR2FBiSYdF0DWd5NMh12y6ovtn41o=",
+        "lastModified": 1789987762,
+        "narHash": "sha256-gfrl1P3WdGBMdOZ9gaGg1UQwjGGwPzcLQUjUEhmBJOE=",
         "ref": "refs/heads/main",
-        "rev": "4def15deec5c0538df16ad2d49028298ee81d7ab",
+        "rev": "00f4f7fdf580eb1807e742977fd581dde137e575",
         "shallow": true,
         "type": "git",
         "url": "ssh://git@github.com/ryan4yin/nix-secrets.git"

hosts/12kingdoms-youko/default.nix

@@ -28,6 +28,16 @@ in
     coreModule
   ];
 
+  # Back up the preserved tree through a short-lived read-only snapshot, so the
+  # backup is a consistent point-in-time; the module's excludes drop the
+  # regenerable bulk and all keys/credentials.
+  modules.restic-backup = {
+    enable = true;
+    repository = "/data/backups/restic/youko";
+    snapshotSource = "/btr_pool/@persistent";
+    requiresMountsFor = "/data/backups";
+  };
+
   modules.btrbk.enable = true;
 
   boot.kernelParams = [

hosts/12kingdoms-youko/homelab-services/restic.nix

@@ -1,87 +0,0 @@
-{ pkgs, ... }:
-let
-  passwordFile = "/etc/agenix/restic-password";
-  sshKeyPath = "/etc/agenix/ssh-key-for-restic-backup";
-  rcloneConfigFile = "/etc/agenix/rclone-conf-for-restic-backup";
-in
-{
-  # DISABLED: this backup has never worked. It needs three agenix secrets that
-  # do not exist in nix-secrets yet (`restic-password`, `ssh-key-for-restic-backup`,
-  # and an rclone config with the `smb-downloads` remote; the existing
-  # `rclone.conf` secret is only defined for the desktop secret set). Re-enable
-  # once the secrets are provisioned.
-  #
-  # https://github.com/NixOS/nixpkgs/blob/nixos-26.05/nixos/modules/services/backup/restic.nix
-  /*
-    services.restic.backups = {
-      homelab-backup = {
-        inherit passwordFile;
-        initialize = true; # Initialize the repository if it doesn't exist.
-        repository = "rclone:smb-downloads:/Downloads/kubevirt-backup/"; # backup to a rclone remote
-
-        # rclone related
-        # rcloneOptions = {
-        #   bwlimit = "100M";  # Limit the bandwidth used by rclone.
-        # };
-        inherit rcloneConfigFile;
-
-        # Which local paths to backup, in addition to ones specified via `dynamicFilesFrom`.
-        paths = [
-          "/tmp/restic-backup-temp"
-        ];
-        #
-        # A script that produces a list of files to back up.  The
-        # results of this command are given to the '--files-from'
-        # option. The result is merged with paths specified via `paths`.
-        # dynamicFilesFrom = "find /home/matt/git -type d -name .git";
-        #
-        # Patterns to exclude when backing up. See
-        #   https://restic.readthedocs.io/en/latest/040_backup.html#excluding-files
-        # for details on syntax.
-        exclude = [ ];
-
-        # A script that must run before starting the backup process.
-        backupPrepareCommand = ''
-          ${pkgs.nushell}/bin/nu -c '
-            let kubevirt_nodes = [
-              "shoryu"
-              "shushou"
-              "youko"
-            ]
-
-            kubevirt_nodes | each {|it|
-              ^${pkgs.rsync}/bin/rsync -avz -e $"ssh -i ${sshKeyPath}" $"($it):/persistent/" $"/tmp/restic-backup-temp/($it)"
-            }
-          '
-        '';
-        # A script that must run after finishing the backup process.
-        backupCleanupCommand = "rm -rf /tmp/restic-backup-temp";
-
-        # Extra extended options to be passed to the restic --option flag.
-        # extraOptions = [];
-
-        # Extra arguments passed to restic backup.
-        # extraBackupArgs = [
-        #   "--exclude-file=/etc/restic/excludes-list"
-        # ];
-
-        # repository = "/mnt/backup-hdd"; # backup to a local directory
-        # When to run the backup. See {manpage}`systemd.timer(5)` for details.
-        timerConfig = {
-          OnCalendar = "01:30";
-          RandomizedDelaySec = "1h";
-        };
-        # A list of options (--keep-* et al.) for 'restic forget --prune',
-        # to automatically prune old snapshots.
-        # The 'forget' command is run *after* the 'backup' command, so
-        # keep that in mind when constructing the --keep-* options.
-        pruneOpts = [
-          "--keep-daily 3"
-          "--keep-weekly 3"
-          "--keep-monthly 3"
-          "--keep-yearly 3"
-        ];
-      };
-    };
-  */
-}

modules/nixos/base/btrbk.nix

@@ -19,8 +19,12 @@ in
   #   /btr_pool/@snapshots/@persistent.<timestamp>
   #
   # These are same-filesystem snapshots: they protect against accidental
-  # deletion and bad edits, NOT against disk loss. Set `target` (plus
-  # `services.btrbk.sshAccess` on the receiving host) for off-host backups.
+  # deletion and bad edits, NOT against disk loss. Off-host copies are handled
+  # by restic (lib/genResticBackup.nix), which can exclude the regenerable bulk
+  # at the file level; btrbk works at the subvolume level, so a `target` would
+  # also ship podman's overlay storage and the VM disk images. Set `target`
+  # (plus `services.btrbk.sshAccess` on the receiving host) only if you want
+  # that.
   #
   # The host MUST mount the btrfs top-level subvolume (subvolid=5) at `volume`;
   # this is enforced by an assertion so a missing mount fails evaluation instead

modules/nixos/base/restic-backup.nix

@@ -0,0 +1,134 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+let
+  cfg = config.modules.restic-backup;
+
+  # A short-lived, read-only btrfs snapshot taken by the backup itself, so the
+  # backup is a consistent point-in-time (a live copy of, say, a running
+  # postgres data directory is only crash-consistent). It is deleted right
+  # after, so it is not a second backup - just a consistency step.
+  snapshotPath = "${builtins.dirOf cfg.snapshotSource}/@restic-snapshot";
+in
+{
+  # ==================================================================
+  #
+  # A restic backup to a LOCAL repository (part 1 of the backup plan).
+  #
+  # Part 2 (copying the critical snapshots to a cloud object store with
+  # `restic copy`) is a later step. btrbk keeps making the cheap local btrfs
+  # snapshots for rollbacks; off-host copies are restic's job (restic can
+  # exclude the regenerable bulk at the file level).
+  #
+  # Keys and credentials are NEVER included: restic's own password lives in
+  # /etc/agenix, so backing that up would store the repository's password
+  # inside the repository.
+  #
+  # ==================================================================
+  options.modules.restic-backup = {
+    enable = lib.mkEnableOption "a restic backup (local repository)";
+
+    repository = lib.mkOption {
+      type = lib.types.str;
+      example = "/data/backups/restic/youko";
+      description = "Restic repository, e.g. a path under a mounted backup disk.";
+    };
+
+    snapshotSource = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "/btr_pool/@persistent";
+      description = ''
+        btrfs subvolume to back up through a short-lived read-only snapshot.
+        When null, `paths` is backed up directly.
+      '';
+    };
+
+    paths = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      description = "Paths to back up (used when {option}`snapshotSource` is null).";
+    };
+
+    exclude = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      description = "Extra exclude patterns, on top of the built-in ones.";
+    };
+
+    requiresMountsFor = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      description = ''
+        Mount point the repository lives on, when it is a `nofail` mount: the
+        backup is then ordered after that mount instead of running against a
+        not-yet-mounted directory.
+      '';
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    services.restic.backups.homelab = {
+      inherit (cfg) repository;
+      initialize = true;
+      # take the path from agenix itself instead of hardcoding its secretsDir
+      passwordFile = config.age.secrets."restic-password".path;
+
+      paths = if cfg.snapshotSource != null then [ snapshotPath ] else cfg.paths;
+
+      # Relative to the backed-up tree, so they work for both the snapshot and
+      # a direct path.
+      exclude = [
+        # regenerable, huge, or unsuitable for file-level backup
+        #
+        # the whole podman storage tree: the image layers are re-pullable, and
+        # its storage DB is not consistent when copied from a running podman.
+        # (uptime-kuma uses a named volume under here; its data is not wanted.)
+        "var/lib/containers"
+        "var/lib/microvms"
+        "var/lib/libvirt"
+        "nfs"
+        "var/cache"
+        "var/tmp"
+        "var/log"
+        "*.qcow2"
+        # keys and credentials are never backed up
+        "etc/agenix"
+        "etc/ssh/ssh_host_*"
+        "**/.ssh"
+        "**/.gnupg"
+        "**/.aws"
+        "**/.config/gcloud"
+      ]
+      ++ cfg.exclude;
+
+      timerConfig = {
+        OnCalendar = "01:30";
+        RandomizedDelaySec = "1h";
+      };
+
+      pruneOpts = [
+        "--keep-daily 3"
+        "--keep-weekly 2"
+        "--keep-monthly 2"
+      ];
+    }
+    // lib.optionalAttrs (cfg.snapshotSource != null) {
+      backupPrepareCommand = ''
+        ${pkgs.btrfs-progs}/bin/btrfs subvolume snapshot -r ${cfg.snapshotSource} ${snapshotPath}
+      '';
+      backupCleanupCommand = ''
+        ${pkgs.btrfs-progs}/bin/btrfs subvolume delete ${snapshotPath}
+      '';
+    };
+
+    systemd.services.restic-backups-homelab.unitConfig =
+      lib.optionalAttrs (cfg.requiresMountsFor != null)
+        {
+          RequiresMountsFor = cfg.requiresMountsFor;
+        };
+  };
+}

secrets/nixos.nix

@@ -220,6 +220,12 @@ in
           mode = "0400";
           owner = "root";
         };
+        # restic repository password (used by modules.restic-backup)
+        "restic-password" = {
+          file = "${mysecrets}/restic-password.age";
+          mode = "0400";
+          owner = "root";
+        };
       };
 
       # place secrets in /etc/