feat(backup): hourly read-only snapshots and per-backup integrity checks (#402)

* feat(restic): verify the repository after each backup

* feat(youko): snapshot the restic store hourly

* docs(backup): document the snapshot and check layers

BACKUP.md

@@ -4,12 +4,12 @@ How this fleet is backed up, where the data lives, and how to restore it.
 
 ## What protects against what
 
-| Threat                        | Defence                                                            |
-| ----------------------------- | ------------------------------------------------------------------ |
-| accidental deletion, bad edit | btrbk: local btrfs snapshots                                       |
-| disk or host loss             | restic: encrypted copies on youko                                  |
-| a compromised backup server   | desktop repositories use a password youko never holds              |
-| a compromised host            | not defended yet: the immutable cloud copy (planned) is that layer |
+| Threat                        | Defence                                                                                                                                                     |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| accidental deletion, bad edit | btrbk: local btrfs snapshots                                                                                                                                |
+| disk or host loss             | restic: encrypted copies on youko                                                                                                                           |
+| a compromised backup server   | desktop repositories use a password youko never holds                                                                                                       |
+| a compromised host            | youko takes hourly read-only btrfs snapshots of the REST store, which a desktop cannot reach; the immutable cloud copy (planned) is still the offsite layer |
 
 ## Layers
 
@@ -22,7 +22,10 @@ How this fleet is backed up, where the data lives, and how to restore it.
    with `--private-repos`, authenticated by htpasswd, published behind caddy as
    `restic.writefor.fun`. Deliberately not append-only: a desktop holds both the plaintext and the
    repository password, so blocking deletion there protects little while denying the client the
-   automatic retention it needs.
+   automatic retention it needs. A compromised desktop can still `forget --prune` its own repository
+   through the REST API, so youko also snapshots `/data/backups` hourly into a read-only btrfs
+   subvolume the desktop cannot reach (see
+   `hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix`).
 3. **cloud copy** (planned). youko will copy the homelab repositories with `restic copy` (it holds
    that password). Desktops copy their own: youko does not hold the desktop password. This layer is
    what protects history against a compromised host, so the target should be immutable or versioned
@@ -61,6 +64,7 @@ collisions.
 | ---------------- | ----------------------------------- |
 | 01:30 + 15m      | desktops push to youko              |
 | 02:30 + 15m      | youko backs itself up               |
+| hourly           | youko snapshots the REST store      |
 | 03:45:20 Tue/Sat | btrbk local snapshots               |
 | 06:00 (planned)  | youko copies homelab repos to cloud |
 
@@ -123,7 +127,8 @@ Restoring a btrbk snapshot (offline; stop writers first):
 ## Verifying
 
 - Units: `systemctl status restic-backups-homelab`, `systemctl list-timers 'restic-backups*'`.
-- Repository integrity (where the password is): `restic check`.
+- Repository integrity (where the password is): `restic check`, which runs after every backup via
+  `modules.restic-backup.checkOpts` (a 5% data read, `1/20`, by default).
 - Server side, ciphertext only: repository directories under `/data/backups/rest-server/<user>/`,
   with no leftover files in `locks/` and no stale `.tmp` files.
 - `restic-rest-server`'s log must not print `Invalid htpasswd entry`: that means the client's REST
@@ -144,7 +149,6 @@ Restoring a btrbk snapshot (offline; stop writers first):
 - **Cloud copy**: youko copies the homelab repositories with `restic copy`; desktops copy their own.
   The target must not live in the homelab, and should be immutable or versioned (object lock) — that
   is the layer which survives a compromised host.
-- **`restic check` timer**: periodic integrity verification, on the host that holds the password.
 
 ## Components
 
@@ -153,6 +157,7 @@ Restoring a btrbk snapshot (offline; stop writers first):
 | [`modules/nixos/base/restic-backup.nix`](./modules/nixos/base/restic-backup.nix) | client module: repository, paths, excludes, schedule |
 | [`modules/nixos/base/btrbk.nix`](./modules/nixos/base/btrbk.nix)                 | local btrfs snapshots                                |
 | `hosts/12kingdoms-youko/default.nix`                                             | the REST server and youko's own backup               |
+| `hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix`            | hourly read-only snapshots of the REST store         |
 | `hosts/12kingdoms-youko/homelab-services/caddy.nix`                              | the `restic.writefor.fun` vhost                      |
 | `secrets/nixos.nix`                                                              | which secret is defined on which host                |
 | `hosts/idols-ai/restic.nix`, `hosts/12kingdoms-youko/default.nix`                | per-host backup configuration                        |

SECURITY.md

@@ -44,7 +44,7 @@ small, reversible controls and narrow exceptions; benchmark before adopting expe
 | Desktop apps    | Selected apps use nixpak/bubblewrap; Podman is rootless                                                                                                 | Rules and shared directories vary per app; user namespaces are needed for these sandboxes                                                                   |
 | Secrets         | agenix declarations here, encrypted data in the private nix-secrets repository                                                                          | Recipients and plaintext file permissions must be reviewed; root on a recipient can read its decrypted secrets                                              |
 | Supply chain    | Locked flake inputs; explicit binary-cache trust keys; normal user is not a Nix trusted-user                                                            | A trusted cache or imported Nix module can supply code executed as root; a lock file is not a security audit                                                |
-| Recovery        | NixOS generations, btrbk/restic and documented backup procedures                                                                                        | A rollback does not undo compromised data or secrets; backup configuration alone does not prove restore works                                               |
+| Recovery        | NixOS generations, btrbk/restic, hourly read-only snapshots of the REST store, and documented backup procedures                                         | A rollback does not undo compromised data or secrets; backup configuration alone does not prove restore works; the offsite immutable copy is still planned  |
 
 Sources: [host layout](hosts/README.md), [k3s deployment](hosts/k8s/README.md),
 [firewall](modules/nixos/base/networking/firewall.nix), [SSH](modules/nixos/base/ssh.nix),

hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix

@@ -0,0 +1,53 @@
+{
+  pkgs,
+  ...
+}:
+let
+  # The restic REST server store, and a directory on the sibling @snapshots
+  # subvolume (same btrfs pool), so the snapshots are real subvolumes the
+  # desktop has no path to.
+  source = "/data/backups";
+  snapshotRoot = "/data/apps-snapshots/restic";
+  keepDays = 14;
+in
+{
+  # The REST server is deliberately not append-only: the desktop holds both the
+  # data and its repository password and still needs to run retention. That
+  # leaves the repository deletable by anyone holding the desktop's REST
+  # credentials. A read-only snapshot the desktop cannot reach makes a
+  # `forget --prune` recoverable without giving up retention.
+  systemd.tmpfiles.rules = [
+    "d ${snapshotRoot} 0700 root root -"
+  ];
+
+  systemd.services.restic-server-snapshot = {
+    description = "Read-only btrfs snapshot of the restic server store";
+    serviceConfig = {
+      Type = "oneshot";
+      User = "root";
+    };
+    path = [
+      pkgs.btrfs-progs
+      pkgs.coreutils
+      pkgs.findutils
+    ];
+    script = ''
+      set -euo pipefail
+      stamp=$(date +%Y%m%d%H%M%S)
+      btrfs subvolume snapshot -r ${source} ${snapshotRoot}/backups.$stamp
+      # Drop snapshots older than ${toString keepDays} days. The desktop cannot
+      # reach this tree, so it cannot shorten the window.
+      find ${snapshotRoot} -mindepth 1 -maxdepth 1 -type d -mtime +${toString keepDays} -print0 |
+        xargs -0 -r -n1 btrfs subvolume delete
+    '';
+    unitConfig.RequiresMountsFor = source;
+  };
+
+  systemd.timers.restic-server-snapshot = {
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      OnCalendar = "hourly";
+      Persistent = true;
+    };
+  };
+}

modules/nixos/base/restic-backup.nix

@@ -133,6 +133,17 @@ in
         after uploading).
       '';
     };
+
+    checkOpts = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ "--read-data-subset=1/20" ];
+      description = ''
+        Arguments for `restic check`, run after each backup. Non-empty enables
+        the check. The default reads 5% of the pack data as a fraction rather
+        than `5%`, so systemd does not treat the percent as a specifier; set
+        to `[ ]` to disable, or `--read-data` for a full read.
+      '';
+    };
   };
 
   config = lib.mkIf cfg.enable {
@@ -168,6 +179,7 @@ in
       };
 
       inherit (cfg) pruneOpts;
+      inherit (cfg) checkOpts;
     }
     // lib.optionalAttrs (cfg.excludeLargerThan != null) {
       extraBackupArgs = [