diff --git a/BACKUP.md b/BACKUP.md index d89a45de..9f2a5de3 100644 --- a/BACKUP.md +++ b/BACKUP.md @@ -4,12 +4,12 @@ How this fleet is backed up, where the data lives, and how to restore it. ## What protects against what -| Threat | Defence | -| ----------------------------- | ------------------------------------------------------------------ | -| accidental deletion, bad edit | btrbk: local btrfs snapshots | -| disk or host loss | restic: encrypted copies on youko | -| a compromised backup server | desktop repositories use a password youko never holds | -| a compromised host | not defended yet: the immutable cloud copy (planned) is that layer | +| Threat | Defence | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| accidental deletion, bad edit | btrbk: local btrfs snapshots | +| disk or host loss | restic: encrypted copies on youko | +| a compromised backup server | desktop repositories use a password youko never holds | +| a compromised host | youko takes hourly read-only btrfs snapshots of the REST store, which a desktop cannot reach; the immutable cloud copy (planned) is still the offsite layer | ## Layers @@ -22,7 +22,10 @@ How this fleet is backed up, where the data lives, and how to restore it. with `--private-repos`, authenticated by htpasswd, published behind caddy as `restic.writefor.fun`. Deliberately not append-only: a desktop holds both the plaintext and the repository password, so blocking deletion there protects little while denying the client the - automatic retention it needs. + automatic retention it needs. A compromised desktop can still `forget --prune` its own repository + through the REST API, so youko also snapshots `/data/backups` hourly into a read-only btrfs + subvolume the desktop cannot reach (see + `hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix`). 3. **cloud copy** (planned). youko will copy the homelab repositories with `restic copy` (it holds that password). Desktops copy their own: youko does not hold the desktop password. This layer is what protects history against a compromised host, so the target should be immutable or versioned @@ -61,6 +64,7 @@ collisions. | ---------------- | ----------------------------------- | | 01:30 + 15m | desktops push to youko | | 02:30 + 15m | youko backs itself up | +| hourly | youko snapshots the REST store | | 03:45:20 Tue/Sat | btrbk local snapshots | | 06:00 (planned) | youko copies homelab repos to cloud | @@ -123,7 +127,8 @@ Restoring a btrbk snapshot (offline; stop writers first): ## Verifying - Units: `systemctl status restic-backups-homelab`, `systemctl list-timers 'restic-backups*'`. -- Repository integrity (where the password is): `restic check`. +- Repository integrity (where the password is): `restic check`, which runs after every backup via + `modules.restic-backup.checkOpts` (a 5% data read, `1/20`, by default). - Server side, ciphertext only: repository directories under `/data/backups/rest-server//`, with no leftover files in `locks/` and no stale `.tmp` files. - `restic-rest-server`'s log must not print `Invalid htpasswd entry`: that means the client's REST @@ -144,7 +149,6 @@ Restoring a btrbk snapshot (offline; stop writers first): - **Cloud copy**: youko copies the homelab repositories with `restic copy`; desktops copy their own. The target must not live in the homelab, and should be immutable or versioned (object lock) — that is the layer which survives a compromised host. -- **`restic check` timer**: periodic integrity verification, on the host that holds the password. ## Components @@ -153,6 +157,7 @@ Restoring a btrbk snapshot (offline; stop writers first): | [`modules/nixos/base/restic-backup.nix`](./modules/nixos/base/restic-backup.nix) | client module: repository, paths, excludes, schedule | | [`modules/nixos/base/btrbk.nix`](./modules/nixos/base/btrbk.nix) | local btrfs snapshots | | `hosts/12kingdoms-youko/default.nix` | the REST server and youko's own backup | +| `hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix` | hourly read-only snapshots of the REST store | | `hosts/12kingdoms-youko/homelab-services/caddy.nix` | the `restic.writefor.fun` vhost | | `secrets/nixos.nix` | which secret is defined on which host | | `hosts/idols-ai/restic.nix`, `hosts/12kingdoms-youko/default.nix` | per-host backup configuration | diff --git a/SECURITY.md b/SECURITY.md index 61c201b0..ffb7786c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -44,7 +44,7 @@ small, reversible controls and narrow exceptions; benchmark before adopting expe | Desktop apps | Selected apps use nixpak/bubblewrap; Podman is rootless | Rules and shared directories vary per app; user namespaces are needed for these sandboxes | | Secrets | agenix declarations here, encrypted data in the private nix-secrets repository | Recipients and plaintext file permissions must be reviewed; root on a recipient can read its decrypted secrets | | Supply chain | Locked flake inputs; explicit binary-cache trust keys; normal user is not a Nix trusted-user | A trusted cache or imported Nix module can supply code executed as root; a lock file is not a security audit | -| Recovery | NixOS generations, btrbk/restic and documented backup procedures | A rollback does not undo compromised data or secrets; backup configuration alone does not prove restore works | +| Recovery | NixOS generations, btrbk/restic, hourly read-only snapshots of the REST store, and documented backup procedures | A rollback does not undo compromised data or secrets; backup configuration alone does not prove restore works; the offsite immutable copy is still planned | Sources: [host layout](hosts/README.md), [k3s deployment](hosts/k8s/README.md), [firewall](modules/nixos/base/networking/firewall.nix), [SSH](modules/nixos/base/ssh.nix), diff --git a/hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix b/hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix new file mode 100644 index 00000000..bc69bd5f --- /dev/null +++ b/hosts/12kingdoms-youko/homelab-services/restic-server-snapshots.nix @@ -0,0 +1,53 @@ +{ + pkgs, + ... +}: +let + # The restic REST server store, and a directory on the sibling @snapshots + # subvolume (same btrfs pool), so the snapshots are real subvolumes the + # desktop has no path to. + source = "/data/backups"; + snapshotRoot = "/data/apps-snapshots/restic"; + keepDays = 14; +in +{ + # The REST server is deliberately not append-only: the desktop holds both the + # data and its repository password and still needs to run retention. That + # leaves the repository deletable by anyone holding the desktop's REST + # credentials. A read-only snapshot the desktop cannot reach makes a + # `forget --prune` recoverable without giving up retention. + systemd.tmpfiles.rules = [ + "d ${snapshotRoot} 0700 root root -" + ]; + + systemd.services.restic-server-snapshot = { + description = "Read-only btrfs snapshot of the restic server store"; + serviceConfig = { + Type = "oneshot"; + User = "root"; + }; + path = [ + pkgs.btrfs-progs + pkgs.coreutils + pkgs.findutils + ]; + script = '' + set -euo pipefail + stamp=$(date +%Y%m%d%H%M%S) + btrfs subvolume snapshot -r ${source} ${snapshotRoot}/backups.$stamp + # Drop snapshots older than ${toString keepDays} days. The desktop cannot + # reach this tree, so it cannot shorten the window. + find ${snapshotRoot} -mindepth 1 -maxdepth 1 -type d -mtime +${toString keepDays} -print0 | + xargs -0 -r -n1 btrfs subvolume delete + ''; + unitConfig.RequiresMountsFor = source; + }; + + systemd.timers.restic-server-snapshot = { + wantedBy = [ "timers.target" ]; + timerConfig = { + OnCalendar = "hourly"; + Persistent = true; + }; + }; +} diff --git a/modules/nixos/base/restic-backup.nix b/modules/nixos/base/restic-backup.nix index f68bbe12..6732cb55 100644 --- a/modules/nixos/base/restic-backup.nix +++ b/modules/nixos/base/restic-backup.nix @@ -133,6 +133,17 @@ in after uploading). ''; }; + + checkOpts = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ "--read-data-subset=1/20" ]; + description = '' + Arguments for `restic check`, run after each backup. Non-empty enables + the check. The default reads 5% of the pack data as a fraction rather + than `5%`, so systemd does not treat the percent as a specifier; set + to `[ ]` to disable, or `--read-data` for a full read. + ''; + }; }; config = lib.mkIf cfg.enable { @@ -168,6 +179,7 @@ in }; inherit (cfg) pruneOpts; + inherit (cfg) checkOpts; } // lib.optionalAttrs (cfg.excludeLargerThan != null) { extraBackupArgs = [