mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-22 15:24:03 +01:00
A desktop holds both the plaintext and the repository password, so blocking deletion on the server protects little while preventing the automatic retention the client needs. Immutability belongs in the offsite copy.
140 lines
7.3 KiB
Markdown
140 lines
7.3 KiB
Markdown
# Backups
|
|
|
|
How this fleet is backed up, where the data lives, and how to restore it.
|
|
|
|
## What protects against what
|
|
|
|
| Threat | Defence |
|
|
| ----------------------------- | -------------------------------------------------------------------- |
|
|
| accidental deletion, bad edit | btrbk: local btrfs snapshots |
|
|
| disk or host loss | restic: encrypted copies on youko |
|
|
| a compromised backup server | desktop repositories use a password youko never holds |
|
|
| a compromised host | not defended yet: the immutable offsite copy (planned) is that layer |
|
|
|
|
## Layers
|
|
|
|
1. **btrbk — local snapshots** ([`modules/nixos/base/btrbk.nix`](./modules/nixos/base/btrbk.nix)).
|
|
Same-filesystem btrfs snapshots of `@persistent`, run on Tue/Sat 03:45:20, kept for 7 days
|
|
(minimum 2). Cheap rollback for mistakes; it does **not** survive disk loss.
|
|
2. **restic — encrypted copies on youko**
|
|
([`modules/nixos/base/restic-backup.nix`](./modules/nixos/base/restic-backup.nix)). Encryption
|
|
happens on the client, so the server only ever stores ciphertext. youko runs `restic-rest-server`
|
|
with `--private-repos`, authenticated by htpasswd, published behind caddy as
|
|
`restic.writefor.fun`. Deliberately not append-only: a desktop holds both the plaintext and the
|
|
repository password, so blocking deletion there protects little while denying the client the
|
|
automatic retention it needs.
|
|
3. **cloud copy** (planned). youko will copy the homelab repositories with `restic copy` (it holds
|
|
that password). Desktops copy their own: youko does not hold the desktop password. This layer is
|
|
what protects history against a compromised host, so the target should be immutable or versioned
|
|
(object lock or equivalent) rather than a plain bucket.
|
|
|
|
## Password model
|
|
|
|
Two independent repository passwords, deliberately not shared:
|
|
|
|
| Group | Secret | Recipients | Decrypts |
|
|
| ----------- | ----------------------------- | ------------- | -------------------- |
|
|
| desktop | `restic-password-desktop.age` | desktops | desktop repositories |
|
|
| homelab | `restic-password-homelab.age` | homelab hosts | homelab repositories |
|
|
| REST access | `restic-rest-credentials.age` | desktops | nothing (auth only) |
|
|
| REST access | `restic-rest-htpasswd.age` | all hosts | nothing (auth only) |
|
|
|
|
Consequences worth remembering:
|
|
|
|
- youko stores desktop repositories but cannot read or verify them. Any content check
|
|
(`restic snapshots`, `restic check`) for a desktop repository must run on that desktop.
|
|
- Keep both repository passwords offline (password manager or paper). If a host dies, its password
|
|
is the only way to restore its data.
|
|
- restic's own password is never stored inside a backed-up tree: `etc/agenix` and all key material
|
|
are excluded (see below).
|
|
|
|
## Schedule
|
|
|
|
Slots are assigned per host; the random delay only smooths load and must not be relied on to prevent
|
|
collisions.
|
|
|
|
| Time | Job |
|
|
| ---------------- | ----------------------------------- |
|
|
| 01:30 + 15m | desktops push to youko |
|
|
| 02:30 + 15m | youko backs itself up |
|
|
| 03:45:20 Tue/Sat | btrbk local snapshots |
|
|
| other slots | other homelab hosts, before 03:30 |
|
|
| 06:00 (planned) | youko copies homelab repos to cloud |
|
|
|
|
Set per host via `modules.restic-backup.onCalendar` and `randomizedDelaySec`.
|
|
|
|
## What is never backed up
|
|
|
|
- **Keys and credentials**, excluded by the module on every host: `etc/agenix`,
|
|
`etc/ssh/ssh_host_*`, `**/.ssh`, `**/.gnupg`, `**/.aws`, `**/.config/gcloud`. Backing these up
|
|
would put repository and host credentials inside the repository itself.
|
|
- **Regenerable bulk**, excluded per host: podman's storage tree, microVM/libvirt images, NFS and
|
|
cache/log directories on youko; language build artefacts, caches, model weights and anything
|
|
larger than 500M on desktops.
|
|
|
|
Prefer a specific `paths` allowlist plus a size cap over a long `exclude` list.
|
|
|
|
## Operations
|
|
|
|
```bash
|
|
# run a backup now, and read its log
|
|
systemctl start restic-backups-homelab
|
|
journalctl -u restic-backups-homelab -n 50 --no-pager
|
|
|
|
# timer state
|
|
systemctl list-timers 'restic-backups*'
|
|
```
|
|
|
|
Inspect or restore a repository where its password lives (a desktop repository on a desktop, a
|
|
homelab repository on a homelab host):
|
|
|
|
```bash
|
|
restic -r <repository> --password-file /run/agenix/restic-password snapshots
|
|
restic -r <repository> --password-file /run/agenix/restic-password check
|
|
restic -r <repository> --password-file /run/agenix/restic-password \
|
|
restore latest --target /tmp/restore --include /home/ryan/Documents
|
|
```
|
|
|
|
Remote (REST) repositories also need the service credentials:
|
|
|
|
```bash
|
|
set -a; . /run/agenix/restic-rest-credentials; set +a
|
|
restic -r rest:https://restic.writefor.fun/idols-ai/ \
|
|
--password-file /run/agenix/restic-password snapshots
|
|
```
|
|
|
|
Restoring a btrbk snapshot (offline; stop writers first):
|
|
|
|
1. `btrfs subvolume delete /btr_pool/@persistent`
|
|
2. `btrfs subvolume snapshot /btr_pool/@snapshots/@persistent.<timestamp> /btr_pool/@persistent`
|
|
3. reboot, or remount `/persistent`, to pick up the restored subvolume.
|
|
|
|
## Retention
|
|
|
|
- btrbk: automatic, 7 days with a 2 day minimum.
|
|
- youko's own restic repository: automatic, `--keep-daily 3 --keep-weekly 2 --keep-monthly 2`.
|
|
- Desktop restic repositories: automatic, the module defaults
|
|
(`--keep-daily 3 --keep-weekly 2 --keep-monthly 2`). Immutability is the offsite copy's job
|
|
(planned), not the local server's.
|
|
|
|
## Verifying
|
|
|
|
- Units: `systemctl status restic-backups-homelab`, `systemctl list-timers 'restic-backups*'`.
|
|
- Repository integrity (where the password is): `restic check`.
|
|
- Server side, ciphertext only: repository directories under `/data/backups/rest-server/<user>/`,
|
|
with no leftover files in `locks/` and no stale `.tmp` files.
|
|
- `restic-rest-server`'s log must not print `Invalid htpasswd entry`: that means the client's REST
|
|
password and the server's htpasswd disagree, and the credentials pair needs rebuilding from one
|
|
password.
|
|
|
|
## Components
|
|
|
|
| Path | Role |
|
|
| -------------------------------------------------------------------------------- | ---------------------------------------------------- |
|
|
| [`modules/nixos/base/restic-backup.nix`](./modules/nixos/base/restic-backup.nix) | client module: repository, paths, excludes, schedule |
|
|
| [`modules/nixos/base/btrbk.nix`](./modules/nixos/base/btrbk.nix) | local btrfs snapshots |
|
|
| `hosts/12kingdoms-youko/default.nix` | the REST server and youko's own backup |
|
|
| `hosts/12kingdoms-youko/homelab-services/caddy.nix` | the `restic.writefor.fun` vhost |
|
|
| `secrets/nixos.nix` | which secret is defined on which host |
|
|
| `hosts/<host>/restic.nix` | per-host backup configuration |
|