* feat(hosts): per-host restic backups to the SMB remote Replace the disabled aggregate job (which rsynced every host's /persistent into /tmp - a tmpfs - and silently produced incomplete backups) with one restic repo per host, backing up its own data directly: - shared module lib/genResticBackup.nix: secrets, rclone/SMB target (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and excludes for the regenerable/bulky paths (podman overlay layers, microvm and libvirt disk images, nfs golden images, logs, caches) - youko additionally backs up the service state under /persistent/var/lib and dumps all postgres databases first (a consistent SQL artifact beside the crash-consistent data dir) - shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token Needs three agenix secrets before it can run: restic-password, ssh-key-for-restic-backup and rclone-conf-for-restic-backup. * refactor(hosts): only restic does off-host; btrbk stays local btrbk works at the subvolume level, so a would ship the whole @persistent - including podman's overlay storage, the microVM/libvirt images and the nfs golden images - which is exactly the regenerable bulk we want out of the backups. So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks), and restic owns the off-host copies with file-level excludes. Back up the whole /persistent minus those excludes, so nothing is missed. Update the btrbk module's comment to say so. * fix(restic): never back up keys; drop the implicit /etc/agenix restic's own password lives in /etc/agenix, so backing that up would store the repository's password inside the repository. Exclude agenix, the ssh host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud). * refactor(restic): make the backup a proper module Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix, which declares modules.restic-backup options (enable, repository, paths, exclude, requiresMountsFor, postgresDump) like the btrbk module does. youko just sets the options now. * refactor(restic): exclude the whole podman storage, trim retention Exclude /persistent/var/lib/containers wholesale: the image layers are re-pullable and podman's storage DB is not consistent when copied from a running podman. uptime-kuma's named volume lives there, but its data is not wanted anyway. Retention: daily 3, weekly 2, monthly 2 - no yearly. * refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump A live copy of a running postgres data dir is only crash-consistent. Instead of a daily pg_dumpall (which duplicates the data in the repository and gets slower as the DB grows), take a short-lived read-only btrfs snapshot in backupPrepareCommand, back that up, and delete it right after - a consistency step, not a second backup. The excludes are relative now so they work for both the snapshot and a direct path. * feat(secrets): restic-password for modules.restic-backup With the rekeyed secret in nix-secrets (storage group, alongside hdd-luks-crypt-key), the module's passwordFile resolves. * fix(restic): agenix's secret is under /run/agenix, not /etc/agenix * fix(restic): take the password path from agenix, not a hardcoded one config.age.secrets."restic-password".path follows agenix's secretsDir, so it stays correct if that changes.
Secrets Management
For Website/App's passwords, see /home/base/tui/password-store for more details.
All my secrets are safely encrypted via agenix, and stored in a separate private GitHub repository and referenced as a flake input in this flake.
The encryption is done using the public keys of all my hosts (/etc/ssh/ssh_host_ed25519_key), so
that they can only be decrypted on any of my configured hosts. The host keys are generated locally
on each host by OpenSSH without a passphrase and are only readable by root. The host keys will
never leave the host.
In this way, all secrets are still encrypted when transmitted over the network and written to
/nix/store. They are decrypted only when they are finally used.
In addition, we further improve the security of secret files by storing them in a separate private repository.
This directory contains this README.md, and a nixos.nix/darwin.nix file that is used to
decrypt all my secrets via agenix. Then, I can use them in this flake.
Adding or Updating Secrets
All the operations in this section should be performed in my private repository:
nix-secrets.
This task is accomplished using the agenix CLI tool with the
./secrets.nix file, so you need to have it installed first:
To use agenix temporarily, run:
nix shell github:ryantm/agenix#agenix
or agenix provided by ragenix, run:
nix shell github:ryan4yin/ragenix#ragenix
Suppose you want to add a new secret file xxx.age. Follow these steps:
- Navigate to your private
nix-secretsrepository. - Edit
secrets.nixand add a new entry forxxx.age, defining the encryption keys and the secret file path, for example:
# This file is not imported into your NixOS configuration. It is only used for the agenix CLI.
# agenix use the public keys defined in this file to encrypt the secrets.
# and users can decrypt the secrets by any of the corresponding private keys.
let
# Get system's ssh public key by command:
# cat /etc/ssh/ssh_host_ed25519_key.pub
# If you do not have this file, you can generate all the host keys by command:
# sudo ssh-keygen -A
idol_ai = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINHZtzeaQyXwuRMLzoOAuTu8P9bu5yc5MBwo5LI3iWBV root@ai";
# A key for recovery purpose, generated by `ssh-keygen -t ed25519 -a 256 -C "ryan@agenix-recovery"` with a strong passphrase
# and kept it offline in a safe place.
recovery_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHnIGH+653Oe+GQaA8zjjj7HWMWp7bWXed4q5KqY4nqG ryan@agenix-recovery";
systems = [
idol_ai
fern
recovery_key
];
{
"./xxx.age".publicKeys = users ++ systems;
}
- Create and edit the secret file
xxx.ageinteractively using the following command:
sudo agenix -e ./xxx.age -i /etc/ssh/ssh_host_ed25519_key
Alternatively, you can encrypt an existing file to xxx.age using the following command:
cat xxx | sudo agenix -e ./xxx.age -i /etc/ssh/ssh_host_ed25519_key
agenix will encrypt the file with all the public keys we defined in secrets.nix, so all the
users and systems defined in secrets.nix can decrypt it with their private keys.
Deploying Secrets
All the operations in this section should be performed in this repository.
First, add your own private nix-secrets repository and agenix as a flake input, and pass them to
sub modules via specialArgs:
{
inputs = {
# ......
# secrets management, lock with git commit at 2023/5/15
agenix.url = "github:ryantm/agenix/db5637d10f797bb251b94ef9040b237f4702cde3";
# my private secrets, it's a private repository, you need to replace it with your own.
mysecrets = { url = "github:ryan4yin/nix-secrets"; flake = false; };
};
outputs = inputs@{ self, nixpkgs, ... }: {
nixosConfigurations = {
nixos-test = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
# Set all input parameters as specialArgs of all sub-modules
# so that we can use the `agenix` & `mysecrets` in sub-modules
specialArgs = inputs;
modules = [
# ......
# import & decrypt secrets in `mysecrets` in this module
./secrets/default.nix
];
};
};
};
}
Then, create ./secrets/default.nix with the following content:
# import & decrypt secrets in `mysecrets` in this module
{ config, pkgs, agenix, mysecrets, ... }:
{
imports = [
agenix.nixosModules.default
];
# if you changed this key, you need to regenerate all encrypt files from the decrypt contents!
age.identityPaths = [
# using the host key for decryption
# the host key is generated on every host locally by openssh, and will never leave the host.
"/etc/ssh/ssh_host_ed25519_key"
];
age.secrets."xxx" = {
# whether secrets are symlinked to age.secrets.<name>.path
symlink = true;
# target path for decrypted file
path = "/etc/xxx/";
# encrypted file path
file = "${mysecrets}/xxx.age"; # refer to ./xxx.age located in `mysecrets` repo
mode = "0400";
owner = "root";
group = "root";
};
}
From now on, every time you run nixos-rebuild switch, it will decrypt the secrets using the
private keys defined in age.identityPaths. It will then symlink the secrets to the path defined by
the age.secrets.<name>.path argument, which defaults to /etc/secrets.
Adding a new host
catthe system-level public key(/etc/ssh/ssh_host_ed25519_key.pub) of the new host, and send it to an old host which has already been configured.- On the old host:
- Add the public key to
secrets.nix, and rekey all the secrets viasudo agenix -r -i /etc/ssh/ssh_host_ed25519_key. - Commit and push the changes to
nix-secrets.
- Add the public key to
- On the new host:
- Clone this repo and run
nixos-rebuild switchto deploy it, all the secrets will be decrypted automatically via the host private key.
- Clone this repo and run
Troubleshooting
1. Nix-Darwin Module
Check logs:
tail -n 100 /Library/Logs/org.nixos.activate-agenix.stderr.log
tail -n 100 /Library/Logs/org.nixos.activate-agenix.stdout.log
2. NixOS Module
Check logs:
journalctl | grep -5 agenix
Other Replacements
- ragenix: A Rust reimplementation of agenix.
- agenix is mainly written in bash, and it's error message is quite obscure, a little typo may cause some errors no one can understand.
- with a type-safe language like Rust, we can get a better error message and less bugs.