Files
nix-config/secrets
ryan4yin | 二花 e37042f61c feat(youko): restic backup of the service state to the HDD (part 1) (#299)
* feat(hosts): per-host restic backups to the SMB remote

Replace the disabled aggregate job (which rsynced every host's /persistent
into /tmp - a tmpfs - and silently produced incomplete backups) with one
restic repo per host, backing up its own data directly:

- shared module lib/genResticBackup.nix: secrets, rclone/SMB target
  (rclone:smb-downloads:/Downloads/homelab-backup/<host>), retention, and
  excludes for the regenerable/bulky paths (podman overlay layers, microvm
  and libvirt disk images, nfs golden images, logs, caches)
- youko additionally backs up the service state under /persistent/var/lib
  and dumps all postgres databases first (a consistent SQL artifact beside
  the crash-consistent data dir)
- shoryu/shushou back up /etc/agenix, /etc/ssh and the k3s token

Needs three agenix secrets before it can run: restic-password,
ssh-key-for-restic-backup and rclone-conf-for-restic-backup.

* refactor(hosts): only restic does off-host; btrbk stays local

btrbk works at the subvolume level, so a  would ship the whole
@persistent - including podman's overlay storage, the microVM/libvirt
images and the nfs golden images - which is exactly the regenerable bulk
we want out of the backups.

So: btrbk keeps making the cheap local CoW snapshots (quick rollbacks),
and restic owns the off-host copies with file-level excludes. Back up the
whole /persistent minus those excludes, so nothing is missed.

Update the btrbk module's comment to say so.

* fix(restic): never back up keys; drop the implicit /etc/agenix

restic's own password lives in /etc/agenix, so backing that up would store
the repository's password inside the repository. Exclude agenix, the ssh
host keys and the usual credential dirs (.ssh, .gnupg, .aws, gcloud).

* refactor(restic): make the backup a proper module

Replace lib/genResticBackup.nix with modules/nixos/base/restic-backup.nix,
which declares modules.restic-backup options (enable, repository, paths,
exclude, requiresMountsFor, postgresDump) like the btrbk module does.
youko just sets the options now.

* refactor(restic): exclude the whole podman storage, trim retention

Exclude /persistent/var/lib/containers wholesale: the image layers are
re-pullable and podman's storage DB is not consistent when copied from a
running podman. uptime-kuma's named volume lives there, but its data is
not wanted anyway.

Retention: daily 3, weekly 2, monthly 2 - no yearly.

* refactor(restic): back up a short-lived btrfs snapshot, drop pg_dump

A live copy of a running postgres data dir is only crash-consistent. Instead
of a daily pg_dumpall (which duplicates the data in the repository and gets
slower as the DB grows), take a short-lived read-only btrfs snapshot in
backupPrepareCommand, back that up, and delete it right after - a
consistency step, not a second backup. The excludes are relative now so they
work for both the snapshot and a direct path.

* feat(secrets): restic-password for modules.restic-backup

With the rekeyed secret in nix-secrets (storage group, alongside
hdd-luks-crypt-key), the module's passwordFile resolves.

* fix(restic): agenix's secret is under /run/agenix, not /etc/agenix

* fix(restic): take the password path from agenix, not a hardcoded one

config.age.secrets."restic-password".path follows agenix's secretsDir, so it
stays correct if that changes.
2026-09-21 19:00:35 +08:00
..

Secrets Management

For Website/App's passwords, see /home/base/tui/password-store for more details.

All my secrets are safely encrypted via agenix, and stored in a separate private GitHub repository and referenced as a flake input in this flake.

The encryption is done using the public keys of all my hosts (/etc/ssh/ssh_host_ed25519_key), so that they can only be decrypted on any of my configured hosts. The host keys are generated locally on each host by OpenSSH without a passphrase and are only readable by root. The host keys will never leave the host.

In this way, all secrets are still encrypted when transmitted over the network and written to /nix/store. They are decrypted only when they are finally used.

In addition, we further improve the security of secret files by storing them in a separate private repository.

This directory contains this README.md, and a nixos.nix/darwin.nix file that is used to decrypt all my secrets via agenix. Then, I can use them in this flake.

Adding or Updating Secrets

All the operations in this section should be performed in my private repository: nix-secrets.

This task is accomplished using the agenix CLI tool with the ./secrets.nix file, so you need to have it installed first:

To use agenix temporarily, run:

nix shell github:ryantm/agenix#agenix

or agenix provided by ragenix, run:

nix shell github:ryan4yin/ragenix#ragenix

Suppose you want to add a new secret file xxx.age. Follow these steps:

  1. Navigate to your private nix-secrets repository.
  2. Edit secrets.nix and add a new entry for xxx.age, defining the encryption keys and the secret file path, for example:
# This file is not imported into your NixOS configuration. It is only used for the agenix CLI.
# agenix use the public keys defined in this file to encrypt the secrets.
# and users can decrypt the secrets by any of the corresponding private keys.

let
  # Get system's ssh public key by command:
  #    cat /etc/ssh/ssh_host_ed25519_key.pub
  # If you do not have this file, you can generate all the host keys by command:
  #    sudo ssh-keygen -A
  idol_ai = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINHZtzeaQyXwuRMLzoOAuTu8P9bu5yc5MBwo5LI3iWBV root@ai";

  # A key for recovery purpose, generated by `ssh-keygen -t ed25519 -a 256 -C "ryan@agenix-recovery"` with a strong passphrase
  # and kept it offline in a safe place.
  recovery_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHnIGH+653Oe+GQaA8zjjj7HWMWp7bWXed4q5KqY4nqG ryan@agenix-recovery";
  systems = [
    idol_ai
    fern

    recovery_key
  ];
{
  "./xxx.age".publicKeys = users ++ systems;
}
  1. Create and edit the secret file xxx.age interactively using the following command:
sudo agenix -e ./xxx.age -i /etc/ssh/ssh_host_ed25519_key

Alternatively, you can encrypt an existing file to xxx.age using the following command:

cat xxx | sudo agenix  -e ./xxx.age -i /etc/ssh/ssh_host_ed25519_key

agenix will encrypt the file with all the public keys we defined in secrets.nix, so all the users and systems defined in secrets.nix can decrypt it with their private keys.

Deploying Secrets

All the operations in this section should be performed in this repository.

First, add your own private nix-secrets repository and agenix as a flake input, and pass them to sub modules via specialArgs:

{
  inputs = {
    # ......

    # secrets management, lock with git commit at 2023/5/15
    agenix.url = "github:ryantm/agenix/db5637d10f797bb251b94ef9040b237f4702cde3";

    # my private secrets, it's a private repository, you need to replace it with your own.
    mysecrets = { url = "github:ryan4yin/nix-secrets"; flake = false; };
  };

  outputs = inputs@{ self, nixpkgs, ... }: {
    nixosConfigurations = {
      nixos-test = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";

        # Set all input parameters as specialArgs of all sub-modules
        # so that we can use the `agenix` & `mysecrets` in sub-modules
        specialArgs = inputs;
        modules = [
          # ......

          # import & decrypt secrets in `mysecrets` in this module
          ./secrets/default.nix
        ];
      };
    };
  };
}

Then, create ./secrets/default.nix with the following content:

# import & decrypt secrets in `mysecrets` in this module
{ config, pkgs, agenix, mysecrets, ... }:

{
  imports = [
     agenix.nixosModules.default
  ];

  # if you changed this key, you need to regenerate all encrypt files from the decrypt contents!
  age.identityPaths = [
    # using the host key for decryption
    # the host key is generated on every host locally by openssh, and will never leave the host.
    "/etc/ssh/ssh_host_ed25519_key"
  ];

  age.secrets."xxx" = {
    # whether secrets are symlinked to age.secrets.<name>.path
    symlink = true;
    # target path for decrypted file
    path = "/etc/xxx/";
    # encrypted file path
    file =  "${mysecrets}/xxx.age";  # refer to ./xxx.age located in `mysecrets` repo
    mode = "0400";
    owner = "root";
    group = "root";
  };
}

From now on, every time you run nixos-rebuild switch, it will decrypt the secrets using the private keys defined in age.identityPaths. It will then symlink the secrets to the path defined by the age.secrets.<name>.path argument, which defaults to /etc/secrets.

Adding a new host

  1. cat the system-level public key(/etc/ssh/ssh_host_ed25519_key.pub) of the new host, and send it to an old host which has already been configured.
  2. On the old host:
    1. Add the public key to secrets.nix, and rekey all the secrets via sudo agenix -r -i /etc/ssh/ssh_host_ed25519_key.
    2. Commit and push the changes to nix-secrets.
  3. On the new host:
    1. Clone this repo and run nixos-rebuild switch to deploy it, all the secrets will be decrypted automatically via the host private key.

Troubleshooting

1. Nix-Darwin Module

Check logs:

tail -n 100 /Library/Logs/org.nixos.activate-agenix.stderr.log
tail -n 100 /Library/Logs/org.nixos.activate-agenix.stdout.log

2. NixOS Module

Check logs:

journalctl | grep -5 agenix

Other Replacements

  • ragenix: A Rust reimplementation of agenix.
    • agenix is mainly written in bash, and it's error message is quite obscure, a little typo may cause some errors no one can understand.
    • with a type-safe language like Rust, we can get a better error message and less bugs.