feat(backup): alert when the last successful backup goes stale

The backup unit publishes `restic_backup_last_success_timestamp_seconds` for the node_exporter textfile collector, and a rejected alert now fails the notify unit instead of exiting 0.

Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-flash)
This commit is contained in:
Ryan Yin authored and ryan4yin | 二花 committed 2026-10-22 00:54:27 +08:00
1 parent d2b11840d0
commit 9b68cfdb8a
4 files changed
+78 -6

No files matched your search

@@ -7,6 +7,12 @@
- [victoria-metrics-k8s-stack/files/rules](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack/files/rules/generated)
- Alert Rules & Recording Rules used by kube-prometheus-stack.
## Backup Freshness
`modules/nixos/base/restic-backup.nix` writes `restic_backup_last_success_timestamp_seconds` to
node_exporter's textfile collector after a successful run. `alert_rules/restic-backup.yml` alerts
when it is older than two days (the schedule is daily) or absent entirely.
## TODO
- **Dead-man switch**: `Watchdog` in `alert_rules/general.yml` always fires (`vector(1)`) but is
@@ -14,8 +20,3 @@
healthy one. Route it to an external heartbeat service (healthchecks.io, Uptime Kuma, PagerDuty
DeadMansSnitch) that notifies when the heartbeat stops, or run an out-of-band probe against
Alertmanager.
- **Backup freshness**: nothing alerts on "no recent successful backup". The systemd `failed` state
only catches crashes, not a job that never ran or a backup that is silently stale. Export the last
successful backup timestamp (the restic rest-server already exposes Go/promhttp metrics; btrfs
snapshot time via the node-exporter textfile collector) and alert when it exceeds the SLO.
Off-host copies are already restic's job (see `BACKUP.md`), so no btrbk `target` is needed.
@@ -0,0 +1,31 @@
groups:
- name: restic-backup.rules
rules:
# restic-backup.nix writes this after every fully successful run (backup,
# prune and check). The schedule is daily (02:30 on youko, 01:30 on the
# desktops, each with up to 15m of random delay), so 48h means two runs in
# a row were missed. Only hosts whose node_exporter is up report this.
- alert: ResticBackupStale
expr: time() - restic_backup_last_success_timestamp_seconds > 2 * 24 * 3600
for: 10m
labels:
severity: warning
annotations:
summary: No successful restic backup on {{ $labels.host }} in 2 days
description:
"The last successful restic backup on {{ $labels.host }} was {{ $value }} seconds ago.
Check `systemctl status restic-backups-homelab.service` and `journalctl -u
restic-backups-homelab` there."
# Staleness needs a sample to compare against; this covers the case where
# no host has ever published (or every textfile was wiped).
- alert: ResticBackupMetricMissing
expr: absent(restic_backup_last_success_timestamp_seconds)
for: 26h
labels:
severity: warning
annotations:
summary: No restic backup reports a successful run
description:
"restic_backup_last_success_timestamp_seconds is absent on every host, so backup
freshness cannot be evaluated."