mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-19 06:08:17 +01:00
feat(backup): alert when the last successful backup goes stale
The backup unit publishes `restic_backup_last_success_timestamp_seconds` for the node_exporter textfile collector, and a rejected alert now fails the notify unit instead of exiting 0. Assisted-by: DeepSeek Harness 0.2.1-rc.2 (deepseek-flash)
This commit is contained in:
1 parent
d2b11840d0
commit
9b68cfdb8a
4 files changed
+78
-6
No files matched your search
@@ -7,6 +7,12 @@
|
||||
- [victoria-metrics-k8s-stack/files/rules](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack/files/rules/generated)
|
||||
- Alert Rules & Recording Rules used by kube-prometheus-stack.
|
||||
|
||||
## Backup Freshness
|
||||
|
||||
`modules/nixos/base/restic-backup.nix` writes `restic_backup_last_success_timestamp_seconds` to
|
||||
node_exporter's textfile collector after a successful run. `alert_rules/restic-backup.yml` alerts
|
||||
when it is older than two days (the schedule is daily) or absent entirely.
|
||||
|
||||
## TODO
|
||||
|
||||
- **Dead-man switch**: `Watchdog` in `alert_rules/general.yml` always fires (`vector(1)`) but is
|
||||
@@ -14,8 +20,3 @@
|
||||
healthy one. Route it to an external heartbeat service (healthchecks.io, Uptime Kuma, PagerDuty
|
||||
DeadMansSnitch) that notifies when the heartbeat stops, or run an out-of-band probe against
|
||||
Alertmanager.
|
||||
- **Backup freshness**: nothing alerts on "no recent successful backup". The systemd `failed` state
|
||||
only catches crashes, not a job that never ran or a backup that is silently stale. Export the last
|
||||
successful backup timestamp (the restic rest-server already exposes Go/promhttp metrics; btrfs
|
||||
snapshot time via the node-exporter textfile collector) and alert when it exceeds the SLO.
|
||||
Off-host copies are already restic's job (see `BACKUP.md`), so no btrbk `target` is needed.
|
||||
@@ -0,0 +1,31 @@
|
||||
groups:
|
||||
- name: restic-backup.rules
|
||||
rules:
|
||||
# restic-backup.nix writes this after every fully successful run (backup,
|
||||
# prune and check). The schedule is daily (02:30 on youko, 01:30 on the
|
||||
# desktops, each with up to 15m of random delay), so 48h means two runs in
|
||||
# a row were missed. Only hosts whose node_exporter is up report this.
|
||||
- alert: ResticBackupStale
|
||||
expr: time() - restic_backup_last_success_timestamp_seconds > 2 * 24 * 3600
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: No successful restic backup on {{ $labels.host }} in 2 days
|
||||
description:
|
||||
"The last successful restic backup on {{ $labels.host }} was {{ $value }} seconds ago.
|
||||
Check `systemctl status restic-backups-homelab.service` and `journalctl -u
|
||||
restic-backups-homelab` there."
|
||||
|
||||
# Staleness needs a sample to compare against; this covers the case where
|
||||
# no host has ever published (or every textfile was wiped).
|
||||
- alert: ResticBackupMetricMissing
|
||||
expr: absent(restic_backup_last_success_timestamp_seconds)
|
||||
for: 26h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: No restic backup reports a successful run
|
||||
description:
|
||||
"restic_backup_last_success_timestamp_seconds is absent on every host, so backup
|
||||
freshness cannot be evaluated."
|
||||
Reference in new issue
Block a user