- Add an offlineHosts list to VictoriaMetrics scrape generation: SBCs
and the whole k3s-prod-1 cluster are powered off, shoukei's exporter
is disabled on the machine; also comment out the dnsmasq-exporter
job (runs on suzi).
- Disable node-exporter on shoukei: a laptop on untrusted networks
should not expose :9100 (the firewall is off repo-wide).
- Route severity none|info alerts to the null receiver; meta alerts
and info noise should never page.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
node-exporter always reports device_error=1 for containerd sandbox shm
bind mounts (tmpfs), producing ~50 bogus critical alerts on the
kubevirt hosts. Verified against live metrics that no non-shm device
errors exist, so nothing real is filtered out.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
The NixOS alertmanager module pipes the generated config through
envsubst to inject secrets, which silently replaced the template's
$i/$a variables with empty strings, so every telegram notification
failed with a template parse error since Friday's deploy.
Rewrite the template without variables: range + dot, define/template
for the per-alert block, and an if/else split to cap a group at 5
alerts (slice errors when fewer than 5). Verified with amtool template
render against post-envsubst content for both branches.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
- Show group-level labels (alertgroup/cluster/env/namespace) once in
the header instead of repeating them per alert.
- Use each rule's summary annotation as the per-alert headline (it
already carries the distinguishing resource name), falling back to
the instance label; show nodename on its own line when present.
- Drop the noisy per-alert label dump and the always-N/A value line.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
- Trim the telegram template: drop the full label dump, render at
most 5 alerts per group, and note how many were omitted, so
messages stay within Telegram's 4096-char limit.
- Route severity=none meta alerts (Watchdog, InfoInhibitor) to a new
null receiver so they no longer notify.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
Ryan Yin