fix(monitoring): stop recurring false alerts (#395)

The Telegram channel was dominated by alerts that can never be actionable:

- DriveTemperatureHigh/Critical did not filter temperature_type, so
  max(smartctl_device_temperature) picked the drive's rated
  op_limit_max/critical_limit_max instead of the actual reading. ai reported
  95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
  hwmon sensor labels itself Tctl, so the CPU control temperature (up to
  ~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
  nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
  six k3s guests. Disable automatic GC in the guest; the host's nix.gc
  reclaims the shared store.

The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.
This commit is contained in:
ryan4yin | 二花 authored and GitHub committed 2026-10-07 12:51:30 +08:00
1 parent a7cc474a50
commit 90bbe8bc4f
3 files changed
+13 -4

No files matched your search

+6
View File
@@ -127,6 +127,12 @@ in
system.stateVersion = "26.05";
# The guest shares the host's read-only /nix/store (virtiofs) and has no store
# of its own, so its nix-gc.service can never succeed; it just failed on every
# k3s guest and tripped HostSystemdServiceCrashed. The host's own nix.gc
# reclaims the shared store, so disable automatic GC inside the guest.
nix.gc.automatic = false;
# MicroVM guests share the host's store and have no physical hardware to
# inspect, nothing to trace, no persisted /var/cache for a `locate` index,
# and nothing to advertise over mDNS.