fix(monitoring): stop recurring false alerts (#395)

The Telegram channel was dominated by alerts that can never be actionable:

- DriveTemperatureHigh/Critical did not filter temperature_type, so
  max(smartctl_device_temperature) picked the drive's rated
  op_limit_max/critical_limit_max instead of the actual reading. ai reported
  95 C and youko 130 C while the drives actually ran at 40-48 C.
- HostPhysicalComponentTooHot excluded label!="tctl", but the AMD k10temp
  hwmon sensor labels itself Tctl, so the CPU control temperature (up to
  ~100 C under load) tripped the alert on every AMD host.
- MicroVM guests share the host's read-only /nix/store, so their
  nix-gc.service always failed and tripped HostSystemdServiceCrashed on all
  six k3s guests. Disable automatic GC in the guest; the host's nix.gc
  reclaims the shared store.

The remaining k3s alert noise comes from workloads running on the masters
that belong on the workers; that is fixed in k8s-gitops, not by resizing
the control-plane VMs.

hosts/12kingdoms-youko/homelab-services/monitoring/alert_rules/node-exporter.yml

@@ -307,8 +307,8 @@ groups:
       - alert: HostPhysicalComponentTooHot
         expr:
           '((node_hwmon_temp_celsius * ignoring(label) group_left(instance, job, node, sensor)
-          node_hwmon_sensor_label{label!="tctl"} > 75)) * on(instance) group_left (nodename)
-          node_uname_info{nodename=~".+"}'
+          node_hwmon_sensor_label{label!~"(?i)^(tctl|tccd|tdie)$"} > 75)) * on(instance) group_left
+          (nodename) node_uname_info{nodename=~".+"}'
         for: 5m
         labels:
           severity: warning

hosts/12kingdoms-youko/homelab-services/monitoring/alert_rules/storage.yml

@@ -70,8 +70,11 @@ groups:
           description: "{{ $value }} unrecovered data-integrity errors recorded."
 
       # Drive temperature (SSD/HDD). Consumer M.2 NVMe throttles around 70-80 C.
+      # The metric carries the drive's rated `op_limit_max`/`critical_limit_max`
+      # alongside `current`; without `temperature_type="current"` the max() alerts on
+      # those constant limits (e.g. youko's 120/130 C) instead of the real reading.
       - alert: DriveTemperatureHigh
-        expr: max by (host, device) (smartctl_device_temperature) > 70
+        expr: max by (host, device) (smartctl_device_temperature{temperature_type="current"}) > 70
         for: 15m
         labels:
           severity: warning
@@ -80,7 +83,7 @@ groups:
           description: "Drive temperature above 70 C; NVMe throttles around 70-80 C."
 
       - alert: DriveTemperatureCritical
-        expr: max by (host, device) (smartctl_device_temperature) > 80
+        expr: max by (host, device) (smartctl_device_temperature{temperature_type="current"}) > 80
         for: 5m
         labels:
           severity: critical

lib/genMicrovmGuestModule.nix

@@ -127,6 +127,12 @@ in
 
   system.stateVersion = "26.05";
 
+  # The guest shares the host's read-only /nix/store (virtiofs) and has no store
+  # of its own, so its nix-gc.service can never succeed; it just failed on every
+  # k3s guest and tripped HostSystemdServiceCrashed. The host's own nix.gc
+  # reclaims the shared store, so disable automatic GC inside the guest.
+  nix.gc.automatic = false;
+
   # MicroVM guests share the host's store and have no physical hardware to
   # inspect, nothing to trace, no persisted /var/cache for a `locate` index,
   # and nothing to advertise over mDNS.