* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
- Add an offlineHosts list to VictoriaMetrics scrape generation: SBCs
and the whole k3s-prod-1 cluster are powered off, shoukei's exporter
is disabled on the machine; also comment out the dnsmasq-exporter
job (runs on suzi).
- Disable node-exporter on shoukei: a laptop on untrusted networks
should not expose :9100 (the firewall is off repo-wide).
- Route severity none|info alerts to the null receiver; meta alerts
and info noise should never page.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
node-exporter always reports device_error=1 for containerd sandbox shm
bind mounts (tmpfs), producing ~50 bogus critical alerts on the
kubevirt hosts. Verified against live metrics that no non-shm device
errors exist, so nothing real is filtered out.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
The NixOS alertmanager module pipes the generated config through
envsubst to inject secrets, which silently replaced the template's
$i/$a variables with empty strings, so every telegram notification
failed with a template parse error since Friday's deploy.
Rewrite the template without variables: range + dot, define/template
for the per-alert block, and an if/else split to cap a group at 5
alerts (slice errors when fewer than 5). Verified with amtool template
render against post-envsubst content for both branches.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
- Show group-level labels (alertgroup/cluster/env/namespace) once in
the header instead of repeating them per alert.
- Use each rule's summary annotation as the per-alert headline (it
already carries the distinguishing resource name), falling back to
the instance label; show nodename on its own line when present.
- Drop the noisy per-alert label dump and the always-N/A value line.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
- Trim the telegram template: drop the full label dump, render at
most 5 alerts per group, and note how many were omitted, so
messages stay within Telegram's 4096-char limit.
- Route severity=none meta alerts (Watchdog, InfoInhibitor) to a new
null receiver so they no longer notify.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
ryan4yin | 二花