Files
nix-config/hosts/idols-ai/hardware-configuration.nix
T
ryan4yin | 二花 dc3ec8c159 feat(storage): relax ai NVMe workarounds and add drive SMART monitoring (#359)
* fix(ai): relax NVMe power-management workarounds and enable smartd

Logs show no NVMe errors, so allow APST again (capped at 100 ms), restore
the default io_timeout, and drop pcie_aspm=off. Add smartd for drive
health visibility.

* feat(storage): add drive/SSD SMART monitoring on physical hosts

node_exporter's nvme collector only exposes node_nvme_info and namespace
capacity, not SMART health, so SSD wear/lifespan and drive temperature were
not observable anywhere.

- enable services.prometheus.exporters.smartctl on physical hosts, gated on
  modules.hardwareTools (off on QEMU/MicroVM guests), bound to the host's LAN
  address on port 9633
- firewall: drop 9633 from anything but the monitoring host
- youko: scrape smartctl-<host> for the physical hosts
- alert rules (storage.yml): SMART health, NVMe wear, available spare, media
  errors, critical warning, and drive temperature (warn 70 / crit 80 C)
- Grafana dashboard "Drive SMART (smartctl)"
2026-10-03 11:13:46 +08:00

93 lines
3.6 KiB
Nix
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Do not modify this file! It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{
config,
lib,
pkgs,
modulesPath,
...
}:
{
imports = [
(modulesPath + "/installer/scan/not-detected.nix")
];
boot.kernelParams = [
# === NVMe power management (relaxed 2026-10) ===
#
# These began as a hard workaround for NVMe freezes on the KINGBANK KP260
# (a DRAM-less QLC drive, prone to slow APST wakeups): APST fully disabled,
# the command timeout pushed to the maximum, and PCIe ASPM turned off.
# Kernel logs from 2026-09-19..10-03 (8 boots) show no NVMe
# timeout/reset/AER, so it is relaxed to reclaim idle power and let the
# kernel recover a stuck command instead of hanging on it:
#
# - APST allowed again, capped at a wide 100 ms latency (was: disabled).
# - io_timeout back to the kernel default 30 s (was: ~49 days, which let
# a stuck command wait forever instead of resetting the controller).
# - pcie_aspm removed: PCIe link power saving is unrelated to the drive's
# own APST.
#
# If freezes return under heavy read/write, restore the old workaround:
# nvme_core.default_ps_max_latency_us=0 nvme_core.io_timeout=4294967295 pcie_aspm=off
# See https://community.frame.work/t/nvme-timeout-woes/54999
"nvme_core.default_ps_max_latency_us=100000"
];
# Use the EFI boot loader.
boot.loader.efi.canTouchEfiVariables = true;
# depending on how you configured your disk mounts, change this to /boot or /boot/efi.
boot.loader.efi.efiSysMountPoint = "/boot";
boot.loader.systemd-boot.enable = true;
# https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/top-level/linux-kernels.nix
boot.kernelPackages = pkgs.linuxPackages_latest;
boot.initrd.availableKernelModules = [
"xhci_pci"
"ahci"
"nvme"
"usbhid"
"usb_storage"
"sd_mod"
];
# clear /tmp on boot to get a stateless /tmp directory.
boot.tmp.cleanOnBoot = true;
# Enable binfmt emulation of aarch64-linux, this is required for cross compilation.
boot.binfmt.emulatedSystems = [
"aarch64-linux"
"riscv64-linux"
];
# This enables the kernel to preload the emulator binaries when the binfmt registrations are added,
# obviating the need to make the emulator binaries available inside chroots and chroot-like sandboxes.
boot.binfmt.preferStaticEmulators = true; # required to work with podman
# supported file systems, so we can mount any removable disks with these filesystems
boot.supportedFilesystems = [
"ext4"
"btrfs"
"xfs"
"ntfs"
"fat"
"vfat"
"exfat"
];
# LUKS initrd, all fileSystems (/, /boot, /btr_pool, /nix, /persistent, /snapshots, /tmp, /swap)
# and swap (including /swap/swapfile bind and swapDevices) are managed by disko (disko-fs.nix).
# Enables DHCP on each ethernet and wireless interface. In case of scripted networking
# (the default) this is the recommended approach. When using systemd-networkd it's
# still possible to use this option, but it's recommended to use it in conjunction
# with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
networking.useDHCP = lib.mkDefault true;
# networking.interfaces.enp5s0.useDHCP = lib.mkDefault true;
# networking.interfaces.wlo1.useDHCP = lib.mkDefault true;
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
powerManagement.cpuFreqGovernor = lib.mkDefault "performance"; # ondemand / powersave / performance
hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}