It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.
- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
* feat(kubevirt-youko): export a shared golden-image store over NFS
* feat(vm-images): upload golden images into the cluster
Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand
The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.
Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.
* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)
zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).
- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
taps to br0 and the host keeps its address on br0. Drop the now-unused
multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
Cilium provides the pod network.
shoukei is a laptop using DHCP and never consumes hostsAddr.shoukei.
Its 192.168.5.108 collided with k3s-prod-1-master-1 and produced a wrong
ssh Host shoukei entry.
- root README: VS Code config moved from home/linux/gui/base/editors.nix to vscode.nix
- READMEs now list both GUI editors (Zed, the primary one, plus VS Code)
- base README: xdg is a directory now, nvidia.nix no longer exists, editors.nix -> vscode.nix
- idols-ai README: disko-fs-media.nix -> disko-fs-data.nix (/persistent/data)
- secrets README: password-store moved from home/base/desktop to home/base/tui
- drop unused link definitions (AstroNvim, wl-clipboard, Catppuccin)
- fix a pre-existing typo flagged by the typos hook (keeped -> kept)
The 2026-09 nixpkgs update changed DRM probe order, renaming the Intel connectors DP-1 -> DP-4 and HDMI-A-2 -> HDMI-A-3. The connector-name based output and open-on-output rules stopped matching, which dropped monitor scale/mode/position and sent every named workspace to the first output. Match by make/model/serial instead, which is stable across connector renumbering.
Guix was enabled on every desktop host, but shoukei had no /gnu mount (store and state lived on the tmpfs root) and it was not used. Drop the module, the modules.guix option, the @guix subvolume and the /var/guix preservation entry.
The restic job needs three agenix secrets that do not exist yet (restic-password, ssh-key-for-restic-backup and an rclone config for the smb-downloads remote). Comment it out until they are provisioned so it does not fail silently every night.
Load both *.yml and *.yaml alert rule files: the Loki rules were named .yaml while only *.yml was loaded, so they silently never ran. Fix the Grafana domain typo (writefo.fun -> writefor.fun). Comment out the postgres-playground datasource, which used the Grafana admin password for a role that has no password and could never authenticate.
* feat(btrbk): make snapshots explicit and guarded
Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.
Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.
Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.
* feat(monitoring): alert on btrbk snapshot failure
Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
RTK compresses only shell output and independent evaluations show little to no end-to-end cost reduction. Drop the package and its install snippets; runtime artifacts were uninstalled separately.
Install microsoft-edge on x86_64 Linux via Home Manager and on macOS via
a brew cask. Enable the chromium and brave-origin Home Manager options
alongside google-chrome, dropping the google-chrome aarch64 fallback to
chromium since google-chrome supports aarch64-linux.
Persist the Brave Origin and microsoft-edge profile directories on the
ephemeral-root hosts.
The r8169 NIC drops carrier on S3 resume, and systemd-networkd reacts by reconfiguring enp130s0: it dropped the static address/default route and deleted foreign routing policy rules (mihomo's 9000-9010 and Tailscale's 5210-5270). clash's TUN then lost its outbound interface and networking stayed broken until clash was restarted.
- IgnoreCarrierLoss=10s keeps the static address/route across the short carrier loss.
- ManageForeignRoutingPolicyRules=false stops networkd from deleting the TUN/VPN ip rules during the reconfiguration.
- render Niri on the Intel iGPU while retaining NVIDIA-connected physical outputs
- encode Sunshine streams with Intel VAAPI and wait for a Niri output before startup
- load VKMS persistently for headless sessions and configure `Virtual-1` at 2560x1600@60
- let Moonlight request the stream bitrate instead of enforcing a 20 Mbps host cap
- remove unsupported Dynamic Boost and global NVIDIA session overrides
- grant Sunshine access to virtual input devices
- add a Nushell recovery helper that starts a temporary greetd/Niri session from SSH
- identify and safely replace normal, standalone, or previous transient Niri sessions
- launch the transient service in the background and verify greetd, Niri, and Sunshine before returning
- ignore local `.worktrees` directories and keep eval regression coverage focused on structural behavior
greetd ran the niri session directly, so exiting the session (e.g.
logout from the lock screen) made greetd immediately restart the
desktop with no authentication.
Compressed writeback cache in front of the disk swapfile keeps cold anon
pages in RAM instead of the SSD, freeing page cache for the ~78GB mmap'd
LLM weights. zstd's ratio keeps ~5G more anon bytes in the capped pool
than lz4 at negligible cost (CPU peaks ~30% on this host); pool capped at
10% so it never competes with the weight cache.
The zram module was auto-imported on all hosts via base scanPaths, and its
swappiness=180/page-cluster=0 tunings leaked onto hosts that disabled zram
(e.g. idols-ai with a disk swapfile, pushing cold anon pages to disk swap).
Wrap the whole module in modules.zram.enable (default true); idols-ai now
turns off zram and its tunings with a single switch.
* security: make firewall secure-by-default, disable explicitly on servers
Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.
* test: fix security-firewall eval test to scalar per-host values
* security: enable AppArmor (complain) on all Linux hosts
* security: bind aquamarine metrics exporters to loopback
* security: restrict k3s kubeconfig file mode to 600
* security: disable SSH X11 forwarding on servers, keep on desktops
* security: add conservative systemd hardening to aquamarine services
* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)
nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.
* fix(aquamarine): scrape same-host exporters over loopback
The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.
* Revert "security: add conservative systemd hardening to aquamarine services"
This reverts commit f9cf99dadb.
* feat(nushell): add trash command using the home trash can
On a tmpfs root with persistent dirs bind-mounted in, `rm --trash`
scatters items into per-mount .Trash-$uid dirs that file managers never
show, and gio refuses to trash on those internal mounts.
Add a `trash` command implementing the freedesktop trash spec
'failsafe' mode: items are moved to ~/.local/share/Trash with
spec-compliant .trashinfo entries, so Thunar can list and restore them.
Also preserve ~/.local/share/Trash on idols-ai so the trash survives
reboots.
* feat(nixos): manage scattered trash dirs with trash-cli retention timer
Replace the hand-written nushell trash command with the established
trash-cli tooling. On a tmpfs root with persistent dirs bind-mounted
in, the trash crate (nushell rm --trash) scatters items into per-mount
.Trash-$uid dirs that file managers never show and nothing cleans up.
- add trash-cli to system packages: trash-list scans/trash-restore
handles every mount point's trash dir
- daily systemd timer runs 'trash-empty 30 -f', purging items older
than 30 days across the home trash and all mount points
- drop the hand-written trash.nu module
- note in preservation.nix: do NOT persist ~/.local/share/Trash; a
bind-mounted trash dir breaks the trash crate's home-topdir match
for files straight under $HOME (it would try /.Trash-$uid, EACCES)
* fix(nixos): order trash-empty after preservation.target
The scattered .Trash-$uid dirs live inside the preservation bind
mounts, which use DefaultDependencies=no and are therefore NOT ordered
after local-fs.target. Add preservation.target to After= so trash-empty
only runs once all those mounts are up.
- Add an offlineHosts list to VictoriaMetrics scrape generation: SBCs
and the whole k3s-prod-1 cluster are powered off, shoukei's exporter
is disabled on the machine; also comment out the dnsmasq-exporter
job (runs on suzi).
- Disable node-exporter on shoukei: a laptop on untrusted networks
should not expose :9100 (the firewall is off repo-wide).
- Route severity none|info alerts to the null receiver; meta alerts
and info noise should never page.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
node-exporter always reports device_error=1 for containerd sandbox shm
bind mounts (tmpfs), producing ~50 bogus critical alerts on the
kubevirt hosts. Verified against live metrics that no non-shm device
errors exist, so nothing real is filtered out.
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>