Commit Graph
536 Commits
Author SHA1 Message Date
ryan4yin | 二花 01c6b3290a chore: remove the never-deployed k3s-prod-1 cluster (#284)
It was never bootstrapped (no flux deploy key, VMs commented out since
2024-10) and only carried a stale duplicate of the test cluster's config.

- hosts/k8s/k3s-prod-1-{master,worker}-{1,2,3} + flake outputs + the home module
- the k3s-prod-1-token agenix entry (the .age file lives in the private secrets repo)
- the k3s-prod-1-* IPs, the offlineHosts entries, the Justfile recipes and the docs
2026-09-21 12:53:43 +08:00
ryan4yin | 二花 a4609501c4 feat(kubevirt-youko): serve VM golden images from a shared NFS store (#282)
* feat(kubevirt-youko): export a shared golden-image store over NFS

* feat(vm-images): upload golden images into the cluster

Replace the rsync-to-orangepi5 step with a CDI upload into the golden-image
PVCs (vms/golden-images in k8s-gitops), so VM images no longer depend on the
single-board file server.
2026-09-21 11:21:33 +08:00
ryan4yin | 二花 12505b415c refactor(kubevirt): memory tuning - drop static hugepages, add zram + earlyoom (#280)
* refactor(kubevirt): drop the static 1G hugetlb pool, use THP on demand

The hosts pre-reserved 48/48/15 GiB of 1G hugepages, but the VMs only used 32/16/11 GiB, so ~52 GiB was stranded (hugetlb is exclusive and cannot be overcommitted or shared with the host / non-hugepage VMs). It also blocked scheduling (a 16 GiB hugepage VM cannot fit on a host with fewer free pages, e.g. youko with 15/24 GiB) and made the full intended VM set (idols 3x8G + k3s-test 3x16G + k3s-prod 3x4G + 3x16G = 132 GiB vs 146 GiB physical) impossible to fit.

Use transparent huge pages on demand (madvise) instead, and let the VMs use ordinary memory (instancetypes updated in k8s-gitops). This frees the stranded memory, removes the scheduling rigidity, and makes overcommit/swap usable again.

* feat(kubevirt): add earlyoom; stop overriding swappiness (zram already enabled)

zram is already provided by modules/nixos/base/zram.nix (enabled by default: 50% RAM, zstd, prio 100 + swappiness/watermark tunings). But genKubeVirtHostModule hard-coded vm.swappiness = 0, which overrode the modules mkDefault 180 and disabled swapping entirely - so the zram device was never used (Swap used: 0B).

- Remove the hard-coded vm.swappiness (let the zram modules tuning apply); the disk swapfile from disko stays as the last resort.
- Add services.earlyoom to kill the greediest process before the host thrashes / OOMs.
2026-09-21 01:39:51 +08:00
ryan4yin | 二花 64a81f16ec fix(victoriametrics): enable dedup to accept duplicate samples (#279)
vmagent (and other senders) can deliver the same series+timestamp twice (retries/HA), which vmsingle rejected with "duplicate sample for timestamp; overrides not allowed". Set -dedup.minScrapeInterval=30s (>= the senders scrapeInterval) so duplicates collapse instead of being rejected.
2026-09-21 01:21:09 +08:00
ryan4yin | 二花 88697ce861 feat(kubevirt): use a Linux bridge + Cilium instead of OVS/flannel (#278)
- genKubeVirtHostModule: replace the Open vSwitch bridge (ovsbr1) with a Linux
  bridge br0 for the VMs secondary network; the bridge CNI plugin attaches the VM
  taps to br0 and the host keeps its address on br0. Drop the now-unused
  multus-cni host package.
- kubevirt hosts: stop forcing disableFlannel = false, so k3s disables flannel and
  Cilium provides the pod network.
2026-09-21 00:37:15 +08:00
Ryan Yin 40b3786f14 docs(kubevirt): refine switch vs boot guidance 2026-09-20 22:45:22 +08:00
Ryan Yin 54016a9da2 fix: address nixpkgs rename/deprecation warnings 2026-09-20 22:42:06 +08:00
Ryan Yin fb522af5c0 feat(kubevirt): cap journald disk usage on all VMs 2026-09-20 22:22:50 +08:00
Ryan Yin 68bea13405 docs(kubevirt): document boot+reboot for KubeVirt hosts 2026-09-20 22:22:47 +08:00
Ryan Yin 4e66a6de57 chore: remove insecure modelscope package 2026-09-20 21:36:23 +08:00
Ryan Yin 9bb608a577 fix(idols-aquamarine): migrate journald.extraConfig to settings.Journal 2026-09-20 21:36:21 +08:00
Ryan Yin 69f60095ae fix(networking): drop shoukei static address
shoukei is a laptop using DHCP and never consumes hostsAddr.shoukei.
Its 192.168.5.108 collided with k3s-prod-1-master-1 and produced a wrong
ssh Host shoukei entry.
2026-09-20 20:26:17 +08:00
Ryan Yin 98fbf010b8 docs: fix broken links and editor references in readmes
- root README: VS Code config moved from home/linux/gui/base/editors.nix to vscode.nix
- READMEs now list both GUI editors (Zed, the primary one, plus VS Code)
- base README: xdg is a directory now, nvidia.nix no longer exists, editors.nix -> vscode.nix
- idols-ai README: disko-fs-media.nix -> disko-fs-data.nix (/persistent/data)
- secrets README: password-store moved from home/base/desktop to home/base/tui
- drop unused link definitions (AstroNvim, wl-clipboard, Catppuccin)
- fix a pre-existing typo flagged by the typos hook (keeped -> kept)
2026-09-19 15:23:09 +08:00
Ryan Yin 33383414a2 fix(niri): match outputs by EDID to survive connector renumbering
The 2026-09 nixpkgs update changed DRM probe order, renaming the Intel connectors DP-1 -> DP-4 and HDMI-A-2 -> HDMI-A-3. The connector-name based output and open-on-output rules stopped matching, which dropped monitor scale/mode/position and sent every named workspace to the first output. Match by make/model/serial instead, which is stable across connector renumbering.
2026-09-19 13:26:24 +08:00
Ryan Yin f23d5d4245 docs(monitoring): note dead-man switch and backup freshness TODOs 2026-09-19 12:14:20 +08:00
Ryan Yin 3318103f8a chore(guix): remove the unused Guix integration
Guix was enabled on every desktop host, but shoukei had no /gnu mount (store and state lived on the tmpfs root) and it was not used. Drop the module, the modules.guix option, the @guix subvolume and the /var/guix preservation entry.
2026-09-19 12:11:15 +08:00
Ryan Yin fc8b9e5ba0 chore(aquamarine): disable the broken restic backup
The restic job needs three agenix secrets that do not exist yet (restic-password, ssh-key-for-restic-backup and an rclone config for the smb-downloads remote). Comment it out until they are provisioned so it does not fail silently every night.
2026-09-19 12:11:11 +08:00
Ryan Yin d78f6badd4 fix(aquamarine): repair monitoring config
Load both *.yml and *.yaml alert rule files: the Loki rules were named .yaml while only *.yml was loaded, so they silently never ran. Fix the Grafana domain typo (writefo.fun -> writefor.fun). Comment out the postgres-playground datasource, which used the Grafana admin password for a role that has no password and could never authenticate.
2026-09-19 12:11:07 +08:00
ryan4yin | 二花 f8d12ada24 feat(btrbk): make snapshots explicit, guarded, and stored in @snapshots (#271)
* feat(btrbk): make snapshots explicit and guarded

Replace the always-on, hardcoded module with modules.btrbk options (enable/volume/subvolume/snapshotDir/target/onCalendar), and assert the btrfs top-level subvolume is mounted so a missing mount fails evaluation instead of silently failing at runtime.

Store snapshots in the @snapshots subvolume (snapshot_dir=@snapshots) rather than next to the live subvolume, make the off-host target explicit and off by default, and document that local snapshots are not off-host backups.

Enable the feature explicitly on idols-ai, shoukei and the three kubevirt hosts; qemu-guest now disables it through the option. Add eval tests for shoukei and ai.

* feat(monitoring): alert on btrbk snapshot failure

Add HostBtrbkSnapshotFailed on node_systemd_unit_state for btrbk-btrbk.service, so a broken snapshot schedule alerts instead of only surfacing in the journal.
2026-09-19 11:43:11 +08:00
Ryan Yin 501c6af220 chore(agents): remove rtk integration
RTK compresses only shell output and independent evaluations show little to no end-to-end cost reduction. Drop the package and its install snippets; runtime artifacts were uninstalled separately.
2026-09-19 09:47:52 +08:00
Ryan Yin d80efddd17 fix(agents): leave Codex config unmanaged 2026-09-18 21:20:33 +08:00
Ryan Yin d47710ebe2 fix(agents): merge Codex privacy settings 2026-09-18 21:20:33 +08:00
Ryan Yin 8c62b5433d chore(agents): remove Cursor integrations 2026-09-18 21:20:33 +08:00
Ryan Yin bcee9f2ac8 feat(agents): add Crush and disable usage tracking 2026-09-18 21:20:33 +08:00
Ryan Yin 23258a06d4 feat(agents): add Pi and OMP 2026-09-18 21:20:33 +08:00
Ryan Yin 46266dd8ea chore(agents): remove Claude integrations 2026-09-18 21:20:33 +08:00
Ryan Yin b10b059d1f chore: add microsoft-edge, chromium and brave-origin
Install microsoft-edge on x86_64 Linux via Home Manager and on macOS via
a brew cask. Enable the chromium and brave-origin Home Manager options
alongside google-chrome, dropping the google-chrome aarch64 fallback to
chromium since google-chrome supports aarch64-linux.

Persist the Brave Origin and microsoft-edge profile directories on the
ephemeral-root hosts.
2026-09-16 18:51:23 +08:00
Ryan Yin c8642fcac4 feat(herdr): declare config in home-manager and persist state 2026-09-14 13:18:57 +08:00
Ryan Yin 77e31bd4cb fix(idols-ai): keep network config across S3 resume
The r8169 NIC drops carrier on S3 resume, and systemd-networkd reacts by reconfiguring enp130s0: it dropped the static address/default route and deleted foreign routing policy rules (mihomo's 9000-9010 and Tailscale's 5210-5270). clash's TUN then lost its outbound interface and networking stayed broken until clash was restarted.

- IgnoreCarrierLoss=10s keeps the static address/route across the short carrier loss.
- ManageForeignRoutingPolicyRules=false stops networkd from deleting the TUN/VPN ip rules during the reconfiguration.
2026-09-14 10:54:33 +08:00
Ryan Yin 324e5d8820 feat(moonlight): persist client data 2026-09-04 20:21:52 +08:00
Ryan Yin 83833bf871 fix: use valid shoukei logind keys 2026-09-03 23:26:03 +08:00
Ryan Yin 1ec29059ad fix(sunshine): stabilize remote Niri sessions 2026-09-01 22:32:19 +08:00
ryan4yin | 二花 9db935bc87 fix(sunshine): use Intel rendering and recover remote Niri sessions (#268)
- render Niri on the Intel iGPU while retaining NVIDIA-connected physical outputs
- encode Sunshine streams with Intel VAAPI and wait for a Niri output before startup
- load VKMS persistently for headless sessions and configure `Virtual-1` at 2560x1600@60
- let Moonlight request the stream bitrate instead of enforcing a 20 Mbps host cap
- remove unsupported Dynamic Boost and global NVIDIA session overrides
- grant Sunshine access to virtual input devices
- add a Nushell recovery helper that starts a temporary greetd/Niri session from SSH
- identify and safely replace normal, standalone, or previous transient Niri sessions
- launch the transient service in the background and verify greetd, Niri, and Sunshine before returning
- ignore local `.worktrees` directories and keep eval regression coverage focused on structural behavior
2026-09-01 03:40:48 -06:00
Ryan Yin c6bbb69b3d fix(shoukei): muvm failed to build currently 2026-08-30 23:02:51 +08:00
Ryan Yin 099752e89f fix(shoukei): restore tuigreet login, autologin bypassed the lock screen
greetd ran the niri session directly, so exiting the session (e.g.
logout from the lock screen) made greetd immediately restart the
desktop with no authentication.
2026-08-30 21:47:57 +08:00
Ryan Yin a0d55a298e feat(remote-desktop): enable Sunshine on ai and Moonlight on darwin 2026-08-30 12:57:09 +08:00
Ryan Yin f08b45d2bc feat(editors): restore vscode alongside zed for build123d workflows
- linux: programs.vscode via pkgs-master with gnome-libsecret password store
- darwin: visual-studio-code cask + aerospace window rules
- idols-ai: persist .vscode and .config/Code (Settings Sync handles extensions)
- xdg: code.desktop editor association and vscode:// scheme handler
2026-08-29 17:05:44 +08:00
Ryan Yin db953c0504 feat(idols-ai): enable zswap with zstd and 10% pool
Compressed writeback cache in front of the disk swapfile keeps cold anon
pages in RAM instead of the SSD, freeing page cache for the ~78GB mmap'd
LLM weights. zstd's ratio keeps ~5G more anon bytes in the capped pool
than lz4 at negligible cost (CPU peaks ~30% on this host); pool capped at
10% so it never competes with the weight cache.
2026-08-29 11:32:15 +08:00
Ryan Yin 4b7375c077 fix(nixos): gate zram kernel sysctl tunings behind modules.zram.enable
The zram module was auto-imported on all hosts via base scanPaths, and its
swappiness=180/page-cluster=0 tunings leaked onto hosts that disabled zram
(e.g. idols-ai with a disk swapfile, pushing cold anon pages to disk swap).
Wrap the whole module in modules.zram.enable (default true); idols-ai now
turns off zram and its tunings with a single switch.
2026-08-29 11:16:12 +08:00
ryan4yin | 二花 21d64e7add security: hardening pass 1 (AppArmor, firewall default, loopback metrics, k3s perms, SSH X11) (#266)
* security: make firewall secure-by-default, disable explicitly on servers

Flip the base firewall default to ON so new hosts are protected by default. Servers keep the firewall off (trusted internal LAN, WAN protected at the router) via explicit overrides in modules/nixos/server/{server,server-aarch64}.nix. Behavior-preserving for all 18 current hosts; adds a security-firewall eval test guarding the per-host state.

* test: fix security-firewall eval test to scalar per-host values

* security: enable AppArmor (complain) on all Linux hosts

* security: bind aquamarine metrics exporters to loopback

* security: restrict k3s kubeconfig file mode to 600

* security: disable SSH X11 forwarding on servers, keep on desktops

* security: add conservative systemd hardening to aquamarine services

* docs(apparmor): correct FHS-alias comment (nixpkgs has no broad FHS layer)

nixpkgs only creates /run/current-system, /usr/bin/env and /bin/sh; it does not
provide a broad FHS->store alias tree, so FHS-oriented abstractions are incomplete
on NixOS. They are safe only because the profiles run in complain mode.

* fix(aquamarine): scrape same-host exporters over loopback

The v2ray/postgres/sftpgo exporters were bound to 127.0.0.1 but VictoriaMetrics
scraped them via the host's routable IP, so those scrapes refused connections.
Point the three same-host scrape targets at 127.0.0.1 (VM runs on the same host).
node-exporter keeps the host IP since it binds 0.0.0.0.

* Revert "security: add conservative systemd hardening to aquamarine services"

This reverts commit f9cf99dadb.
2026-08-27 00:13:01 +08:00
ryan4yin | 二花 6e0d44c50d feat(nixos): manage scattered trash dirs with trash-cli retention timer (#265)
* feat(nushell): add trash command using the home trash can

On a tmpfs root with persistent dirs bind-mounted in, `rm --trash`
scatters items into per-mount .Trash-$uid dirs that file managers never
show, and gio refuses to trash on those internal mounts.

Add a `trash` command implementing the freedesktop trash spec
'failsafe' mode: items are moved to ~/.local/share/Trash with
spec-compliant .trashinfo entries, so Thunar can list and restore them.

Also preserve ~/.local/share/Trash on idols-ai so the trash survives
reboots.

* feat(nixos): manage scattered trash dirs with trash-cli retention timer

Replace the hand-written nushell trash command with the established
trash-cli tooling. On a tmpfs root with persistent dirs bind-mounted
in, the trash crate (nushell rm --trash) scatters items into per-mount
.Trash-$uid dirs that file managers never show and nothing cleans up.

- add trash-cli to system packages: trash-list scans/trash-restore
  handles every mount point's trash dir
- daily systemd timer runs 'trash-empty 30 -f', purging items older
  than 30 days across the home trash and all mount points
- drop the hand-written trash.nu module
- note in preservation.nix: do NOT persist ~/.local/share/Trash; a
  bind-mounted trash dir breaks the trash crate's home-topdir match
  for files straight under $HOME (it would try /.Trash-$uid, EACCES)

* fix(nixos): order trash-empty after preservation.target

The scattered .Trash-$uid dirs live inside the preservation bind
mounts, which use DefaultDependencies=no and are therefore NOT ordered
after local-fs.target. Add preservation.target to After= so trash-empty
only runs once all those mounts are up.
2026-08-23 23:40:53 +08:00
Ryan Yin 6f100d94a9 feat(gh): persist user-managed configuration 2026-08-04 17:45:24 +08:00
Ryan Yin cce15300d6 feat(niri): reorganize workspaces 2026-08-03 22:42:37 +08:00
Ryan Yin 6264190c49 fix(idols-ai): reset camera after resume 2026-08-01 10:04:22 +08:00
Ryan Yin 41512097e5 feat(grafana): add kubernetes service and storage dashboards 2026-07-31 17:34:52 +08:00
Ryan Yin 071212e40f feat(niri): reorganize workspace layout 2026-07-31 12:06:27 +08:00
Ryan Yin 64ecc1cf66 fix(grafana): scope hwmon joins to selected node 2026-07-27 20:57:10 +08:00
Ryan Yin 095dc5dee8 feat(desktop): enable the firewall on desktop machines
Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
2026-07-26 22:26:01 +08:00
Ryan Yin 74598b6f6b chore(monitoring): stop scraping offline hosts, mute noisy alert levels
- Add an offlineHosts list to VictoriaMetrics scrape generation: SBCs
  and the whole k3s-prod-1 cluster are powered off, shoukei's exporter
  is disabled on the machine; also comment out the dnsmasq-exporter
  job (runs on suzi).
- Disable node-exporter on shoukei: a laptop on untrusted networks
  should not expose :9100 (the firewall is off repo-wide).
- Route severity none|info alerts to the null receiver; meta alerts
  and info noise should never page.

Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
2026-07-26 22:19:56 +08:00
Ryan Yin ec208ab4d1 fix(monitoring): exclude shm from HostFilesystemDeviceError
node-exporter always reports device_error=1 for containerd sandbox shm
bind mounts (tmpfs), producing ~50 bogus critical alerts on the
kubevirt hosts. Verified against live metrics that no non-shm device
errors exist, so nothing real is filtered out.

Signed-off-by: Ryan Yin <xiaoyin_c@qq.com>
2026-07-26 15:28:41 +08:00