starred/nix-config
1
0 Fork 0
mirror of https://github.com/ryan4yin/nix-config.git synced 2026-11-13 12:13:46 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

docs(security): record the 2026-10-22 audit follow-up and WA-043

Browse Source 
Baseline sysctls, exporter binds and the #399/#400/#401 rules verified
live on 13 hosts; kiali self-escalation and the list-vs-get probe blind
spot recorded; WA-043 registers the k3s-test-1 Cilium/kube-vip bootstrap
cycle (WA-040 is the Root CA blacklist row); the TODO table gains the
cluster-recovery line. The same section records the manual recovery and
its fix in k8s-gitops, plus the 2026-10-24 full cold start that brought
the VIP back without manual action; WA-043 drops when #54 merges.

Assisted-by: DeepSeek Harness (gpt-6-astra)
This commit is contained in:
Ryan Yin authored and ryan4yin | 二花 committed 2026-10-24 14:16:14 +08:00
1 parent 5088535ac3
commit 8affae69d6
2 files changed
+43 -9
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

SECURITY.md
+38 -5
View File
@@ -162,10 +162,10 @@ checks remain necessary after deployment.
### Implementation status
| Component | Status | Runtime verification |
| ---------------------------------------------------------------- | --------------------------------------- | --------------------------------------------------- |
| Architecture, agent-first policy, audits and canonical API tests | In this repository | Only the dated observations below were checked live |
| Explicit sysctl baseline and IPv6 exporter defence | In `hardening/` and the shared firewall | Not deployed or verified on running hosts |
| Component | Status | Runtime verification |
| ---------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------- |
| Architecture, agent-first policy, audits and canonical API tests | In this repository | Only the dated observations below were checked live |
| Explicit sysctl baseline and IPv6 exporter defence | In `hardening/` and the shared firewall | Live on 13 reachable hosts (readback 2026-10-22); ai root-privileged items still owed |
### Baseline implementation checks
@@ -208,7 +208,7 @@ checks remain necessary after deployment.
- **Remote consoles:** the libvirt VNC consoles listen on `127.0.0.1` only, and the k3s firewall
drops 5900-5903 from pod sources. Reach a console through an SSH tunnel.
- **Pod-to-host firewall:** two independent layers deny pod access to the credential-bearing host
ports (SSH, NFS, Postgres, restic, VNC, Immich, exporter/monitoring). The
ports (SSH, NFS, Postgres, restic, VNC, Immich, etcd, exporter/monitoring). The
`deny-pod-to-node-admin-ports` `CiliumClusterwideNetworkPolicy` in
[k8s-gitops#52](https://github.com/ryan4yin/k8s-gitops/pull/52) denies pod egress to
`host`/`remote-node` on those ports at the pod's own node; the k3s nodes' nftables rule drops the
@@ -254,6 +254,38 @@ export, credential, DNS record or firmware configuration was changed during the
per-role passwords for full loopback `scram-sha-256` is the follow-up; the disabled Grafana
`postgres-playground` datasource also waits on a `playground` password.
### Follow-up audit — 2026-10-22
- Baseline readback on live hosts: the ten unprivileged sysctl keys and the esp4/esp6/rxrpc modprobe
deny are in effect on all 13 reachable hosts; the #399/#400/#401 nft rules, NFS export scope and
pg_hba rules are live. `net.core.bpf_jit_harden` and ai's exact rule order need a root readback;
shoukei and akane were unreachable. k3s MicroVM guests have no system profile — compare
`/run/current-system` with `/run/booted-system` there.
- Exporter sockets are LAN-IPv4-bound on every scraped host except ai's NVIDIA exporter (`*:9835`),
a gap #373 missed; `hosts/idols-ai/hardware-nvidia.nix` now binds it and the `security-exporters`
eval covers every enabled exporter's `listenAddress`.
- Runtime drift on shoryu: the libvirt domains on 5902/5903 predate #400 and still bind wildcard;
only the nft VNC drop protects them until the guests are redefined and restarted.
- The pod-to-node deny lists omitted etcd's client port (2379, bound `192.168.5.114:2379`); both
layers now include it. 2381 (etcd metrics, scraped by in-cluster monitoring) stays reachable by
design.
- ServiceAccount effective access is now measured for all 41 live ns/SA pairs (on-node probes): the
Kiali operator gained a self-escalation path (`update`/`patch clusterroles` on a role bound to
itself, no `bind`/`escalate` verb needed); Kiali and kube-state-metrics deny `get secrets -A` but
allow `list`/`watch`, so probe sets must pair both verbs; 16 accounts enumerate secrets
cluster-wide and cert-manager can delete them. Narrowing proposals target k8s-gitops.
- k3s-test-1 pod networking has been down since the 2026-10-14 full guest cold-restart: Cilium
bootstraps through the VIP while the VIP requires kube-vip's lease over ClusterIP — a cycle the
kube-proxy-replacement switch created (WA-043). The host nftables drops stay loaded, but the
CLiCNP layer enforces nothing while Cilium is down and the nft rule's pod-source-IP precondition
is not met, so the pod-to-node boundary is degraded until the cluster recovers.
- The cluster recovered manually (VIP attached on one master, then handed back to kube-vip), and the
cycle itself is fixed in k8s-gitops: kube-vip's pod now overrides the kubelet-injected
`KUBERNETES_SERVICE_HOST/PORT` to reach its own node's apiserver (port 6443) instead of the
ClusterIP, so the VIP no longer needs Cilium on a cold start. A full cold start of all six guests
on 2026-10-24 brought the VIP back on master-1 with no manual action (6/6 nodes, Cilium 6/6,
kube-vip 3/3, no connection to the ClusterIP); WA-043 is dropped once that change merges.
### Sandbox and agent boundaries
The installed agent CLIs run as the normal user, not inside a repository-defined OS sandbox.
@@ -322,6 +354,7 @@ separate verification.
| Priority | Work | Acceptance / constraints |
| -------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| P1 | Re-verify the pod-to-node drops and the CLiCNP on k3s-test-1 | The bootstrap fix passed a full cold start on 2026-10-24; probe the host nftables rules and the CLiCNP from a pod now that networking is back, and confirm the 2379 deny once #54 merges |
| P1 | Roll out and verify the implemented baseline/exporter rules | Runtime sysctl readback, allowed IPv4 scrape and denied IPv6/non-monitoring scrape tests; preserve Nix/Podman/Cilium/NFS functionality |
| P1 | Review router mappings and tailnet ACLs; narrow service exposure where warranted | Audit was limited to sampled host listeners; test intended and denied access without locking out deployment |
| P1 | Review NFS client scope and root identity requirements | Restrict to required VM hosts/CSI nodes where practical; test provisioning, permissions and existing PVCs before changing squash semantics |
WORKAROUNDS.md
+5 -4
View File
@@ -84,10 +84,11 @@ A known limitation with no fix yet.
An unfinished feature or gap.
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
| ------ | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ---------- | ----------------------- | ------ |
| WA-013 | NixOS tests — `outputs/README.md` | full-host tests still rebuild custom packages (no private cache) and need the host ssh key to decrypt agenix; the shared-firewall security-exporters VM test now runs in CI (KVM); `outputs/x86_64-linux/nixos-tests/idols-ruby.nix` is exposed as `packages.x86_64-linux.ruby-nixos-tests` but is not in `checks` or CI | Resolve when a private cache exists and tests can run per-service | 2024-03-13 | quarterly | active |
| WA-027 | NFS golden store `no_root_squash` — `hosts/12kingdoms-youko/nfs-golden.nix` | the existing PVC subdirectories are root-owned, so enabling `root_squash` would break running workloads until they are migrated; the CSI node plugin is hostNetwork and sources from the allowlisted node IPs | Drop when the PVC subdirectories are migrated and `root_squash` is enabled | 2026-10-07 | next maintenance window | active |
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
| ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------------------- | ------ |
| WA-013 | NixOS tests — `outputs/README.md` | full-host tests still rebuild custom packages (no private cache) and need the host ssh key to decrypt agenix; the shared-firewall security-exporters VM test now runs in CI (KVM); `outputs/x86_64-linux/nixos-tests/idols-ruby.nix` is exposed as `packages.x86_64-linux.ruby-nixos-tests` but is not in `checks` or CI | Resolve when a private cache exists and tests can run per-service | 2024-03-13 | quarterly | active |
| WA-027 | NFS golden store `no_root_squash` — `hosts/12kingdoms-youko/nfs-golden.nix` | the existing PVC subdirectories are root-owned, so enabling `root_squash` would break running workloads until they are migrated; the CSI node plugin is hostNetwork and sources from the allowlisted node IPs | Drop when the PVC subdirectories are migrated and `root_squash` is enabled | 2026-10-07 | next maintenance window | active |
| WA-043 | k3s-test-1 bootstrap cycle: kube-vip's API client pinned to the local apiserver — k8s-gitops `infra/pre-controllers/base/kube-vip/helm-release.yaml` | a full cold start of all six microVM guests (2026-10-14) left no node holding the VIP: the in-cluster kube-vip could only win its lease through ClusterIP, ClusterIP is Cilium's, and Cilium's config init crash-loops until the VIP exists (`no route to host 192.168.5.193:6443`); all pod networking stays down until broken by hand. The pod now overrides the kubelet-injected service env (`KUBERNETES_SERVICE_HOST` = `status.hostIP`, `KUBERNETES_SERVICE_PORT` = 6443), so kube-vip reaches the local apiserver before any CNI exists | Drop once k8s-gitops#54 is merged and reconciled; the 2026-10-24 full cold start already returned the VIP without manual action | 2026-10-24 | when #54 merges | active |
## Related