mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:36 +01:00
ryan4yin | 二花
0dcbbb02e9
Hosts now accept only the two CanoKeys' resident SSH keys for ryan, plus the romantic recovery key for root; per-host user keys stop working. Darwin builds work again: the shared base no longer sets nixpkgs.config next to nix-darwin's nixpkgs.pkgs. Assisted-by: DeepSeek Harness (claude-opus-5-5) Assisted-by: DeepSeek Harness (deepseek-flash)
436 lines
44 KiB
Markdown
436 lines
44 KiB
Markdown
# OS and Homelab Security
|
|
|
|
This is a personal infrastructure repository, not a supported security product. This document
|
|
describes the intended architecture, controls present in source, their limitations, and remaining
|
|
work. A configured control is not evidence that it is active on a running machine.
|
|
|
|
Related documents: [application hardening](../hardening/README.md) for sandbox wrappers and Linux
|
|
hardening notes, and [WORKAROUNDS.md](./WORKAROUNDS.md) for temporary exceptions and known gaps.
|
|
|
|
## Reporting problems
|
|
|
|
For non-sensitive configuration bugs, open an issue with the affected host role and a redacted
|
|
reproduction. Do not post credentials, kubeconfigs, private keys, secret contents, or exploit
|
|
details that would expose a live service. For sensitive findings, contact the maintainer privately
|
|
through the contact information on their GitHub profile first. There is no guaranteed response SLA.
|
|
|
|
## Threat model and priorities
|
|
|
|
Protect administrator credentials, desktop application data, cluster/storage state, and the ability
|
|
to recover after compromise or an unsuccessful update. Relevant attackers include compromised
|
|
applications and dependencies, malicious workloads, and devices or accounts on trusted networks.
|
|
Physical theft and boot tampering matter for desktops and physical hosts too.
|
|
|
|
Prioritize **patching, reducing exposure, least privilege, isolation, and recovery**, in that order
|
|
for each affected service. Kernel mitigations and module deny lists are defence in depth, not fixes
|
|
for vulnerabilities. A container shares its host kernel; a VM provides a separate kernel but still
|
|
exposes a hypervisor and host-side device/share implementations. Neither makes arbitrary code safe.
|
|
|
|
Use proportionate hardening: preserve normal performance, developer workflows and required cluster
|
|
features. Do not globally disable user namespaces, io_uring, SMT or hardware acceleration, switch to
|
|
a hardened kernel, or enforce untested AppArmor policies just to maximize restrictions. Prefer
|
|
small, reversible controls and narrow exceptions; benchmark before adopting expensive mitigations.
|
|
|
|
## Architecture and trust boundaries
|
|
|
|
| Boundary | Current design | Limitation |
|
|
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| Administration | Key-based SSH; root key login retained for Colmena/MicroVM deployment; only keys declared in Nix are honoured, and no password or keyboard-interactive login; remote-builder host keys only reach `nix-daemon` | Deployment credentials are root-equivalent file keys, usable by any same-user process through the desktop SSH agent; one stolen key reaches every host |
|
|
| Host network | Shared nftables firewall trusts the home LAN, tailnet interface and local Podman bridge; IPv6 link-local is accepted | These are broad trust zones, not service-level least privilege; router exposure and tailnet ACLs require separate verification |
|
|
| Monitoring | node/smartctl exporters bind the host LAN IPv4 (9100/9633); firewall rules restrict exporter ports 9100, 9835 and 9633 to the monitoring host over IPv4. RAPL counters are exposed only on x86 server hosts, to `node-exporter` through the `rapl` group. | nvidia_gpu_exporter (9835) still binds wildcard; the IPv6 denial for those three ports is in the shared firewall; current running hosts require separate verification. The node_exporter account can measure package energy for local side channels; interactive users, desktops, and guests keep the kernel-default root-only permission. |
|
|
| VM hosts | shoryu, shushou and youko run guests behind `br0`; ruby/kana are libvirt guests | A bridge is not network isolation; host compromise affects its guests |
|
|
| k3s | Three control-plane and three worker MicroVMs; Cilium, API VIP and NFS CSI | Cluster-admin, privileged/host-mounted pods and writable NFS data are powerful trust boundaries |
|
|
| MicroVM storage | Host store shared read-only through virtiofs; guest `/etc`, `/var` and `/home` persisted in separate images | Read-only store does not isolate the host-side virtiofs implementation or protect writable guest state |
|
|
| Desktop apps | Selected apps use nixpak/bubblewrap; Podman is rootless | Rules and shared directories vary per app; user namespaces are needed for these sandboxes |
|
|
| Security keys | idols-ai/shoukei use a CanoKey for touch-based sudo (pam_u2f, password fallback; servers via pam_rssh and the forwarded agent), a resident `ed25519-sk` SSH key per CanoKey as the only user SSH keys for GitHub and all hosts, FIDO2 LUKS unlock on shoukei and on-card OpenPGP subkeys; key management happens only in the phone app ([setup](./security-keys.md)) | Touch-only sudo can be triggered by any same-user process, and on a server by root through the forwarded agent; losing both CanoKeys leaves only root logins with the `romantic` recovery key |
|
|
| Secrets | agenix declarations here, encrypted data in the private nix-secrets repository | Recipients and plaintext file permissions must be reviewed; root on a recipient can read its decrypted secrets |
|
|
| Supply chain | Locked flake inputs; explicit binary-cache trust keys on NixOS; normal user is not a Nix trusted-user | A trusted cache or imported Nix module can supply code executed as root; a lock file is not a security audit |
|
|
| Recovery | NixOS generations, btrbk/restic, hourly read-only snapshots of the REST store, and documented backup procedures | A rollback does not undo compromised data or secrets; backup configuration alone does not prove restore works; the offsite immutable copy is still planned |
|
|
|
|
Sources: [host layout](../hosts/README.md), [k3s deployment](../hosts/k8s/README.md),
|
|
[firewall](../modules/nixos/base/networking/firewall.nix), [SSH](../modules/nixos/base/ssh.nix),
|
|
[Nix trust](../modules/base/nix.nix), [secrets](../secrets/README.md), and [backup](./BACKUP.md).
|
|
Cluster resources are maintained in the separate k8s-gitops repository.
|
|
|
|
The Macs set `nix.enable = false` ([nix-core.nix](../modules/darwin/nix-core.nix)), so nix-darwin
|
|
writes no `/etc/nix/nix.conf`: the substituters, trusted keys and access-token include declared here
|
|
do not reach them. Their Nix settings come from the Determinate installation, which this repository
|
|
does not manage; those settings were not verified on the Macs.
|
|
|
|
## Kernel and process baseline
|
|
|
|
The following explicit, overridable baseline is implemented in
|
|
[kernel-hardening.nix](../hardening/kernel-hardening.nix), which holds the values; the table below
|
|
summarizes them. It has Linux eval coverage and explicit ARM/RISC-V server imports. Source
|
|
implementation is not runtime verification. Some values are already kernel/systemd/NixOS defaults;
|
|
Darwin does not receive Linux sysctls.
|
|
|
|
| Setting | Value | Effect and compatibility |
|
|
| ------------------------------------------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `kernel.dmesg_restrict` | 1 | Kernel log access requires CAP_SYSLOG; journal access has separate permissions |
|
|
| `kernel.kptr_restrict` | 2 | Hide pointers printed with `%pK`, even from privileged readers; not a blanket ban on address leaks |
|
|
| `kernel.yama.ptrace_scope` | 1 | Restrict unrelated-process tracing; child-process debugging and explicit PR_SET_PTRACER exceptions remain possible |
|
|
| `kernel.perf_event_paranoid` | 2 | Restrict unprivileged kernel profiling; user-space profiling remains available; do not assume distro-specific 3/4 semantics |
|
|
| `kernel.unprivileged_bpf_disabled` | 2 | Disable unprivileged `bpf()` with an administrator-reversible setting; privileged Cilium/agents retain access |
|
|
| `net.core.bpf_jit_harden` | 1 | Harden only unprivileged BPF JIT; do not enable JIT blinding for privileged Cilium/observability workloads |
|
|
| `vm.unprivileged_userfaultfd` | 0 | Restrict unprivileged kernel-mode fault handling, not all userfaultfd usage; `/dev/userfaultfd` permissions are a separate boundary |
|
|
| `fs.protected_symlinks`, `fs.protected_hardlinks` | 1 | Mitigate cross-user link attacks; these controls are boolean |
|
|
| `fs.protected_fifos`, `fs.protected_regular` | 2 | Restrict unsafe O_CREAT access in world/group-writable sticky directories; shared temporary-file workflows may need adjustment |
|
|
|
|
Some values already come from kernel/systemd/NixOS defaults. Explicit declarations make the desired
|
|
baseline reviewable; they do not imply the previous system had no protection. Unsupported kernel
|
|
features may lack their sysctl paths, especially on custom kernels: runtime verification is
|
|
required.
|
|
|
|
JIT hardening is deliberately limited to unprivileged programs rather than forced on privileged
|
|
agents. Together with `unprivileged_bpf_disabled=2`, this preserves Cilium's normal privileged BPF
|
|
path while retaining protection if an administrator temporarily permits unprivileged BPF. This is a
|
|
configuration trade-off, not a measured claim of zero overhead.
|
|
|
|
User namespaces remain enabled: disabling them globally conflicts with Nix sandboxing and breaks
|
|
rootless containers and application sandboxes. Nonprivileged seccomp filters are not disabled by the
|
|
unprivileged `bpf()` restriction. Do not disable Cilium's privileged BPF, tc, container networking,
|
|
NFS, or required VM/filesystem modules in pursuit of a larger deny list.
|
|
|
|
The existing `esp4`, `esp6`, `rxrpc` deny list blocks modprobe/autoload paths. It does not unload
|
|
already loaded modules, affect built-in code, or stop a privileged direct module insertion. Verify
|
|
an advisory and affected kernel versions before adding CVE claims; search snippets are not evidence.
|
|
|
|
### Boot integrity and mandatory access control
|
|
|
|
- ai configures Lanzaboote in [secureboot.nix](../hosts/idols-ai/secureboot.nix). Confirm firmware
|
|
key enrollment and actual Secure Boot state with `sbctl status`; do not assume all hosts have it.
|
|
- Secure Boot, signed boot images, kernel lockdown and module signature enforcement are distinct
|
|
controls. Test custom kernels and NVIDIA/out-of-tree modules before extending enforcement.
|
|
- AppArmor is enabled, but the only explicit policy, sudo's, is in complain mode. Complain mode logs
|
|
rather than blocks. Existing FHS-oriented abstractions need Nix store path review; this is not
|
|
broad enforced application confinement.
|
|
- Systemd service sandboxing is per-service, not a global policy. The exposed services nixpkgs
|
|
leaves weakly sandboxed (caddy, rustfs, vmalert, whisper-asr) use the preset in
|
|
[systemdHardening.nix](../lib/systemdHardening.nix). Review capabilities, writable paths, network
|
|
access and syscall requirements before tightening an individual service.
|
|
|
|
See [application hardening](../hardening/README.md) for wrapper details.
|
|
|
|
### Audit trail
|
|
|
|
`modules.security.audit` ([auditd.nix](../modules/nixos/base/auditd.nix)) runs auditd on youko,
|
|
shoryu, shushou and aquamarine. It records, for login sessions only (SSH, and sudo or su from one):
|
|
every root command (`root-session`), access to decrypted secrets and SSH host keys
|
|
(`credential-access`), writes to SSH, account/PAM, Nix trust and audit configuration, module loads,
|
|
and denials.
|
|
|
|
- Services are not audited: a `never,task` rule gives tasks without a login uid no audit context, so
|
|
virtiofsd, hypervisors, NFS, PostgreSQL and Nix builds skip the per-syscall audit path. A
|
|
compromised service is therefore invisible until it opens a login session.
|
|
- Logs stay on the host and root can erase them; shipping them off-host is a P1 TODO.
|
|
- One watch on a missing path makes `auditctl -R` stop and leaves auditing off;
|
|
`nix build .#checks.x86_64-linux.security-auditd` loads the rules in a VM.
|
|
|
|
Query as root; sshd is built without libaudit, so use PAM's `USER_START` records for logins:
|
|
|
|
```sh
|
|
ausearch -i -m USER_START --start today # logins with address and session id
|
|
ausearch -i -k root-session --start today # root commands, with auid and ses
|
|
ausearch -i --session <ses> # everything one session did
|
|
```
|
|
|
|
## Updating, deploying and recovering
|
|
|
|
1. Check the advisory and affected package/kernel; prioritize active exploitation and reachable
|
|
attack paths. Review changed inputs and cache trust instead of treating updates as inherently
|
|
safe.
|
|
2. Run `just test` (must pass), build affected systems/runners and review closure differences.
|
|
Evaluate tests cannot prove runtime enforcement or workload compatibility.
|
|
3. Record the old system generation/runner and ensure backups and recovery access exist. Use `boot`
|
|
mode for bridge/network-stack changes or broad VM-host updates.
|
|
4. Deploy MicroVM closures to the **physical host**, not the guest's read-only store. Install the
|
|
reviewed runner and activate/restart through the documented MicroVM path. Preserve all images.
|
|
5. Keep control-plane restarts sequential to retain etcd quorum. Worker concurrency is an explicit
|
|
availability trade-off; simultaneous worker restarts can interrupt all worker workloads.
|
|
6. Verify the active/booted generation, guest boot ID, failed units, API VIP `/readyz`, Nodes,
|
|
Cilium, CSI and workloads. Updating a kernel on disk does not patch the running kernel: reboot
|
|
and verify.
|
|
7. On failure, restore the recorded generation/runner and recheck health before continuing. Preserve
|
|
rollback roots until stable. If compromise is suspected, isolate first, rotate affected secrets,
|
|
and restore verified data; merely switching generations is not incident recovery.
|
|
|
|
For the sysctl baseline, query the keys above with `sysctl <key>`, check
|
|
`journalctl -b -u systemd-sysctl`, and test rootless Podman, a sandboxed Nix build, desktop
|
|
sandboxes, debugging, and cluster networking/storage after activation. Do not globally relax
|
|
hardening to hide one application's incompatibility: make a narrow documented exception and test it.
|
|
|
|
These sysctls are reversible by administrator configuration, unlike `unprivileged_bpf_disabled=1` or
|
|
`kexec_load_disabled=1`, which cannot be relaxed without rebooting. This baseline does not enable
|
|
the latter or `security.protectKernelImage` (which also disables hibernation).
|
|
|
|
## API endpoint and TLS
|
|
|
|
The canonical endpoint is **`https://test-cluster-1.writefor.fun:6443`**. DNS points it to
|
|
kube-vip's `192.168.5.193`; the three servers include the DNS name in their serving-certificate
|
|
SANs, and the joining servers/agents use that domain. The first server initializes the cluster
|
|
without a join address. This is already a domain-based HA endpoint, not a bare-IP client design.
|
|
|
|
Use the existing authenticated client configuration to verify it:
|
|
|
|
```sh
|
|
k3s kubectl --server=https://test-cluster-1.writefor.fun:6443 --request-timeout=15s get --raw=/readyz
|
|
```
|
|
|
|
Bare-IP access may fail TLS hostname verification because the certificate does not include the VIP
|
|
IP SAN. That is not a fault in the canonical endpoint and does not require certificate rotation. If
|
|
bare-IP disaster-recovery access becomes an explicit requirement, add that SAN through normal
|
|
configuration and certificate lifecycle procedures. Never bypass TLS verification to conceal a
|
|
hostname mismatch. Eval tests protect the domain join endpoint and server SAN declarations; live TLS
|
|
checks remain necessary after deployment.
|
|
|
|
## Implementation status and completed audit findings
|
|
|
|
### Implementation status
|
|
|
|
| Component | Status | Runtime verification |
|
|
| ---------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
| Architecture, agent-first policy, audits and canonical API tests | In this repository | Only the dated observations below were checked live |
|
|
| Explicit sysctl baseline and IPv6 exporter defence | In `hardening/` and the shared firewall | Live on 13 reachable hosts (readback 2026-10-22); ai root-privileged items still owed |
|
|
|
|
### Baseline implementation checks
|
|
|
|
- An explicit, overridable sysctl baseline preserves user namespaces and privileged BPF performance.
|
|
- IPv6 exporter denial precedes broad LAN/tailnet/container rules. The monitoring configuration uses
|
|
IPv4 targets for these ports, so no scrape endpoint migration is needed.
|
|
- x86_64/aarch64 eval checks cover baseline values, normal host overrides and generated exporter
|
|
rules. Canonical API endpoint checks cover all six cluster nodes.
|
|
- A reusable isolated VM check exercises allowed monitoring IPv4 access, denied other IPv4/IPv6
|
|
access to all three exporter ports, retained IPv6 loopback and a reachable ordinary IPv4/IPv6
|
|
service. Run it with `nix build .#checks.x86_64-linux.security-exporters`. It imports only the
|
|
shared firewall, not real-host secrets or workloads, and does not replace deployment-time checks.
|
|
|
|
### Read-only audit snapshot — 2026-10-03
|
|
|
|
- **API:** the canonical domain resolves to the VIP; authenticated `/readyz` returned `ok` with
|
|
normal TLS validation. No DNS or certificate changes were needed.
|
|
- **ai boot:** `sbctl status` reports Secure Boot enabled and Setup Mode disabled. It also reports
|
|
legacy sbctl configuration; migration is separate work, not permission to change firmware keys.
|
|
- **Listeners:** SSH listens on IPv4/IPv6 on sampled VM hosts and a master. SMART exporters bind the
|
|
physical hosts' LAN IPv4 addresses; node_exporter has wildcard listeners. youko's NFS listens on
|
|
IPv4/IPv6, and the sampled master API has a wildcard listener. These are listener observations,
|
|
not proof of reachability from every network or of router/WAN protection.
|
|
- **Workload inventory:** 36 of 78 observed pods had host networking, hostPath or explicitly
|
|
privileged containers. They were infrastructure components: Cilium, Istio CNI/ztunnel, NFS CSI,
|
|
kube-vip and OpenTelemetry log agents. Their host access is not automatically a misconfiguration;
|
|
remove it only after validating the component's requirements. Application pods were not flagged by
|
|
this limited check; that does not establish full workload isolation.
|
|
- **RBAC:** explicit cluster-admin bindings include `system:masters` and the Flux kustomize/Helm
|
|
controller service accounts. Operators also have broad resource permissions. Flux's bootstrap
|
|
privileges are intentional for cluster reconciliation; Git write access and controller credentials
|
|
therefore remain administration boundaries. This inventory is not a complete effective-access or
|
|
privilege-escalation analysis.
|
|
- **NFS:** the golden store is `0755 root:root`, exported read-write to an explicit client list (the
|
|
k3s nodes and the VM hosts) with `sec=sys,no_root_squash`. The client list stops other LAN devices
|
|
reaching the export, but a root user on an allowed client still has server-root write access.
|
|
`root_squash` is deferred: the existing PVC subdirectories are root-owned (observed
|
|
`drwxr-xr-x root root` and `drwxrwsr-x root 107`), so enabling it needs a one-time ownership
|
|
migration and a maintenance window. See docs/WORKAROUNDS.md.
|
|
- **Remote consoles:** the libvirt VNC consoles listen on `127.0.0.1` only, and the k3s firewall
|
|
drops 5900-5903 from pod sources. Reach a console through an SSH tunnel.
|
|
- **Pod-to-host firewall:** two independent layers deny pod access to the credential-bearing host
|
|
ports (SSH, NFS, Postgres, restic, VNC, Immich, etcd, exporter/monitoring). The
|
|
`deny-pod-to-node-admin-ports` `CiliumClusterwideNetworkPolicy` in
|
|
[k8s-gitops#52](https://github.com/ryan4yin/k8s-gitops/pull/52) denies pod egress to
|
|
`host`/`remote-node` on those ports at the pod's own node; the k3s nodes' nftables rule drops the
|
|
same ports for pod CIDR sources (`10.0.0.0/8`, `fd00::/104`). The nftables rule only matches when
|
|
the pod IP survives to the destination, which requires Cilium's eBPF datapath: kube-proxy's
|
|
iptables masquerade rewrote pod -> node traffic to the source node IP and defeated it. That is why
|
|
k3s runs `--disable-kube-proxy` ([#404](https://github.com/ryan4yin/nix-config/pull/404)) and
|
|
Cilium runs `kubeProxyReplacement` with `bpf.masquerade`. Verified live on all nodes: pod ->
|
|
node:22 is dropped while kubelet, ingress (80/443 stay open), ClusterIP and DNS keep working.
|
|
|
|
Only selected pod security fields, RBAC rules/binding metadata, listeners and export metadata were
|
|
queried. Secret contents and raw kubeconfigs were not inspected. No live access policy, workload,
|
|
export, credential, DNS record or firmware configuration was changed during the audit.
|
|
|
|
### Effective-access and credential follow-up — 2026-10-03
|
|
|
|
- Four authenticated `kubectl auth can-i` probes covered all 40 ServiceAccounts used by current
|
|
pods: reading secrets across namespaces, creating ClusterRoleBindings, escalating ClusterRoles,
|
|
and creating pods in the account's own namespace. Impersonation included the account's normal
|
|
service-account and authenticated groups; no secret contents or real pods/bindings were created.
|
|
- Flux Helm/Kustomize controllers and the VictoriaMetrics operator were allowed all four probes.
|
|
Kiali's operator could create ClusterRoleBindings, but the escalation probe was denied; create
|
|
permission alone does not prove unrestricted binding authority. Several controllers, including
|
|
Loki's account, can read secrets across namespaces and need a feature-specific scope review.
|
|
- Sampled staging application/default accounts were denied all four probes. This does not prove they
|
|
have no other permissions or cannot read particular secrets in their own namespace; unused
|
|
accounts, named-resource grants, token issuance and indirect escalation remain separate checks.
|
|
- Metadata checks on ai, reachable youko and the sampled master found owner-only decrypted files
|
|
(`0400`, `0500`, `0600` or `0000`), not group/world-readable files. ai's user-consumed
|
|
`/etc/agenix` copies have explicit owner-only modes. A symlink's `0777` is not the effective
|
|
target's access mode.
|
|
- User-readable credentials are intentional, including `nix-access-tokens` on servers as well as
|
|
desktops. Same-user agents are not isolated from these credentials by Unix file permissions. Mode
|
|
`0000` also does not stop privileged root. Other host/runtime ownership checks and recipient
|
|
membership were not completed; no private recipient repository or secret value was inspected.
|
|
- Darwin's `/etc/agenix` entries are agenix `path` symlinks, so agenix applies each secret's
|
|
declared owner and mode and the root-only `rclone.conf` stays root-owned. Runtime ownership
|
|
(`ls -lL /etc/agenix`) on each Darwin host still needs a check after the next activation.
|
|
- PostgreSQL's loopback TCP listener still trusts every role except `postgres` without a password,
|
|
so any local process can log in as `immich` or `playground`. No service configured here uses that
|
|
path: Immich and the exporter connect over the Unix socket with peer authentication. `postgres`
|
|
and replication now require `scram-sha-256`, closing the local-superuser path. Provisioning
|
|
per-role passwords for full loopback `scram-sha-256` is the follow-up; the disabled Grafana
|
|
`postgres-playground` datasource also waits on a `playground` password.
|
|
|
|
### Follow-up audit — 2026-10-22
|
|
|
|
- Baseline readback on live hosts: the ten unprivileged sysctl keys and the esp4/esp6/rxrpc modprobe
|
|
deny are in effect on all 13 reachable hosts; the #399/#400/#401 nft rules, NFS export scope and
|
|
pg_hba rules are live. `net.core.bpf_jit_harden` and ai's exact rule order need a root readback;
|
|
shoukei and akane were unreachable. k3s MicroVM guests have no system profile — compare
|
|
`/run/current-system` with `/run/booted-system` there.
|
|
- Exporter sockets are LAN-IPv4-bound on every scraped host except ai's NVIDIA exporter (`*:9835`),
|
|
a gap #373 missed; `hosts/idols-ai/hardware-nvidia.nix` now binds it and the `security-exporters`
|
|
eval covers every enabled exporter's `listenAddress`.
|
|
- Runtime drift on shoryu: the libvirt domains on 5902/5903 predate #400 and still bind wildcard;
|
|
only the nft VNC drop protects them until the guests are redefined and restarted.
|
|
- The pod-to-node deny lists omitted etcd's client port (2379, bound `192.168.5.114:2379`); both
|
|
layers now include it. 2381 (etcd metrics, scraped by in-cluster monitoring) stays reachable by
|
|
design.
|
|
- ServiceAccount effective access is now measured for all 41 live ns/SA pairs (on-node probes): the
|
|
Kiali operator gained a self-escalation path (`update`/`patch clusterroles` on a role bound to
|
|
itself, no `bind`/`escalate` verb needed); Kiali and kube-state-metrics deny `get secrets -A` but
|
|
allow `list`/`watch`, so probe sets must pair both verbs; 16 accounts enumerate secrets
|
|
cluster-wide and cert-manager can delete them. Narrowing proposals target k8s-gitops.
|
|
- k3s-test-1 pod networking has been down since the 2026-10-14 full guest cold-restart: Cilium
|
|
bootstraps through the VIP while the VIP requires kube-vip's lease over ClusterIP — a cycle the
|
|
kube-proxy-replacement switch created. The host nftables drops stay loaded, but the CLiCNP layer
|
|
enforces nothing while Cilium is down and the nft rule's pod-source-IP precondition is not met, so
|
|
the pod-to-node boundary is degraded until the cluster recovers.
|
|
- The cluster recovered manually (VIP attached on one master, then handed back to kube-vip), and the
|
|
cycle itself is fixed: kube-vip's pod now overrides the kubelet-injected
|
|
`KUBERNETES_SERVICE_HOST/PORT` to reach its own node's apiserver (port 6443) instead of the
|
|
ClusterIP, so the VIP no longer needs Cilium on a cold start. A full cold start of all six guests
|
|
on 2026-10-24 brought the VIP back with no manual action (6/6 nodes, Cilium 6/6, kube-vip 3/3, no
|
|
connection to the ClusterIP), and k8s-gitops#54 is merged and reconciled. A probe from a pod
|
|
confirms 2379 is denied while 2381 stays open for in-cluster metric scraping.
|
|
|
|
### Sandbox and agent boundaries
|
|
|
|
The installed agent CLIs run as the normal user, not inside a repository-defined OS sandbox.
|
|
AGENTS.md/tool permissions remain important execution policy, but they are not Unix process
|
|
isolation. Source review and socket metadata—not reads of private credentials—established:
|
|
|
|
- The current desktop session's SSH-agent socket is owned by and writable to the current UID.
|
|
Same-user subprocesses can potentially use its signing interface; read-only mounts do not prevent
|
|
socket use. Private-key bytes were not inspected and no signing operation was requested.
|
|
- Agent forwarding is on for `192.168.*` destinations and the `hostsAddr` aliases (servers use it
|
|
for pam_rssh sudo), not for GitHub or other hosts. Do not remove it without checking
|
|
deployment/build workflows.
|
|
- All nine desktop wrappers use shared nixpak policies: cleared environment, filtered session D-Bus
|
|
and selected mounts. The [permission inventory](../hardening/README.md#app-permissions) lists each
|
|
app's file, network, credential and media grants.
|
|
- Only Firefox/Chromium share host networking. Chromium is an intentional local-development
|
|
exception; all other apps isolate loopback with pasta but can still reach the LAN/Internet.
|
|
Browser media and browserpass are enabled only for Firefox/Chromium/Chrome. Brave Origin and Edge
|
|
are backup browsers with no media grants. browserpass exposes password-store/GPG data and agent
|
|
operations.
|
|
- Firefox/Chromium/Chrome receive only a security key's FIDO interface (mechanism and limits in the
|
|
[hardening README](../hardening/README.md#limits)). A compromised browser can use that interface;
|
|
the key's PIN and touch policy remain the authentication boundary.
|
|
- Nixpak supplements Chromium's renderer sandbox, which remains enabled. Zoom/WeChat retain their
|
|
FHS runtimes inside the outer boundary; inner mounts cannot recover hidden host resources.
|
|
- Desktop access has limits: GPU drivers remain exposed, PulseAudio permits recording, read-only
|
|
sockets permit agent operations, and Zoom/WeChat/QQ's X11 connections have shared-display trust.
|
|
Zoom's 7.2.1 client receives only `/dev/video0` and `/dev/video1` because it captures the webcam
|
|
directly via V4L2; no other app gets raw camera nodes and no raw PipeWire access is added.
|
|
Chromium-family browsers use only the app-scoped Secret portal for v12 cookie encryption; no
|
|
browser receives broad GNOME Keyring authority. Firefox uses its own NSS storage. Firefox camera
|
|
and remote screen sharing are intentionally disabled: Firefox does not support the XDG
|
|
Camera/ScreenCast portal path used by this policy, and enabling those features would require
|
|
granting raw video-device or direct PipeWire access. We deliberately keep that privilege closed.
|
|
- **Rollout note (WA-045):** v12 is verified on a disposable profile, but if the portal is
|
|
unavailable Chromium may fall back to weaker v10 storage; portal initialization is monitored.
|
|
- Policy/build passes for all nine wrappers. Interactive validation passed for Firefox, Chrome,
|
|
Chromium, Brave Origin, Edge, QQ, Telegram and Zoom; the Chromium smoke test also validates
|
|
renderer sandboxing, environment clearing, bind scope and profile persistence. WeChat's
|
|
interactive Wayland check remains pending. See the
|
|
[runtime validation](../hardening/tests/runtime-validation.md).
|
|
|
|
#### Agent-first diagnostics by host role
|
|
|
|
- **Homelab:** configured root SSH is an acceptable path for autonomous, task-scoped read-only
|
|
diagnosis. Root access does not authorize deployment, restart, permission changes, destructive
|
|
operations or secret inspection merely because a diagnostic command is available.
|
|
- **Core desktops:** prefer normal-user service status, network information, metrics, own-process
|
|
inspection and journal access. Do not add passwordless sudo or remove existing human-admin
|
|
membership/workflows as an incidental security change. ai's normal user can query system-journal
|
|
metadata, including the kernel transport, without sudo; other desktops need their own readback.
|
|
Escalation to root, including local root SSH, needs explicit authorization rather than an
|
|
automatic fallback when a normal-user diagnostic is denied.
|
|
- A read-only API or log can still reveal credentials. Use bounded service/time filters and redact
|
|
sensitive output; never dump process environments or raw kubeconfigs. Narrow, reviewed per-service
|
|
log access is preferable when a desktop lacks access. Do not blanket-grant all journals or add a
|
|
generic privileged diagnostic-command dispatcher just for convenience.
|
|
- Preserve functioning SSH/browser/GPU integration. Stronger agent OS isolation, if needed, should
|
|
be an explicit workflow design with positive/negative tests, not a collection of surprise denials.
|
|
|
|
For routine diagnosis, prefer bounded reads such as `systemctl --failed --no-pager`,
|
|
`systemctl status SERVICE --no-pager`, `journalctl -u SERVICE -n 200 --no-pager`,
|
|
`journalctl --user -u SERVICE -n 200 --no-pager`, `ip -br address`, `ss -lntu` and
|
|
`ps -eo pid,comm,stat`. On Homelab these may use the configured root SSH connection; on desktops try
|
|
normal-user access first. Do not confuse read-only status with permission to restart a service, or
|
|
diagnose a log-access denial by reading credential files or removing protection globally.
|
|
|
|
### Update/reboot routine and kernel visibility
|
|
|
|
Recommended cadence: review advisories and pinned inputs weekly, with an out-of-band review for
|
|
active exploitation or a reachable high-impact vulnerability. Check affected versions and exposure,
|
|
then validate an update and schedule the required rollout/reboot. Do not auto-apply every input bump
|
|
or promise that a kernel version string alone proves security coverage.
|
|
|
|
Verify the booted kernel after each rollout instead of trusting the selected generation. The booted
|
|
image and the image selected for the next boot are independent:
|
|
|
|
- `readlink -f /run/booted-system/kernel` — kernel of the running system closure.
|
|
- `readlink -f /nix/var/nix/profiles/system/kernel` — kernel selected for the next boot.
|
|
|
|
A mismatch means the host still runs the old kernel and a reboot is pending. This catches boot-only
|
|
deployments even when `/run/current-system` is unchanged; `uname -r` gives the running release
|
|
string. A matching kernel path is not proof of vulnerability age, livepatch state or driver
|
|
compatibility, and a generation change that only affects the initrd, modules or userspace leaves the
|
|
kernel path unchanged. Guest profiles and a runner staged only on a MicroVM's physical host require
|
|
separate verification.
|
|
|
|
## Remaining prioritized TODOs
|
|
|
|
| Priority | Work | Acceptance / constraints |
|
|
| -------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| P1 | Move root/deploy SSH to hardware-backed keys | `sk-ssh-ed25519` keys with touch for root, with SSH multiplexing to keep Colmena usable; an agent lifetime or confirmation; a tested break-glass path |
|
|
| P1 | Alert on SSH logins and ship auth/audit logs off-host | Off-host copy host root cannot rewrite; alert on an unknown key or source, a session-rate spike and `root-session`/`credential-access`; redact key material |
|
|
| P1 | Roll out and verify the implemented baseline/exporter rules | Runtime sysctl readback, allowed IPv4 scrape and denied IPv6/non-monitoring scrape tests; preserve Nix/Podman/Cilium/NFS functionality |
|
|
| P1 | Review router mappings and tailnet ACLs; narrow service exposure where warranted | Audit was limited to sampled host listeners; test intended and denied access without locking out deployment |
|
|
| P1 | Review NFS client scope and root identity requirements | Restrict to required VM hosts/CSI nodes where practical; test provisioning, permissions and existing PVCs before changing squash semantics |
|
|
| P1 | Narrow operator/secret access where justified; finish permission checks | Selected effective probes and ai/youko modes audited; validate broad secret-reader needs, remaining hosts and recipient scope; Darwin per-file ownership is now explicit and needs a runtime check |
|
|
| P1 | Adopt the documented cadence and verify the booted kernel after rollouts | Routine documented; compare booted and selected kernel paths and recheck API/workloads after reboot; no automated metric is deployed |
|
|
| P1 | Exercise backup restore and credential rotation | Use an approved isolated restore target and recovery access; never overwrite live data for a drill |
|
|
| P2 | Promote selected AppArmor profiles to enforce; harden exposed systemd services | Per-app positive/negative tests, store-path coverage, reviewed capabilities and reversible rollout |
|
|
| P2 | Maintain ai boot integrity and assess other host roles | Review sbctl configuration migration, recovery boot and signed custom/NVIDIA modules; ai Secure Boot state already verified |
|
|
| P2 | Review unused-module deny list against real workloads and verified advisories | Check loaded/built-in modules and autoload dependencies; do not block required filesystems, WiFi, VM or Cilium/NFS functionality |
|
|
| P2 | Evaluate stronger controls only against a concrete threat | Account for hibernation, crash recovery, applications and performance; no blanket io_uring/SMT/userns ban or hardened-kernel switch |
|
|
| P2 | Validate narrower sandbox/agent boundaries where needed | Source and selected SSH/journal metadata audited; use disposable data/socket probes and preserve core-desktop diagnosis, browserpass and Homelab root workflows |
|
|
| P3 | Improve security-event alerts and configuration drift checks | Actionable redacted alerts and periodic runtime control readback |
|
|
|
|
Record temporary exceptions in [WORKAROUNDS.md](./WORKAROUNDS.md) with a removal condition. Revisit
|
|
this architecture after a new exposed service, network/storage change, host addition or significant
|
|
kernel/nixpkgs update. No control here guarantees containment after host root compromise.
|
|
|
|
## References
|
|
|
|
- [Linux kernel sysctls](https://docs.kernel.org/admin-guide/sysctl/kernel.html)
|
|
- [Filesystem protections](https://docs.kernel.org/admin-guide/sysctl/fs.html)
|
|
- [BPF JIT hardening](https://docs.kernel.org/admin-guide/sysctl/net.html#bpf-jit-harden)
|
|
- [userfaultfd restrictions](https://docs.kernel.org/admin-guide/sysctl/vm.html#unprivileged-userfaultfd)
|
|
- [Yama](https://docs.kernel.org/admin-guide/LSM/Yama.html)
|
|
- [Kernel module signing](https://docs.kernel.org/admin-guide/module-signing.html)
|
|
- [NixOS security advisories](https://github.com/NixOS/nixpkgs/security/advisories)
|
|
- [CISA Known Exploited Vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
|