mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:46 +01:00
Merge pull request #489 from ryan4yin/harden-incident-110
feat(security): harden homelab SSH, service sandboxing and audit trail
This commit is contained in:
33 files changed
+670
-20
No files matched your search
+30
-4
@@ -34,8 +34,8 @@ small, reversible controls and narrow exceptions; benchmark before adopting expe
|
||||
## Architecture and trust boundaries
|
||||
|
||||
| Boundary | Current design | Limitation |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Administration | Key-based SSH; root key login retained for Colmena/MicroVM deployment | Deployment credentials are root-equivalent; password login being disabled does not restrict stolen keys |
|
||||
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Administration | Key-based SSH; root key login retained for Colmena/MicroVM deployment; only keys declared in Nix are honoured, and no password or keyboard-interactive login; remote-builder host keys only reach `nix-daemon` | Deployment credentials are root-equivalent file keys, usable by any same-user process through the desktop SSH agent; one stolen key reaches every host |
|
||||
| Host network | Shared nftables firewall trusts the home LAN, tailnet interface and local Podman bridge; IPv6 link-local is accepted | These are broad trust zones, not service-level least privilege; router exposure and tailnet ACLs require separate verification |
|
||||
| Monitoring | node/smartctl exporters bind the host LAN IPv4 (9100/9633); firewall rules restrict exporter ports 9100, 9835 and 9633 to the monitoring host over IPv4 | nvidia_gpu_exporter (9835) still binds wildcard; the IPv6 denial for those three ports is in the shared firewall; current running hosts require separate verification |
|
||||
| VM hosts | shoryu, shushou and youko run guests behind `br0`; ruby/kana are libvirt guests | A bridge is not network isolation; host compromise affects its guests |
|
||||
@@ -104,11 +104,35 @@ an advisory and affected kernel versions before adding CVE claims; search snippe
|
||||
- AppArmor is enabled, but the only explicit policy, sudo's, is in complain mode. Complain mode logs
|
||||
rather than blocks. Existing FHS-oriented abstractions need Nix store path review; this is not
|
||||
broad enforced application confinement.
|
||||
- Systemd service sandboxing is per-service, not a global policy. Review capabilities, writable
|
||||
paths, network access and syscall requirements before tightening an individual service.
|
||||
- Systemd service sandboxing is per-service, not a global policy. The exposed services nixpkgs
|
||||
leaves weakly sandboxed (caddy, rustfs, vmalert, whisper-asr) use the preset in
|
||||
[systemdHardening.nix](lib/systemdHardening.nix). Review capabilities, writable paths, network
|
||||
access and syscall requirements before tightening an individual service.
|
||||
|
||||
See [application hardening](hardening/README.md) for wrapper details.
|
||||
|
||||
### Audit trail
|
||||
|
||||
`modules.security.audit` ([auditd.nix](modules/nixos/base/auditd.nix)) runs auditd on youko, shoryu,
|
||||
shushou and aquamarine. It records, for login sessions only (SSH, and sudo or su from one): every
|
||||
root command (`root-session`), access to decrypted secrets and SSH host keys (`credential-access`),
|
||||
writes to SSH, account/PAM, Nix trust and audit configuration, module loads, and denials.
|
||||
|
||||
- Services are not audited: a `never,task` rule gives tasks without a login uid no audit context, so
|
||||
virtiofsd, hypervisors, NFS, PostgreSQL and Nix builds skip the per-syscall audit path. A
|
||||
compromised service is therefore invisible until it opens a login session.
|
||||
- Logs stay on the host and root can erase them; shipping them off-host is a P1 TODO.
|
||||
- One watch on a missing path makes `auditctl -R` stop and leaves auditing off;
|
||||
`nix build .#checks.x86_64-linux.security-auditd` loads the rules in a VM.
|
||||
|
||||
Query as root; sshd is built without libaudit, so use PAM's `USER_START` records for logins:
|
||||
|
||||
```sh
|
||||
ausearch -i -m USER_START --start today # logins with address and session id
|
||||
ausearch -i -k root-session --start today # root commands, with auid and ses
|
||||
ausearch -i --session <ses> # everything one session did
|
||||
```
|
||||
|
||||
## Updating, deploying and recovering
|
||||
|
||||
1. Check the advisory and affected package/kernel; prioritize active exploitation and reachable
|
||||
@@ -355,6 +379,8 @@ separate verification.
|
||||
|
||||
| Priority | Work | Acceptance / constraints |
|
||||
| -------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| P1 | Move root/deploy SSH to hardware-backed keys | `sk-ssh-ed25519` keys with touch for root, with SSH multiplexing to keep Colmena usable; an agent lifetime or confirmation; a tested break-glass path |
|
||||
| P1 | Alert on SSH logins and ship auth/audit logs off-host | Off-host copy host root cannot rewrite; alert on an unknown key or source, a session-rate spike and `root-session`/`credential-access`; redact key material |
|
||||
| P1 | Roll out and verify the implemented baseline/exporter rules | Runtime sysctl readback, allowed IPv4 scrape and denied IPv6/non-monitoring scrape tests; preserve Nix/Podman/Cilium/NFS functionality |
|
||||
| P1 | Review router mappings and tailnet ACLs; narrow service exposure where warranted | Audit was limited to sampled host listeners; test intended and denied access without locking out deployment |
|
||||
| P1 | Review NFS client scope and root identity requirements | Restrict to required VM hosts/CSI nodes where practical; test provisioning, permissions and existing PVCs before changing squash semantics |
|
||||
|
||||
+2
-1
@@ -83,9 +83,10 @@ A known limitation with no fix yet.
|
||||
An unfinished feature or gap.
|
||||
|
||||
| ID | What & where | Why | Removal condition | Added | Revisit | Status |
|
||||
| ------ | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ---------- | ----------------------- | ------ |
|
||||
| ------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ---------- | ------------------------ | ------ |
|
||||
| WA-013 | NixOS tests — `outputs/README.md` | full-host tests still rebuild custom packages (no private cache) and need the host ssh key to decrypt agenix; the shared-firewall security-exporters VM test now runs in CI (KVM); `outputs/x86_64-linux/nixos-tests/idols-ruby.nix` is exposed as `packages.x86_64-linux.ruby-nixos-tests` but is not in `checks` or CI | Resolve when a private cache exists and tests can run per-service | 2024-03-13 | quarterly | active |
|
||||
| WA-027 | NFS golden store `no_root_squash` — `hosts/12kingdoms-youko/nfs-golden.nix` | the existing PVC subdirectories are root-owned, so enabling `root_squash` would break running workloads until they are migrated; the CSI node plugin is hostNetwork and sources from the allowlisted node IPs | Drop when the PVC subdirectories are migrated and `root_squash` is enabled | 2026-10-07 | next maintenance window | active |
|
||||
| WA-044 | transmission AppArmor lock permission — `hosts/12kingdoms-youko/homelab-services/transmission.nix` | nixpkgs' transmission 4.1.4 profile grants `rw` on the daemon's config dir but not `k`, while 4.1.4 takes an fcntl lock on `~/.config/transmission-daemon/lock`; the daemon exits with "Another process is already using". A `local/bin.transmission-daemon` include adds `rwk` for that lock file only | Drop once the nixpkgs profile includes the lock permission | 2026-10-31 | on the next nixpkgs bump | active |
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ in
|
||||
];
|
||||
|
||||
modules.btrbk.enable = true;
|
||||
modules.security.audit.enable = true;
|
||||
|
||||
# MoreFine S500Plus: Radeon iGPU. The ROCm build of btop shows the GPU panel;
|
||||
# node_exporter's drm collector exposes its utilization/VRAM.
|
||||
|
||||
@@ -25,7 +25,15 @@ in
|
||||
preservation.preserveAt."/persistent" = {
|
||||
directories = [
|
||||
"/etc/NetworkManager/system-connections"
|
||||
"/etc/ssh"
|
||||
# Mount in initrd: NixOS copies mode-carrying /etc entries (such as
|
||||
# authorized_keys.d) during stage-2 activation, which runs before systemd
|
||||
# starts. A mount created by preservation.target comes later and would
|
||||
# hide those copies behind the previous `switch`, so key changes only
|
||||
# applied on `switch`, never on `boot` + reboot.
|
||||
{
|
||||
directory = "/etc/ssh";
|
||||
inInitrd = true;
|
||||
}
|
||||
"/etc/nix/inputs"
|
||||
"/etc/secureboot" # lanzaboote - secure boot
|
||||
# my secrets
|
||||
|
||||
@@ -24,6 +24,7 @@ in
|
||||
];
|
||||
|
||||
modules.btrbk.enable = true;
|
||||
modules.security.audit.enable = true;
|
||||
|
||||
# MinisForum UM560: Radeon iGPU. The ROCm build of btop shows the GPU panel;
|
||||
# node_exporter's drm collector exposes its utilization/VRAM.
|
||||
|
||||
@@ -58,6 +58,7 @@ in
|
||||
};
|
||||
|
||||
modules.btrbk.enable = true;
|
||||
modules.security.audit.enable = true;
|
||||
|
||||
# HDD tuning/diagnostics for the two spinning disks behind the USB-SATA bridge.
|
||||
environment.systemPackages = [ pkgs.hdparm ];
|
||||
|
||||
@@ -232,11 +232,22 @@ in
|
||||
};
|
||||
};
|
||||
|
||||
# caddy loads the lego-issued files at startup, so wait for the acme service;
|
||||
# otherwise its config fails to load before the cert exists.
|
||||
systemd.services.caddy = {
|
||||
# caddy loads the lego-issued files at startup, so wait for the acme
|
||||
# service; otherwise its config fails to load before the cert exists.
|
||||
wants = [ "acme-writefor.fun.service" ];
|
||||
after = [ "acme-writefor.fun.service" ];
|
||||
|
||||
# Sandbox the LAN front door; it writes only its State/LogsDirectory.
|
||||
serviceConfig = mylib.systemdHardening.mkDefaults mylib.systemdHardening.service // {
|
||||
# Only 80/443. No CAP_NET_ADMIN: the module already raises the UDP buffer
|
||||
# sysctls quic-go would force. "" resets the package unit's list.
|
||||
AmbientCapabilities = [
|
||||
""
|
||||
"CAP_NET_BIND_SERVICE"
|
||||
];
|
||||
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
|
||||
};
|
||||
};
|
||||
|
||||
# caddy is reached from the LAN/tailnet only; the shared firewall
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
{ pkgs, ... }:
|
||||
{
|
||||
# Git over SSH reads Gitea's own ~/.ssh/authorized_keys (keys pinned to
|
||||
# `gitea serv`), which modules/nixos/base/ssh.nix otherwise ignores.
|
||||
services.openssh.extraConfig = ''
|
||||
Match User gitea
|
||||
AuthorizedKeysFile %h/.ssh/authorized_keys /etc/ssh/authorized_keys.d/%u
|
||||
'';
|
||||
|
||||
# https://github.com/NixOS/nixpkgs/blob/nixos-26.05/nixos/modules/services/misc/gitea.nix
|
||||
services.gitea = {
|
||||
enable = true;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{ config, ... }:
|
||||
{ config, mylib, ... }:
|
||||
{
|
||||
# https://docs.victoriametrics.com/victoriametrics/vmalert/
|
||||
services.vmalert.instances."homelab" = {
|
||||
@@ -34,6 +34,9 @@
|
||||
};
|
||||
};
|
||||
|
||||
# nixpkgs only sets DynamicUser; vmalert talks to loopback and keeps no state.
|
||||
systemd.services.vmalert-homelab.serviceConfig = mylib.systemdHardening.mkDefaults mylib.systemdHardening.service;
|
||||
|
||||
services.prometheus.alertmanager = {
|
||||
enable = true;
|
||||
listenAddress = "127.0.0.1";
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
{
|
||||
config,
|
||||
mylib,
|
||||
...
|
||||
}:
|
||||
let
|
||||
inherit (mylib.systemdHardening) mkDefaults service;
|
||||
in
|
||||
{
|
||||
# The service module comes from the default nixpkgs (services.rustfs); it
|
||||
# creates the rustfs user/group, the data volume via tmpfiles, and a hardened
|
||||
@@ -26,6 +30,18 @@
|
||||
environmentFile = config.age.secrets."rustfs.env".path;
|
||||
};
|
||||
|
||||
systemd.services.rustfs = {
|
||||
# The data volume lives on the encrypted HDD subvolume; wait for its mount.
|
||||
systemd.services.rustfs.unitConfig.RequiresMountsFor = "/data/apps";
|
||||
unitConfig.RequiresMountsFor = "/data/apps";
|
||||
|
||||
# Fill the sandboxing the nixpkgs unit leaves out.
|
||||
serviceConfig = mkDefaults service // {
|
||||
# ProtectSystem=strict: the data volume is the only writable path.
|
||||
ReadWritePaths = [ config.services.rustfs.settings.RUSTFS_VOLUMES ];
|
||||
# Its startup endpoint list enumerates the host's interface addresses.
|
||||
RestrictAddressFamilies = service.RestrictAddressFamilies ++ [ "AF_NETLINK" ];
|
||||
# Capacity and observability metrics read host-wide /proc files.
|
||||
ProcSubset = "all";
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -57,6 +57,13 @@ let
|
||||
'';
|
||||
in
|
||||
{
|
||||
# transmission 4.1.4 takes an fcntl lock on its config-dir lock file, but the
|
||||
# AppArmor profile shipped by nixpkgs grants rw without k (lock), so the
|
||||
# daemon exits with "Another process is already using". See WORKAROUNDS.md.
|
||||
security.apparmor.includes."local/bin.transmission-daemon" = ''
|
||||
/var/lib/transmission/.config/transmission-daemon/lock rwk,
|
||||
'';
|
||||
|
||||
# Join the shared fileshare group so transmission can read/write files
|
||||
# created by sftpgo, and vice versa (via setgid directories).
|
||||
users.users.${name}.extraGroups = [ "fileshare" ];
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
lib,
|
||||
mylib,
|
||||
myvars,
|
||||
pkgs,
|
||||
pkgs-npu,
|
||||
@@ -86,23 +87,26 @@ in
|
||||
ZE_ENABLE_ALT_DRIVERS = "${lib.getLib pkgs-npu.intel-npu-driver}/lib/libze_intel_npu.so";
|
||||
};
|
||||
|
||||
serviceConfig = {
|
||||
# Reads its model, writes its cache, uses the NPU and listens on the LAN.
|
||||
serviceConfig = mylib.systemdHardening.service // {
|
||||
ExecStart = "${pythonEnv}/bin/python ${./asr/server.py}";
|
||||
CacheDirectory = "whisper-asr";
|
||||
DynamicUser = true;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 5;
|
||||
|
||||
# The service only reads its model, writes its cache and listens on the
|
||||
# LAN; it needs no other part of the filesystem or any privilege.
|
||||
NoNewPrivileges = true;
|
||||
PrivateTmp = true;
|
||||
ProtectHome = true;
|
||||
ProtectSystem = "strict";
|
||||
RestrictAddressFamilies = [
|
||||
"AF_INET"
|
||||
"AF_UNIX"
|
||||
];
|
||||
# The NPU is /dev/accel/accel0; allow that device class only.
|
||||
PrivateDevices = false;
|
||||
DevicePolicy = "closed";
|
||||
DeviceAllow = [ "char-accel rw" ];
|
||||
# OpenVINO's oneDNN kernels and libffi closures generate code at runtime.
|
||||
MemoryDenyWriteExecute = false;
|
||||
# OpenVINO sizes its thread pools from /proc/cpuinfo and /proc/meminfo.
|
||||
ProcSubset = "all";
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -58,6 +58,7 @@ in
|
||||
# Local btrfs snapshots of @persistent (SSH host keys, /etc/secureboot, the
|
||||
# NixOS state). Same layout as the other physical hosts; no off-host target.
|
||||
modules.btrbk.enable = true;
|
||||
modules.security.audit.enable = true;
|
||||
|
||||
# smartd autodetects devices and logs to the journal; the smartctl exporter
|
||||
# (modules/nixos/base/monitoring.nix) is what feeds wear and media errors to
|
||||
@@ -81,7 +82,12 @@ in
|
||||
environment.sessionVariables.ZE_ENABLE_ALT_DRIVERS = "${lib.getLib pkgs-npu.intel-npu-driver}/lib/libze_intel_npu.so";
|
||||
|
||||
# The daemons of the hosts that build here (modules/nixos/base/remote-builders.nix).
|
||||
users.users.root.openssh.authorizedKeys.keys = myvars.remoteBuilderClientKeys;
|
||||
# Their host keys have no passphrase, so they only get the daemon protocol, as
|
||||
# nix.sshServe does. `--store daemon` hands builds to the system daemon instead
|
||||
# of running them inside this audited login session (modules/nixos/base/auditd.nix).
|
||||
users.users.root.openssh.authorizedKeys.keys = map (
|
||||
key: ''restrict,command="nix-daemon --stdio --store daemon" ${key}''
|
||||
) myvars.remoteBuilderClientKeys;
|
||||
|
||||
# systemd-journal - cap disk usage
|
||||
# https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html
|
||||
|
||||
@@ -24,8 +24,17 @@ in
|
||||
# If the directory/file already exists in the root filesystem you should
|
||||
# move those files/directories to /persistent first!
|
||||
preservation.preserveAt."/persistent" = {
|
||||
# Mount in initrd: NixOS copies mode-carrying /etc entries (such as
|
||||
# authorized_keys.d) during stage-2 activation, which runs before systemd
|
||||
# starts. A mount created by preservation.target comes later and would hide
|
||||
# those copies behind the previous `switch`, so key changes only applied on
|
||||
# `switch`, never on `boot` + reboot. Preparing the mount in initrd puts
|
||||
# /persistent/etc/ssh at /sysroot/etc/ssh before activation runs.
|
||||
directories = [
|
||||
"/etc/ssh"
|
||||
{
|
||||
directory = "/etc/ssh";
|
||||
inInitrd = true;
|
||||
}
|
||||
"/etc/nix/inputs"
|
||||
"/etc/secureboot" # lanzaboote - secure boot (see ./secureboot.nix)
|
||||
|
||||
|
||||
+5
-1
@@ -21,9 +21,13 @@ reduce code duplication and make it easier to add new machines.
|
||||
7. **`genMicrovmGuestModule.nix`** - NixOS microVM guest configuration generator
|
||||
8. **`genLibvirtDomainXml.nix`** - libvirt domain XML generator (imported directly by host configs)
|
||||
|
||||
### Shared Presets
|
||||
|
||||
9. **`systemdHardening.nix`** - systemd sandboxing preset for exposed services
|
||||
|
||||
### Entry Point
|
||||
|
||||
9. **`default.nix`** - Main entry point that imports the generators and exports them as a single
|
||||
10. **`default.nix`** - Main entry point that imports the generators and exports them as a single
|
||||
attribute set, plus the `relativeToRoot` and `scanPaths` helpers
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
genVmHostModule = import ./genVmHostModule.nix;
|
||||
genMicrovmGuestModule = import ./genMicrovmGuestModule.nix;
|
||||
|
||||
systemdHardening = import ./systemdHardening.nix { inherit lib; };
|
||||
|
||||
# use path relative to the root of the project
|
||||
relativeToRoot = lib.path.append ../.;
|
||||
scanPaths =
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Shared systemd sandboxing preset, adapted from xddxdd's serviceHardenArgs
|
||||
# (github.com/xddxdd/nixos-config, nixos/minimal-modules/harden.nix).
|
||||
{ lib }:
|
||||
rec {
|
||||
# For a network daemon that runs as an unprivileged user and only needs its
|
||||
# own state; override keys where a service needs more.
|
||||
service = {
|
||||
AmbientCapabilities = "";
|
||||
CapabilityBoundingSet = "";
|
||||
LockPersonality = true;
|
||||
MemoryDenyWriteExecute = true;
|
||||
NoNewPrivileges = true;
|
||||
PrivateDevices = true;
|
||||
PrivateMounts = true;
|
||||
PrivateTmp = true;
|
||||
ProcSubset = "pid";
|
||||
ProtectClock = true;
|
||||
ProtectControlGroups = true;
|
||||
ProtectHome = true;
|
||||
ProtectHostname = true;
|
||||
ProtectKernelLogs = true;
|
||||
ProtectKernelModules = true;
|
||||
ProtectKernelTunables = true;
|
||||
ProtectProc = "invisible";
|
||||
ProtectSystem = "strict";
|
||||
RemoveIPC = true;
|
||||
RestrictAddressFamilies = [
|
||||
"AF_UNIX"
|
||||
"AF_INET"
|
||||
"AF_INET6"
|
||||
];
|
||||
RestrictNamespaces = true;
|
||||
RestrictRealtime = true;
|
||||
RestrictSUIDSGID = true;
|
||||
SystemCallArchitectures = "native";
|
||||
# Fail a denied syscall rather than kill the service while it is new to
|
||||
# the preset; switch to "kill" once it has run stably.
|
||||
SystemCallErrorNumber = "EPERM";
|
||||
SystemCallFilter = [
|
||||
"@system-service"
|
||||
"~@clock @cpu-emulation @debug @module @mount @obsolete @privileged @raw-io @reboot @swap"
|
||||
];
|
||||
};
|
||||
|
||||
# Apply a preset at mkOptionDefault priority, so nixpkgs module values win.
|
||||
mkDefaults = lib.mapAttrs (_: lib.mkOptionDefault);
|
||||
}
|
||||
@@ -27,6 +27,7 @@ modules/
|
||||
│ └── users.nix # macOS user management
|
||||
└── nixos/ # NixOS-specific modules
|
||||
├── base/ # Base NixOS configuration
|
||||
│ ├── auditd.nix # Audit trail for root sessions and credentials
|
||||
│ ├── btop.nix # System resource monitor
|
||||
│ ├── btrbk.nix # Local btrfs snapshots
|
||||
│ ├── core.nix # Core system settings
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
{
|
||||
config,
|
||||
lib,
|
||||
options,
|
||||
pkgs,
|
||||
...
|
||||
}:
|
||||
let
|
||||
cfg = config.modules.security.audit;
|
||||
inherit (config.services) openssh;
|
||||
|
||||
# Audit trail for login sessions, so that a stolen deployment key leaves a
|
||||
# record of what each session did; see SECURITY.md ("Audit trail").
|
||||
# `auditctl -R` aborts at the first watch whose path is missing, leaving
|
||||
# auditing off, so every watched path must exist when the rules load.
|
||||
|
||||
# Secrets are symlinks into a per-deploy generation under the agenix ramfs,
|
||||
# which a per-file watch would lose on the next deploy; watch the ramfs tree.
|
||||
agenixSecrets = lib.optionals (options ? age) (lib.attrValues config.age.secrets);
|
||||
hasAgenix = agenixSecrets != [ ];
|
||||
credentialDirs = lib.optional hasAgenix config.age.secretsMountPoint;
|
||||
credentialFiles = lib.unique (
|
||||
map (secret: secret.path) (lib.filter (secret: !secret.symlink) agenixSecrets)
|
||||
# The host key is also agenix's identity: reading it decrypts every secret.
|
||||
++ lib.optionals openssh.enable (map (key: key.path) openssh.hostKeys)
|
||||
++ lib.optionals hasAgenix config.age.identityPaths
|
||||
);
|
||||
|
||||
identityFiles = map (name: "/etc/${name}") [
|
||||
"passwd"
|
||||
"shadow"
|
||||
"group"
|
||||
"gshadow"
|
||||
"sudoers"
|
||||
];
|
||||
|
||||
watchedDirs = credentialDirs ++ [
|
||||
"/etc/ssh"
|
||||
"/etc/pam.d"
|
||||
"/etc/nix"
|
||||
"/etc/audit"
|
||||
"/var/log/audit"
|
||||
];
|
||||
watchedFiles = credentialFiles ++ identityFiles ++ [ "/nix/var/nix/profiles/system" ];
|
||||
|
||||
watchFile =
|
||||
key: perm: path:
|
||||
"-a always,exit -F path=${path} -F perm=${perm} -k ${key}";
|
||||
watchDir =
|
||||
key: perm: dir:
|
||||
"-a always,exit -F dir=${dir} -F perm=${perm} -k ${key}";
|
||||
# A b64-only syscall rule misses 32-bit binaries.
|
||||
arches = [ "b64" ] ++ lib.optional pkgs.stdenv.hostPlatform.isx86_64 "b32";
|
||||
perArch = rule: map (arch: "-a always,exit -F arch=${arch} ${rule}") arches;
|
||||
in
|
||||
{
|
||||
options.modules.security.audit.enable = lib.mkEnableOption ''
|
||||
the Linux audit trail (auditd) for login sessions: every command run as
|
||||
root, access to decrypted secrets and SSH host keys, and changes to SSH,
|
||||
account, PAM, Nix trust and audit configuration. Logs stay in
|
||||
/var/log/audit on the host
|
||||
'';
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
assertions = [
|
||||
{
|
||||
# auditctl -R splits fields on whitespace; it does not shell-unquote paths.
|
||||
assertion = lib.all (path: builtins.match ".*[[:space:]].*" path == null) (
|
||||
watchedDirs ++ watchedFiles
|
||||
);
|
||||
message = "modules.security.audit: audited paths cannot contain whitespace in the native auditctl rule file.";
|
||||
}
|
||||
];
|
||||
|
||||
security.auditd.enable = true;
|
||||
security.audit = {
|
||||
enable = true;
|
||||
backlogLimit = lib.mkDefault 8192;
|
||||
# The first matching exit rule sets the key: specific watches go first.
|
||||
rules = lib.mkBefore (
|
||||
# Evaluated at fork: tasks without a login uid (everything systemd
|
||||
# starts) get no audit context, so no per-syscall cost.
|
||||
[ "-a never,task -F auid=unset" ]
|
||||
++ map (watchFile "credential-access" "rwa") credentialFiles
|
||||
++ map (watchDir "credential-access" "rwa") credentialDirs
|
||||
++ [ (watchDir "ssh-config" "wa" "/etc/ssh") ]
|
||||
++ map (watchFile "identity" "wa") identityFiles
|
||||
++ [
|
||||
(watchDir "identity" "wa" "/etc/pam.d")
|
||||
(watchDir "nix-trust" "wa" "/etc/nix")
|
||||
(watchFile "nix-trust" "wa" "/nix/var/nix/profiles/system")
|
||||
(watchDir "audit-config" "wa" "/etc/audit")
|
||||
(watchDir "audit-config" "wa" "/var/log/audit")
|
||||
]
|
||||
++ perArch "-S init_module,finit_module,delete_module -k kernel-module"
|
||||
++ perArch "-S execve,execveat -F euid=0 -F auid!=unset -k root-session"
|
||||
++ [
|
||||
"-a always,exit -S all -F exit=-EACCES -F auid!=unset -k access-denied"
|
||||
"-a always,exit -S all -F exit=-EPERM -F auid!=unset -k access-denied"
|
||||
]
|
||||
);
|
||||
};
|
||||
|
||||
security.auditd.settings = {
|
||||
log_file = "/var/log/audit/audit.log";
|
||||
max_log_file_action = "rotate";
|
||||
max_log_file = 50; # MiB
|
||||
num_logs = 5;
|
||||
};
|
||||
|
||||
# Parents of watches that may not exist yet on a fresh system.
|
||||
systemd.tmpfiles.rules = [
|
||||
"d /var/log/audit 0700 root root -"
|
||||
"d /nix/var/nix/profiles 0755 root root -"
|
||||
];
|
||||
|
||||
# A watch binds to the inode present at load time, so load after the
|
||||
# bind mounts (preservation), tmpfiles and the agenix ramfs.
|
||||
systemd.services.audit-rules-nixos = {
|
||||
after = [
|
||||
"local-fs.target"
|
||||
"systemd-tmpfiles-setup.service"
|
||||
]
|
||||
++ lib.optional hasAgenix "agenix-install-secrets.service";
|
||||
unitConfig.RequiresMountsFor = lib.unique (watchedDirs ++ map dirOf watchedFiles);
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -14,8 +14,12 @@
|
||||
# root login stays available for Colmena/MicroVM deployment, keys only
|
||||
PermitRootLogin = lib.mkDefault "prohibit-password";
|
||||
PasswordAuthentication = false; # disable password login
|
||||
# nixpkgs defaults it to true, and PAM then still accepts the Unix password
|
||||
KbdInteractiveAuthentication = false;
|
||||
};
|
||||
openFirewall = true;
|
||||
# Only keys declared in Nix log in; a hand- or intruder-added key is ignored
|
||||
authorizedKeysInHomedir = false;
|
||||
};
|
||||
|
||||
# NixOS appends an `Include` of systemd's ssh_config.d drop-in to
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
{ lib, outputs }:
|
||||
import ../../../x86_64-linux/tests/security-audit/expected.nix { inherit lib outputs; }
|
||||
@@ -0,0 +1,2 @@
|
||||
{ lib, outputs }:
|
||||
import ../../../x86_64-linux/tests/security-audit/expr.nix { inherit lib outputs; }
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
myvars,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
{
|
||||
# Neither password nor PAM keyboard-interactive login is offered.
|
||||
PasswordAuthentication = false;
|
||||
KbdInteractiveAuthentication = false;
|
||||
# Undeclared keys in ~/.ssh/authorized_keys are not honoured.
|
||||
authorizedKeysInHomedir = false;
|
||||
# Only the interactive admin keys open a root shell; remote-builder host keys
|
||||
# are pinned to the nix-daemon protocol, served by the system daemon.
|
||||
unrestrictedRootKeys = builtins.length (
|
||||
myvars.mainSshAuthorizedKeys ++ myvars.secondaryAuthorizedKeys
|
||||
);
|
||||
}
|
||||
// lib.optionalAttrs (name == "aquamarine") {
|
||||
builderKeys = map (
|
||||
key: ''restrict,command="nix-daemon --stdio --store daemon" ${key}''
|
||||
) myvars.remoteBuilderClientKeys;
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
let
|
||||
inherit (outputs.nixosConfigurations.${name}.config) services users;
|
||||
in
|
||||
{
|
||||
inherit (services.openssh.settings) PasswordAuthentication KbdInteractiveAuthentication;
|
||||
inherit (services.openssh) authorizedKeysInHomedir;
|
||||
# Keys that may open a root session without a forced command.
|
||||
unrestrictedRootKeys = builtins.length (
|
||||
lib.filter (key: !lib.hasPrefix "restrict," key) users.users.root.openssh.authorizedKeys.keys
|
||||
);
|
||||
}
|
||||
// lib.optionalAttrs (name == "aquamarine") {
|
||||
builderKeys = lib.filter (lib.hasPrefix "restrict,") users.users.root.openssh.authorizedKeys.keys;
|
||||
}
|
||||
)
|
||||
@@ -214,6 +214,18 @@ in
|
||||
pkgs = nixpkgs.legacyPackages.${system};
|
||||
networking = myvars.networking;
|
||||
};
|
||||
# Load the audit rules in a VM: one rule that fails to load leaves
|
||||
# auditing off, which evaluation cannot catch.
|
||||
security-auditd = import ../tests/security-auditd.nix {
|
||||
pkgs = nixpkgs.legacyPackages.${system};
|
||||
inherit (inputs) agenix;
|
||||
};
|
||||
# A persisted /etc/ssh must receive the activation copy at boot, which
|
||||
# only happens when preservation prepares the mount in initrd.
|
||||
security-preservation-etc = import ../tests/security-preservation-etc.nix {
|
||||
pkgs = nixpkgs.legacyPackages.${system};
|
||||
inherit (inputs) preservation;
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
let
|
||||
# Homelab servers reachable with the root deployment keys; not desktops.
|
||||
audited = builtins.elem name [
|
||||
"youko"
|
||||
"shoryu"
|
||||
"shushou"
|
||||
"aquamarine"
|
||||
];
|
||||
in
|
||||
{
|
||||
auditd = audited;
|
||||
rootSession = audited;
|
||||
sessionsOnly = audited;
|
||||
# Only emitted where agenix has secrets: a watch on a missing path
|
||||
# aborts loading the remaining rules.
|
||||
agenixWatch = name == "youko";
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
let
|
||||
inherit (outputs.nixosConfigurations.${name}) config;
|
||||
in
|
||||
{
|
||||
auditd = config.security.auditd.enable;
|
||||
rootSession = lib.any (lib.hasSuffix "-k root-session") config.security.audit.rules;
|
||||
sessionsOnly = lib.elem "-a never,task -F auid=unset" config.security.audit.rules;
|
||||
agenixWatch = lib.elem "-a always,exit -F dir=/run/agenix.d -F perm=rwa -k credential-access" config.security.audit.rules;
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,24 @@
|
||||
let
|
||||
sandbox = {
|
||||
CapabilityBoundingSet = "";
|
||||
NoNewPrivileges = true;
|
||||
PrivateDevices = true;
|
||||
ProtectSystem = "strict";
|
||||
SystemCallFilter = [
|
||||
"@system-service"
|
||||
"~@clock @cpu-emulation @debug @module @mount @obsolete @privileged @raw-io @reboot @swap"
|
||||
];
|
||||
};
|
||||
in
|
||||
{
|
||||
# Only 80/443.
|
||||
caddy = sandbox // {
|
||||
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
|
||||
};
|
||||
rustfs = sandbox;
|
||||
vmalert-homelab = sandbox;
|
||||
# The NPU is allowed through DeviceAllow instead.
|
||||
whisper-asr = sandbox // {
|
||||
PrivateDevices = false;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{ lib, outputs }:
|
||||
let
|
||||
keys = [
|
||||
"CapabilityBoundingSet"
|
||||
"NoNewPrivileges"
|
||||
"PrivateDevices"
|
||||
"ProtectSystem"
|
||||
"SystemCallFilter"
|
||||
];
|
||||
units =
|
||||
host: names:
|
||||
lib.genAttrs names (
|
||||
n: lib.getAttrs keys outputs.nixosConfigurations.${host}.config.systemd.services.${n}.serviceConfig
|
||||
);
|
||||
in
|
||||
units "youko" [
|
||||
"caddy"
|
||||
"rustfs"
|
||||
"vmalert-homelab"
|
||||
]
|
||||
// units "aquamarine" [ "whisper-asr" ]
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
myvars,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
{
|
||||
# Neither password nor PAM keyboard-interactive login is offered.
|
||||
PasswordAuthentication = false;
|
||||
KbdInteractiveAuthentication = false;
|
||||
# Undeclared keys in ~/.ssh/authorized_keys are not honoured.
|
||||
authorizedKeysInHomedir = false;
|
||||
# Only the interactive admin keys open a root shell; remote-builder host keys
|
||||
# are pinned to the nix-daemon protocol, served by the system daemon.
|
||||
unrestrictedRootKeys = builtins.length (
|
||||
myvars.mainSshAuthorizedKeys ++ myvars.secondaryAuthorizedKeys
|
||||
);
|
||||
}
|
||||
// lib.optionalAttrs (name == "aquamarine") {
|
||||
builderKeys = map (
|
||||
key: ''restrict,command="nix-daemon --stdio --store daemon" ${key}''
|
||||
) myvars.remoteBuilderClientKeys;
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
lib,
|
||||
outputs,
|
||||
}:
|
||||
lib.genAttrs (builtins.attrNames outputs.nixosConfigurations) (
|
||||
name:
|
||||
let
|
||||
inherit (outputs.nixosConfigurations.${name}.config) services users;
|
||||
in
|
||||
{
|
||||
inherit (services.openssh.settings) PasswordAuthentication KbdInteractiveAuthentication;
|
||||
inherit (services.openssh) authorizedKeysInHomedir;
|
||||
# Keys that may open a root session without a forced command.
|
||||
unrestrictedRootKeys = builtins.length (
|
||||
lib.filter (key: !lib.hasPrefix "restrict," key) users.users.root.openssh.authorizedKeys.keys
|
||||
);
|
||||
}
|
||||
// lib.optionalAttrs (name == "aquamarine") {
|
||||
builderKeys = lib.filter (lib.hasPrefix "restrict,") users.users.root.openssh.authorizedKeys.keys;
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,100 @@
|
||||
{ pkgs, agenix }:
|
||||
let
|
||||
# Throwaway age identity and secret, generated at build time for the VM.
|
||||
testSecrets = pkgs.runCommand "auditd-test-secrets" { nativeBuildInputs = [ pkgs.age ]; } ''
|
||||
mkdir $out
|
||||
age-keygen -o $out/key.txt
|
||||
echo not-a-real-secret | age -r "$(age-keygen -y $out/key.txt)" -o $out/secret.age
|
||||
'';
|
||||
in
|
||||
pkgs.testers.runNixOSTest {
|
||||
name = "security-auditd";
|
||||
nodes.machine =
|
||||
{ config, ... }:
|
||||
{
|
||||
imports = [
|
||||
../modules/nixos/base/auditd.nix
|
||||
agenix.nixosModules.default
|
||||
];
|
||||
modules.security.audit.enable = true;
|
||||
services.openssh.enable = true;
|
||||
age.identityPaths = [ "${testSecrets}/key.txt" ];
|
||||
age.secrets.test-secret.file = "${testSecrets}/secret.age";
|
||||
users.users.alice.isNormalUser = true;
|
||||
environment.etc."audit-rule-count".text = toString (builtins.length config.security.audit.rules);
|
||||
virtualisation.memorySize = 1024;
|
||||
};
|
||||
testScript = ''
|
||||
import shlex
|
||||
|
||||
def ssh(cmd):
|
||||
return machine.execute(
|
||||
"ssh -i /root/id -o StrictHostKeyChecking=no root@localhost " + shlex.quote(cmd)
|
||||
)
|
||||
|
||||
# auditd writes asynchronously, so poll; `-ul 0` keeps records whose
|
||||
# login uid (auid) is root, i.e. from the SSH session. Without
|
||||
# --input-logs, ausearch reads stdin when it is not a terminal.
|
||||
def wait_logged(key, match, flags="", count=1):
|
||||
machine.wait_until_succeeds(
|
||||
f"test $(ausearch --input-logs -i -k {key} {flags} | grep -c -- {shlex.quote(match)}) -ge {count}"
|
||||
)
|
||||
|
||||
machine.wait_for_unit("audit-rules-nixos.service")
|
||||
machine.wait_for_unit("auditd.service")
|
||||
machine.wait_for_unit("sshd.service")
|
||||
|
||||
with subtest("every rule loads and auditing is enabled"):
|
||||
t.assertIn("enabled 1", machine.succeed("auditctl -s"))
|
||||
rules = machine.succeed("auditctl -l")
|
||||
t.assertEqual(len(rules.splitlines()), int(machine.succeed("cat /etc/audit-rule-count")))
|
||||
t.assertIn("-w /run/agenix.d -p rwa -k credential-access", rules)
|
||||
t.assertIn("-a never,task -F auid=-1", rules)
|
||||
|
||||
# The test driver's shell is a service: this setup is not audited.
|
||||
machine.succeed(
|
||||
"ssh-keygen -q -t ed25519 -N ''' -f /root/id",
|
||||
"install -d -m 700 /root/.ssh",
|
||||
"cat /root/id.pub >> /root/.ssh/authorized_keys",
|
||||
)
|
||||
|
||||
with subtest("an SSH root session logs its commands and secret reads"):
|
||||
ssh("cat /run/agenix/test-secret; ${pkgs.hello}/bin/hello")
|
||||
wait_logged("root-session", "hello", "-ul 0")
|
||||
# sshd is built without libaudit; PAM records the login and its address.
|
||||
machine.succeed("ausearch --input-logs -i -m USER_START -ul 0 | grep -q 'addr=::1 terminal=ssh'")
|
||||
wait_logged("credential-access", "name=/run/agenix/test-secret", "-ul 0")
|
||||
|
||||
with subtest("a service is not audited; the same actions from a session are"):
|
||||
machine.succeed(
|
||||
"systemd-run --wait -p Environment=PATH=${pkgs.coreutils}/bin ${pkgs.runtimeShell} -c "
|
||||
"'cat /run/agenix/test-secret >/dev/null; touch /etc/ssh/from-service'"
|
||||
)
|
||||
ssh("cat /run/agenix/test-secret >/dev/null; touch /etc/ssh/from-session")
|
||||
wait_logged("ssh-config", "from-session", "-ul 0")
|
||||
wait_logged("credential-access", "name=/run/agenix/test-secret", "-ul 0", count=2)
|
||||
machine.succeed("test -e /etc/ssh/from-service")
|
||||
machine.fail("ausearch --input-logs -i -k ssh-config | grep -q from-service")
|
||||
# auditctl's own rule loads also carry the key (CONFIG_CHANGE, no auid).
|
||||
machine.fail("ausearch --input-logs -i -k credential-access | grep auid=unset | grep -q comm=cat")
|
||||
machine.fail("ausearch --input-logs -i -k root-session | grep -q from-service")
|
||||
# systemd-executor opens the agenix mount point with O_PATH while it
|
||||
# builds a sandboxed unit's namespace; it has no auid either.
|
||||
machine.succeed("systemctl restart nscd")
|
||||
machine.fail("ausearch --input-logs -i -k credential-access | grep -q comm=.nsncd")
|
||||
|
||||
with subtest("the agenix watch survives a new secrets generation"):
|
||||
machine.succeed("/run/current-system/activate")
|
||||
machine.succeed("readlink /run/agenix | grep -q '/2$'")
|
||||
ssh("cat /run/agenix/test-secret >/dev/null")
|
||||
wait_logged("credential-access", "name=/run/agenix/test-secret", "-ul 0", count=3)
|
||||
|
||||
with subtest("denials in a login session are recorded"):
|
||||
ssh("su alice -s /bin/sh -c 'cat /etc/shadow'")
|
||||
wait_logged("access-denied", "shadow", "-ul 0")
|
||||
|
||||
with subtest("auditd keeps its log; the journal keeps a copy"):
|
||||
machine.succeed("test -s /var/log/audit/audit.log")
|
||||
machine.succeed("journalctl _TRANSPORT=audit -q | grep -q root-session")
|
||||
'';
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
{ pkgs, preservation }:
|
||||
# `/etc` entries that carry a mode are *copied* into /etc by stage-2
|
||||
# activation (setup-etc.pl), which runs before systemd. preservation mounts a
|
||||
# persisted directory only after systemd starts, so a persisted /etc/ssh
|
||||
# shadows the fresh copies and sshd keeps reading the previous `switch`.
|
||||
#
|
||||
# This boots a VM with /etc/ssh persisted from a separate disk and checks that
|
||||
# the declared authorized_keys file reached the persistent volume, which only
|
||||
# happens when the mount is prepared in initrd.
|
||||
pkgs.testers.runNixOSTest {
|
||||
name = "security-preservation-etc";
|
||||
|
||||
nodes.machine =
|
||||
{ lib, ... }:
|
||||
{
|
||||
imports = [ preservation.nixosModules.default ];
|
||||
|
||||
virtualisation.emptyDiskImages = [ 256 ];
|
||||
virtualisation.memorySize = 1024;
|
||||
|
||||
fileSystems."/persistent" = {
|
||||
device = "/dev/vdb";
|
||||
fsType = "ext4";
|
||||
autoFormat = true;
|
||||
neededForBoot = true;
|
||||
};
|
||||
|
||||
preservation.enable = true;
|
||||
preservation.preserveAt."/persistent".directories = [
|
||||
{
|
||||
directory = "/etc/ssh";
|
||||
inInitrd = true;
|
||||
}
|
||||
];
|
||||
|
||||
services.openssh.enable = true;
|
||||
users.users.alice = {
|
||||
isNormalUser = true;
|
||||
openssh.authorizedKeys.keys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITestOnlyKeyMaterial000000000000000000000 test@example"
|
||||
];
|
||||
};
|
||||
};
|
||||
|
||||
testScript = ''
|
||||
# First boot only to get the persistent volume formatted.
|
||||
machine.start()
|
||||
machine.wait_for_unit("multi-user.target")
|
||||
|
||||
# Seed a stale authorized_keys file, as a previous `switch` would have left
|
||||
# it, then reboot into the same generation with a different declared key.
|
||||
machine.succeed("mkdir -p /persistent/etc/ssh/authorized_keys.d")
|
||||
machine.succeed("echo 'ssh-ed25519 AAAAOld stale@example' > /persistent/etc/ssh/authorized_keys.d/alice")
|
||||
machine.succeed("chmod 444 /persistent/etc/ssh/authorized_keys.d/alice")
|
||||
machine.shutdown()
|
||||
|
||||
machine.start()
|
||||
machine.wait_for_unit("multi-user.target")
|
||||
machine.succeed("mountpoint -q /etc/ssh")
|
||||
|
||||
# The boot-time copy must land on the persistent volume and replace the
|
||||
# stale file; with the mount ordered after activation it stays stale.
|
||||
machine.succeed("grep -q test@example /persistent/etc/ssh/authorized_keys.d/alice")
|
||||
machine.fail("grep -q stale@example /etc/ssh/authorized_keys.d/alice")
|
||||
machine.succeed("grep -q test@example /etc/ssh/authorized_keys.d/alice")
|
||||
machine.succeed("test \"$(stat -c %a /etc/ssh/authorized_keys.d/alice)\" = 444")
|
||||
|
||||
machine.wait_for_unit("sshd.service")
|
||||
'';
|
||||
}
|
||||
Reference in new issue
Block a user