nix-config-ryan4yin/outputs

Flake Outputs

Is such a complex and fine-grained structure necessary?

There is no need to do this when you have a small number of machines.

But when you have a large number of machines, it is necessary to manage them in a fine-grained way, otherwise, it will be difficult to manage and maintain them.

I have more than a dozen machines now, and a fine-grained layout keeps them manageable.

Tests

Simple configurations do not need tests, but a dozen machines make a broken setting easy to miss.

There are two types of tests: eval tests and NixOS tests. They catch configuration errors before a change reaches a real machine.

Related projects & docs:

• haumea: Filesystem-based module system for Nix
• Unveiling the Power of the NixOS Integration Test Driver (Part 1)
• NixOS Tests - NixOS Manual

1. Eval Tests

Eval Tests evaluate the expressions and compare the results with the expected results. It runs fast, but it doesn't build a real machine. We use eval tests to ensure that some attributes are correctly set for each NixOS and nix-darwin host.

How to run all the eval tests:

just test

just test exits non-zero unless the suite returns true. The underlying nix eval .#evalTests --show-trace --print-build-logs --verbose prints the result but exits 0 even on false, so read its output when you need the failing trace.

Each test is a directory outputs/<system>/tests/<name>/ holding expr.nix and expected.nix; the suite passes when every expr equals its expected.

Which tests cover a new host

Some tests iterate over every configuration, so a new host is covered automatically and must satisfy them. Others name specific hosts — or a name prefix, like k3s-api-endpoint, which covers every k3s-test-1-* configuration — and only cover a new host if you add it.

Kind	Tests
Every configuration	hostname, kernel, nix-system-features, security-apparmor, security-container-groups, security-exporters, security-firewall, security-k3s-kubeconfig, security-k3s-pod-firewall, security-kernel, security-ssh-x11
Named hosts only	btrbk, cpu-power, home-manager, home-manager-xdg, computer-use-headless, k3s-api-endpoint, k3s-node-home-manager, idols-ai-gpu, idols-ai-net, idols-ai-voice-input, idols-aquamarine, security-postgres-auth, youko-metrics, ups-metrics, shoukei-logind

hostname expects each configuration's networking.hostName to equal its name. Niri desktop configurations are the exception (ai-niri → ai, shoukei-niri → shoukei) and are listed in specialExpected in that platform's hostname/expected.nix.

2. NixOS Tests

WIP: not working yet

NixOS Tests builds and starts virtual machines using our NixOS configuration and run tests on them. Comparing to eval tests, it runs slow, but it builds a real machine, and we can test the whole system actually works as expected.

Problems:

• We need a private cache server, so that our NixOS tests do not need to build some custom packages every time we run the tests.
• Cannot test the whole host, because my host relies on its unique ssh host key to decrypt its agenix secrets.
  • Maybe it's better to test every service separately, not the whole host?

Full-host tests are tracked as WA-013 in WORKAROUNDS.md. outputs/x86_64-linux/nixos-tests/idols-ruby.nix is exported as packages.x86_64-linux.ruby-nixos-tests, but it is not a checks entry, so CI never builds it and nothing else runs it. What runs today is the eval set above with just test, plus one NixOS-driver check of the shared firewall — checks.x86_64-linux.security-exporters, built in CI.

Overview

All the outputs of this flake are defined here.

› tree
.
├── default.nix       # The entry point, all the outputs are composed here.
├── README.md
├── aarch64-darwin    # All outputs for macOS Apple Silicon
│   ├── default.nix
│   ├── src           # every host has its own file in this directory
│   │   ├── fern.nix
│   │   └── frieren.nix
│   └── tests         # eval tests
├── aarch64-linux     # All outputs for Linux ARM64
│   ├── default.nix
│   ├── src           # every host has its own file in this directory
│   │   ├── 12kingdoms-shoukei.nix
│   │   └── idols-akane.nix
│   └── tests         # eval tests
└── x86_64-linux      # All outputs for Linux x86_64
    ├── default.nix
    ├── nixos-tests
    ├── src           # every host has its own file in this directory
    │   ├── 12kingdoms-shoryu.nix
    │   ├── 12kingdoms-shushou.nix
    │   ├── 12kingdoms-youko.nix
    │   ├── idols-ai.nix
    │   ├── idols-kana.nix
    │   ├── idols-ruby.nix
    │   ├── k3s-test-1-master-1.nix
    │   ├── k3s-test-1-master-2.nix
    │   ├── k3s-test-1-master-3.nix
    │   ├── k3s-test-1-worker-1.nix
    │   ├── k3s-test-1-worker-2.nix
    │   └── k3s-test-1-worker-3.nix
    └── tests         # eval tests (btrbk, computer-use-headless, hostname,
                      # kernel, security-*, ups-metrics, ...)

The tree lists the per-host sources and tests. The flake also exposes a checks output: every system has eval-tests and pre-commit-check, and x86_64-linux adds security-exporters, which imports the top-level tests/security-exporters.nix. See ../SECURITY.md for when to run it.