mirror of
https://github.com/ryan4yin/nix-config.git
synced 2026-11-13 12:13:46 +01:00
chore: drop the KubeVirt guest bits
The k3s-test workers never ran (they were stuck waiting for volume
binding) and will be re-added as microVMs, so remove their KubeVirt
configs and everything that only existed for KubeVirt guests:
- hosts/k8s/k3s-test-1-worker-{1,2,3} and their outputs
- lib/genKubeVirtGuestModule.nix and lib/genK3sAgentModule.nix (the
latter was only used by the workers)
- the KubeVirt image upload recipes (upload-vm, upload-idols,
upload-k3s-test) and the kubevirt image package in the idols outputs
- refresh the docs; keep only accurate historical mentions (e.g. the
restic backup path must stay as it is)
This commit is contained in:
1 parent
621da40081
commit
6180072646
33 files changed
+39
-408
No files matched your search
@@ -61,10 +61,10 @@ deployments. Keep repository guidance here; reusable global rules live in `agent
|
|||||||
- Deployment and upload recipes change systems; use eval/build commands for validation. Remote
|
- Deployment and upload recipes change systems; use eval/build commands for validation. Remote
|
||||||
deployment requires an explicit request. When deployment behavior changes, report the exact `just`
|
deployment requires an explicit request. When deployment behavior changes, report the exact `just`
|
||||||
command to run.
|
command to run.
|
||||||
- KubeVirt hosts (`shoryu`/`shushou`/`youko`): avoid mid-flight restarts of the network
|
- VM hosts (`shoryu`/`shushou`/`youko`): avoid mid-flight restarts of the network
|
||||||
stack (`systemd-networkd` / the `br0` bridge) — that can drop the VM network. Use `boot` + serial
|
stack (`systemd-networkd` / the `br0` bridge) — that can drop the VM network. Use `boot` + serial
|
||||||
reboot for networking or nixpkgs changes. See
|
reboot for networking or nixpkgs changes. See
|
||||||
[hosts/README.md](./hosts/README.md#deploying-kubevirt-hosts).
|
[hosts/README.md](./hosts/README.md#deploying-vm-hosts).
|
||||||
- `just clean`, `just gc`, `just ggc`, and `just game` remove history or amend commits; they are not
|
- `just clean`, `just gc`, `just ggc`, and `just game` remove history or amend commits; they are not
|
||||||
validation steps and require explicit authorization for their target and scope.
|
validation steps and require explicit authorization for their target and scope.
|
||||||
- Do not use `just penvof` for process inspection: it can expose secret values.
|
- Do not use `just penvof` for process inspection: it can expose secret values.
|
||||||
|
|||||||
@@ -179,7 +179,7 @@ reset-launchpad:
|
|||||||
|
|
||||||
############################################################################
|
############################################################################
|
||||||
#
|
#
|
||||||
# Homelab - Kubevirt Cluster related commands
|
# Homelab - VM host related commands
|
||||||
#
|
#
|
||||||
############################################################################
|
############################################################################
|
||||||
|
|
||||||
@@ -189,14 +189,6 @@ reset-launchpad:
|
|||||||
col tag mode="switch":
|
col tag mode="switch":
|
||||||
colmena apply {{mode}} --on '@{{tag}}' --verbose --show-trace
|
colmena apply {{mode}} --on '@{{tag}}' --verbose --show-trace
|
||||||
|
|
||||||
# Build and upload a vm image
|
|
||||||
[linux]
|
|
||||||
[group('homelab')]
|
|
||||||
upload-vm name size verbosity="normal":
|
|
||||||
#!/usr/bin/env nu
|
|
||||||
use {{utils_nu}} *;
|
|
||||||
upload-vm {{name}} {{size}} {{verbosity}}
|
|
||||||
|
|
||||||
# Deploy all the VM hosts (physical machines running the VMs)
|
# Deploy all the VM hosts (physical machines running the VMs)
|
||||||
[linux]
|
[linux]
|
||||||
[group('homelab')]
|
[group('homelab')]
|
||||||
@@ -224,15 +216,6 @@ youko mode="switch":
|
|||||||
#
|
#
|
||||||
############################################################################
|
############################################################################
|
||||||
|
|
||||||
# Build and upload a vm image
|
|
||||||
[linux]
|
|
||||||
[group('homelab')]
|
|
||||||
upload-idols verbosity="normal":
|
|
||||||
#!/usr/bin/env nu
|
|
||||||
use {{utils_nu}} *;
|
|
||||||
upload-vm ruby 100Gi {{verbosity}}
|
|
||||||
upload-vm kana 100Gi {{verbosity}}
|
|
||||||
|
|
||||||
[linux]
|
[linux]
|
||||||
[group('homelab')]
|
[group('homelab')]
|
||||||
ruby mode="switch":
|
ruby mode="switch":
|
||||||
@@ -249,19 +232,6 @@ kana mode="switch":
|
|||||||
#
|
#
|
||||||
############################################################################
|
############################################################################
|
||||||
|
|
||||||
# Build and upload a vm image
|
|
||||||
[linux]
|
|
||||||
[group('homelab')]
|
|
||||||
upload-k3s-test verbosity="normal":
|
|
||||||
#!/usr/bin/env nu
|
|
||||||
use {{utils_nu}} *;
|
|
||||||
upload-vm k3s-test-1-master-1 50Gi {{verbosity}};
|
|
||||||
upload-vm k3s-test-1-master-2 50Gi {{verbosity}};
|
|
||||||
upload-vm k3s-test-1-master-3 50Gi {{verbosity}};
|
|
||||||
upload-vm k3s-test-1-worker-1 50Gi {{verbosity}};
|
|
||||||
upload-vm k3s-test-1-worker-2 50Gi {{verbosity}};
|
|
||||||
upload-vm k3s-test-1-worker-3 50Gi {{verbosity}};
|
|
||||||
|
|
||||||
[linux]
|
[linux]
|
||||||
[group('homelab')]
|
[group('homelab')]
|
||||||
k3s-test mode="switch":
|
k3s-test mode="switch":
|
||||||
|
|||||||
@@ -27,11 +27,11 @@ This repository is home to the Nix code that builds all of my systems:
|
|||||||
agenix.
|
agenix.
|
||||||
2. **macOS desktops** — nix-darwin + Home Manager, sharing the same `home/` configuration with the
|
2. **macOS desktops** — nix-darwin + Home Manager, sharing the same `home/` configuration with the
|
||||||
NixOS desktops.
|
NixOS desktops.
|
||||||
3. **NixOS servers** — KubeVirt VMs running on three physical mini PCs, hosting K3s clusters,
|
3. **NixOS servers** — VMs running on three physical mini PCs, hosting K3s clusters,
|
||||||
monitoring, and other self-hosted services.
|
monitoring, and other self-hosted services.
|
||||||
|
|
||||||
See [./hosts](./hosts) for the host inventory, [./outputs](./outputs) for how the flake outputs are
|
See [./hosts](./hosts) for the host inventory, [./outputs](./outputs) for how the flake outputs are
|
||||||
composed, [./Virtual-Machine.md](./Virtual-Machine.md) for creating & managing KubeVirt VMs, and
|
composed, [./Virtual-Machine.md](./Virtual-Machine.md) for creating & managing VMs, and
|
||||||
[./AGENTS.md](./AGENTS.md) for the repository conventions.
|
[./AGENTS.md](./AGENTS.md) for the repository conventions.
|
||||||
|
|
||||||
## Why NixOS & Flakes?
|
## Why NixOS & Flakes?
|
||||||
|
|||||||
+3
-3
@@ -1,7 +1,7 @@
|
|||||||
## How to create & managage KubeVirt's Virtual Machine from this flake?
|
## How to create & manage VMs (the retired KubeVirt path)
|
||||||
|
|
||||||
> [!NOTE] KubeVirt is being retired. aquamarine runs natively on `youko` and the k3s-test
|
> [!NOTE] KubeVirt is being retired. aquamarine runs natively on `youko` and the k3s-test
|
||||||
> nodes run as [microVMs](./hosts/k8s/README.md); this document only covers the KubeVirt VMs that
|
> nodes run as [microVMs](./hosts/k8s/README.md); this document only covers the VMs that
|
||||||
> are left to move (currently the Windows VM).
|
> are left to move (currently the Windows VM).
|
||||||
|
|
||||||
Use `aquamarine` as an example, first build and upload the virtual machine's qcow2 image to the file
|
Use `aquamarine` as an example, first build and upload the virtual machine's qcow2 image to the file
|
||||||
@@ -15,7 +15,7 @@ Then create the virtual machine by creating a yaml file at
|
|||||||
[ryan4yin/k8s-gitops](https://github.com/ryan4yin/k8s-gitops/tree/main/vms), set the
|
[ryan4yin/k8s-gitops](https://github.com/ryan4yin/k8s-gitops/tree/main/vms), set the
|
||||||
`spec.dataVolumeTemplates[0].source.http.url` to the uploaded file's URL, and fluxcd will
|
`spec.dataVolumeTemplates[0].source.http.url` to the uploaded file's URL, and fluxcd will
|
||||||
automatically apply the changes, then a virtual machine named `aquamarine` will be created in the
|
automatically apply the changes, then a virtual machine named `aquamarine` will be created in the
|
||||||
KubeVirt cluster.
|
VM cluster.
|
||||||
|
|
||||||
Once the virtual machine `aquamarine` is created, we can deploy updates to it with the following
|
Once the virtual machine `aquamarine` is created, we can deploy updates to it with the following
|
||||||
commands:
|
commands:
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 116 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 981 KiB |
@@ -50,7 +50,7 @@
|
|||||||
# get some latest packages from the master branch
|
# get some latest packages from the master branch
|
||||||
nixpkgs-master.url = "github:nixos/nixpkgs/master";
|
nixpkgs-master.url = "github:nixos/nixpkgs/master";
|
||||||
|
|
||||||
# NixOS MicroVMs -- run the VMs directly on the hosts, replacing KubeVirt
|
# NixOS MicroVMs -- run the VMs directly on the hosts, replacing the former KubeVirt
|
||||||
microvm = {
|
microvm = {
|
||||||
url = "github:microvm-nix/microvm.nix";
|
url = "github:microvm-nix/microvm.nix";
|
||||||
inputs.nixpkgs.follows = "nixpkgs";
|
inputs.nixpkgs.follows = "nixpkgs";
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
...
|
...
|
||||||
}:
|
}:
|
||||||
let
|
let
|
||||||
# Shared golden-image store for the KubeVirt cluster. It is exported over
|
# Shared golden-image store for the VM cluster. It is exported over
|
||||||
# NFSv4 so the cluster's NFS CSI driver can hand out RWX volumes to every
|
# NFSv4 so the cluster's NFS CSI driver can hand out RWX volumes to every
|
||||||
# node, and VM disks are cloned from it into local-path.
|
# node, and VM disks are cloned from it into local-path.
|
||||||
#
|
#
|
||||||
@@ -10,7 +10,7 @@ let
|
|||||||
# free space on this host.
|
# free space on this host.
|
||||||
goldenDir = "/persistent/nfs/golden";
|
goldenDir = "/persistent/nfs/golden";
|
||||||
|
|
||||||
# Only the homelab LAN may mount the export. The KubeVirt nodes are the
|
# Only the homelab LAN may mount the export. The VM hosts are the
|
||||||
# clients (the CSI node plugin mounts from the node, not from the pod).
|
# clients (the CSI node plugin mounts from the node, not from the pod).
|
||||||
clientCidr = "192.168.5.0/24";
|
clientCidr = "192.168.5.0/24";
|
||||||
in
|
in
|
||||||
|
|||||||
+8
-8
@@ -13,7 +13,7 @@ Named after characters from "Oshi no Ko":
|
|||||||
| Host | Platform | Hardware | Purpose | Status |
|
| Host | Platform | Hardware | Purpose | Status |
|
||||||
| ------------ | ----------- | --------------------- | --------------------- | ----------- |
|
| ------------ | ----------- | --------------------- | --------------------- | ----------- |
|
||||||
| `ai` | NixOS | i5-13600KF + RTX 4090 | Gaming & Daily Use | ✅ Active |
|
| `ai` | NixOS | i5-13600KF + RTX 4090 | Gaming & Daily Use | ✅ Active |
|
||||||
| `aquamarine` | KubeVirt VM | Virtual | Monitoring & Services | ✅ Active |
|
| `aquamarine` | VM | Virtual | Monitoring & Services | ✅ Active |
|
||||||
| `kana` | NixOS | Virtual | Reserved | ⚪ Not Used |
|
| `kana` | NixOS | Virtual | Reserved | ⚪ Not Used |
|
||||||
| `ruby` | NixOS | Virtual | Reserved | ⚪ Not Used |
|
| `ruby` | NixOS | Virtual | Reserved | ⚪ Not Used |
|
||||||
|
|
||||||
@@ -33,15 +33,15 @@ Named after "Twelve Kingdoms":
|
|||||||
| Host | Platform | Hardware | Purpose | Status |
|
| Host | Platform | Hardware | Purpose | Status |
|
||||||
| --------- | -------- | -------------------------------------- | -------------------------- | --------- |
|
| --------- | -------- | -------------------------------------- | -------------------------- | --------- |
|
||||||
| `shoukei` | NixOS | MacBook Pro M2 | NixOS on Apple Silicon | ✅ Active |
|
| `shoukei` | NixOS | MacBook Pro M2 | NixOS on Apple Silicon | ✅ Active |
|
||||||
| `shoryu` | NixOS | MoreFine S500Plus (AMD Ryzen 9 5900HX) | KubeVirt Host & K3s Master | ✅ Active |
|
| `shoryu` | NixOS | MoreFine S500Plus (AMD Ryzen 9 5900HX) | VM Host | ✅ Active |
|
||||||
| `shushou` | NixOS | MinisForum UM560 (AMD Ryzen 5 5625U) | KubeVirt Host & K3s Master | ✅ Active |
|
| `shushou` | NixOS | MinisForum UM560 (AMD Ryzen 5 5625U) | VM Host | ✅ Active |
|
||||||
| `youko` | NixOS | MinisForum HX99G (AMD Ryzen 9 6900HX) | KubeVirt Host & K3s Master | ✅ Active |
|
| `youko` | NixOS | MinisForum HX99G (AMD Ryzen 9 6900HX) | VM Host | ✅ Active |
|
||||||
|
|
||||||
### Virtual Machines & Clusters
|
### Virtual Machines & Clusters
|
||||||
|
|
||||||
#### `k8s` - Kubernetes Infrastructure
|
#### `k8s` - Kubernetes Infrastructure
|
||||||
|
|
||||||
- **KubeVirt Cluster**: 3 physical mini PCs (shoryu, shushou, youko) running all VMs
|
- **VM Cluster**: 3 physical mini PCs (shoryu, shushou, youko) running all VMs
|
||||||
- **K3s Testing**: 3 masters + 3 workers for testing and development
|
- **K3s Testing**: 3 masters + 3 workers for testing and development
|
||||||
|
|
||||||
### External Systems
|
### External Systems
|
||||||
@@ -104,13 +104,13 @@ Use existing hosts as templates. The key files typically include:
|
|||||||
### Examples to Reference
|
### Examples to Reference
|
||||||
|
|
||||||
- **Desktop systems**: See `idols-ai/` for gaming/workstation setup
|
- **Desktop systems**: See `idols-ai/` for gaming/workstation setup
|
||||||
- **Server systems**: See `shoryu/` for K8s/KubeVirt hosts
|
- **Server systems**: See `shoryu/` for K8s/VM hosts
|
||||||
- **macOS systems**: See `darwin-fern/` for macOS configurations
|
- **macOS systems**: See `darwin-fern/` for macOS configurations
|
||||||
- **Apple Silicon**: See `12kingdoms-shoukei/` for ARM Linux setup
|
- **Apple Silicon**: See `12kingdoms-shoukei/` for ARM Linux setup
|
||||||
|
|
||||||
## Deploying KubeVirt Hosts
|
## Deploying VM Hosts
|
||||||
|
|
||||||
The three KubeVirt hosts (`shoryu`, `shushou`, `youko`) put their VMs'
|
The three VM hosts (`shoryu`, `shushou`, `youko`) put their VMs'
|
||||||
secondary network on the Linux bridge `br0` (attached by the `bridge` CNI plugin); the pod network
|
secondary network on the Linux bridge `br0` (attached by the `bridge` CNI plugin); the pod network
|
||||||
is Cilium (flannel disabled).
|
is Cilium (flannel disabled).
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
}:
|
}:
|
||||||
#############################################################
|
#############################################################
|
||||||
#
|
#
|
||||||
# Kana - a NixOS VM running on Proxmox/KubeVirt
|
# Kana - a NixOS VM running under libvirt
|
||||||
#
|
#
|
||||||
#############################################################
|
#############################################################
|
||||||
let
|
let
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
}:
|
}:
|
||||||
#############################################################
|
#############################################################
|
||||||
#
|
#
|
||||||
# Ruby - a NixOS VM running on Proxmox/KubeVirt
|
# Ruby - a NixOS VM running under libvirt
|
||||||
#
|
#
|
||||||
#############################################################
|
#############################################################
|
||||||
let
|
let
|
||||||
|
|||||||
@@ -64,7 +64,6 @@ in
|
|||||||
# cloud-native
|
# cloud-native
|
||||||
kubectl
|
kubectl
|
||||||
istioctl
|
istioctl
|
||||||
kubevirt # virtctl
|
|
||||||
kubernetes-helm
|
kubernetes-helm
|
||||||
fluxcd
|
fluxcd
|
||||||
terraform
|
terraform
|
||||||
|
|||||||
+4
-6
@@ -7,13 +7,11 @@ I'm running a Kubernetes cluster for testing and development.
|
|||||||
I prefer to use [k3s] as the Kubernetes distribution, because it's lightweight, easy to install, and
|
I prefer to use [k3s] as the Kubernetes distribution, because it's lightweight, easy to install, and
|
||||||
full featured(see [what-have-k3s-removed-from-upstream-kubernetes] for details).
|
full featured(see [what-have-k3s-removed-from-upstream-kubernetes] for details).
|
||||||
|
|
||||||
## KubeVirt Cluster
|
## VM Cluster
|
||||||
|
|
||||||
The KubeVirt cluster is running on physical machines, all my virtual machines are running on this
|
The VM cluster is running on physical machines, all my virtual machines are running on this
|
||||||
cluster, including other Kubernetes clusters.
|
cluster, including other Kubernetes clusters.
|
||||||
|
|
||||||

|
|
||||||

|
|
||||||
|
|
||||||
## K3s Clusters
|
## K3s Clusters
|
||||||
|
|
||||||
@@ -30,7 +28,7 @@ image has to be built or uploaded; only the volumes above persist. The guest's t
|
|||||||
bridged onto `br0`, and its name is derived from the guest IP (e.g. `192.168.5.114` -> `vm114`, as
|
bridged onto `br0`, and its name is derived from the guest IP (e.g. `192.168.5.114` -> `vm114`, as
|
||||||
`IFNAMSIZ` caps interface names at 15 characters).
|
`IFNAMSIZ` caps interface names at 15 characters).
|
||||||
|
|
||||||
To move an existing KubeVirt VM over in place, copy the host key and `machine-id` from its disk into
|
To move an existing VM over in place, copy the host key and `machine-id` from its disk into
|
||||||
`etc.img`, and `/var/lib/rancher/k3s/server` into `var.img`, **before the first boot** — then it
|
`etc.img`, and `/var/lib/rancher/k3s/server` into `var.img`, **before the first boot** — then it
|
||||||
keeps its identity and etcd membership instead of bootstraping a new cluster.
|
keeps its identity and etcd membership instead of bootstraping a new cluster.
|
||||||
|
|
||||||
@@ -41,7 +39,7 @@ keeps its identity and etcd membership instead of bootstraping a new cluster.
|
|||||||
|
|
||||||
- **Retire KubeVirt.** aquamarine now runs natively on `youko`, and the k3s-test masters
|
- **Retire KubeVirt.** aquamarine now runs natively on `youko`, and the k3s-test masters
|
||||||
run as microVMs. Remove KubeVirt/CDI/multus from the cluster (k8s-gitops) once nothing else needs
|
run as microVMs. Remove KubeVirt/CDI/multus from the cluster (k8s-gitops) once nothing else needs
|
||||||
them, and the Windows VM is the only KubeVirt VM left to move.
|
them, and the Windows VM is the only VM left to move.
|
||||||
- **The USB HDD bridge is flaky.** aquamarine's two HDDs sit behind a JMicron JMS567 USB-SATA bridge
|
- **The USB HDD bridge is flaky.** aquamarine's two HDDs sit behind a JMicron JMS567 USB-SATA bridge
|
||||||
that keeps resetting (`dmesg` on `youko`). Find out how often it resets and how much it
|
that keeps resetting (`dmesg` on `youko`). Find out how often it resets and how much it
|
||||||
matters before putting anything critical (e.g. an NFS export) on it.
|
matters before putting anything critical (e.g. an NFS export) on it.
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
{
|
|
||||||
config,
|
|
||||||
pkgs,
|
|
||||||
myvars,
|
|
||||||
mylib,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hostName = "k3s-test-1-worker-1"; # define your hostname.
|
|
||||||
|
|
||||||
coreModule = mylib.genKubeVirtGuestModule {
|
|
||||||
inherit pkgs hostName;
|
|
||||||
inherit (myvars) networking;
|
|
||||||
};
|
|
||||||
k3sModule = mylib.genK3sAgentModule {
|
|
||||||
inherit pkgs;
|
|
||||||
tokenFile = config.age.secrets."k3s-test-1-token".path;
|
|
||||||
# use my own domain & kube-vip's virtual IP for the API server
|
|
||||||
# so that the API server can always be accessed even if some nodes are down
|
|
||||||
masterHost = "test-cluster-1.writefor.fun";
|
|
||||||
# workloads run on the workers; the masters are tainted (see k3s-test-1-master-*)
|
|
||||||
nodeLabels = [ "node-role.kubernetes.io/worker=true" ];
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
imports = (mylib.scanPaths ./.) ++ [
|
|
||||||
coreModule
|
|
||||||
k3sModule
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
{
|
|
||||||
config,
|
|
||||||
pkgs,
|
|
||||||
myvars,
|
|
||||||
mylib,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hostName = "k3s-test-1-worker-2"; # define your hostname.
|
|
||||||
|
|
||||||
coreModule = mylib.genKubeVirtGuestModule {
|
|
||||||
inherit pkgs hostName;
|
|
||||||
inherit (myvars) networking;
|
|
||||||
};
|
|
||||||
k3sModule = mylib.genK3sAgentModule {
|
|
||||||
inherit pkgs;
|
|
||||||
tokenFile = config.age.secrets."k3s-test-1-token".path;
|
|
||||||
# use my own domain & kube-vip's virtual IP for the API server
|
|
||||||
# so that the API server can always be accessed even if some nodes are down
|
|
||||||
masterHost = "test-cluster-1.writefor.fun";
|
|
||||||
# workloads run on the workers; the masters are tainted (see k3s-test-1-master-*)
|
|
||||||
nodeLabels = [ "node-role.kubernetes.io/worker=true" ];
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
imports = (mylib.scanPaths ./.) ++ [
|
|
||||||
coreModule
|
|
||||||
k3sModule
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
{
|
|
||||||
config,
|
|
||||||
pkgs,
|
|
||||||
myvars,
|
|
||||||
mylib,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hostName = "k3s-test-1-worker-3"; # define your hostname.
|
|
||||||
|
|
||||||
coreModule = mylib.genKubeVirtGuestModule {
|
|
||||||
inherit pkgs hostName;
|
|
||||||
inherit (myvars) networking;
|
|
||||||
};
|
|
||||||
k3sModule = mylib.genK3sAgentModule {
|
|
||||||
inherit pkgs;
|
|
||||||
tokenFile = config.age.secrets."k3s-test-1-token".path;
|
|
||||||
# use my own domain & kube-vip's virtual IP for the API server
|
|
||||||
# so that the API server can always be accessed even if some nodes are down
|
|
||||||
masterHost = "test-cluster-1.writefor.fun";
|
|
||||||
# workloads run on the workers; the masters are tainted (see k3s-test-1-master-*)
|
|
||||||
nodeLabels = [ "node-role.kubernetes.io/worker=true" ];
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
imports = (mylib.scanPaths ./.) ++ [
|
|
||||||
coreModule
|
|
||||||
k3sModule
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@@ -18,8 +18,6 @@ easier to add new machines.
|
|||||||
|
|
||||||
5. **`genK3sAgentModule.nix`** - K3s agent node configuration generator
|
5. **`genK3sAgentModule.nix`** - K3s agent node configuration generator
|
||||||
6. **`genK3sServerModule.nix`** - K3s server node configuration generator
|
6. **`genK3sServerModule.nix`** - K3s server node configuration generator
|
||||||
7. **`genKubeVirtGuestModule.nix`** - KubeVirt guest VM configuration generator
|
|
||||||
8. **`genKubeVirtHostModule.nix`** - KubeVirt host configuration generator
|
|
||||||
|
|
||||||
### Entry Point
|
### Entry Point
|
||||||
|
|
||||||
|
|||||||
@@ -7,9 +7,7 @@
|
|||||||
attrs = import ./attrs.nix { inherit lib; };
|
attrs = import ./attrs.nix { inherit lib; };
|
||||||
|
|
||||||
genK3sServerModule = import ./genK3sServerModule.nix;
|
genK3sServerModule = import ./genK3sServerModule.nix;
|
||||||
genK3sAgentModule = import ./genK3sAgentModule.nix;
|
|
||||||
genVmHostModule = import ./genVmHostModule.nix;
|
genVmHostModule = import ./genVmHostModule.nix;
|
||||||
genKubeVirtGuestModule = import ./genKubeVirtGuestModule.nix;
|
|
||||||
genMicrovmGuestModule = import ./genMicrovmGuestModule.nix;
|
genMicrovmGuestModule = import ./genMicrovmGuestModule.nix;
|
||||||
|
|
||||||
# use path relative to the root of the project
|
# use path relative to the root of the project
|
||||||
|
|||||||
@@ -1,45 +0,0 @@
|
|||||||
{
|
|
||||||
pkgs,
|
|
||||||
masterHost,
|
|
||||||
tokenFile,
|
|
||||||
nodeLabels ? [ ],
|
|
||||||
k3sExtraArgs ? [ ],
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
package = pkgs.k3s;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
environment.systemPackages = [ package ];
|
|
||||||
|
|
||||||
# Kernel modules required by cilium
|
|
||||||
boot.kernelModules = [
|
|
||||||
"ip6_tables"
|
|
||||||
"ip6table_mangle"
|
|
||||||
"ip6table_raw"
|
|
||||||
"ip6table_filter"
|
|
||||||
];
|
|
||||||
networking.enableIPv6 = true;
|
|
||||||
networking.nat = {
|
|
||||||
enable = true;
|
|
||||||
enableIPv6 = true;
|
|
||||||
};
|
|
||||||
|
|
||||||
services.k3s = {
|
|
||||||
enable = true;
|
|
||||||
inherit package tokenFile;
|
|
||||||
|
|
||||||
role = "agent";
|
|
||||||
serverAddr = "https://${masterHost}:6443";
|
|
||||||
# https://docs.k3s.io/cli/agent
|
|
||||||
extraFlags =
|
|
||||||
let
|
|
||||||
flagList = [
|
|
||||||
"--data-dir /var/lib/rancher/k3s"
|
|
||||||
]
|
|
||||||
++ (map (label: "--node-label=${label}") nodeLabels)
|
|
||||||
++ k3sExtraArgs;
|
|
||||||
in
|
|
||||||
pkgs.lib.concatStringsSep " " flagList;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -62,7 +62,7 @@ in
|
|||||||
"--write-kubeconfig=${kubeconfigFile}"
|
"--write-kubeconfig=${kubeconfigFile}"
|
||||||
"--write-kubeconfig-mode=600"
|
"--write-kubeconfig-mode=600"
|
||||||
"--service-node-port-range=80-32767"
|
"--service-node-port-range=80-32767"
|
||||||
"--kube-apiserver-arg='--allow-privileged=true'" # required by kubevirt
|
"--kube-apiserver-arg='--allow-privileged=true'" # required by the former KubeVirt VMs
|
||||||
"--data-dir /var/lib/rancher/k3s"
|
"--data-dir /var/lib/rancher/k3s"
|
||||||
"--etcd-expose-metrics=true"
|
"--etcd-expose-metrics=true"
|
||||||
"--etcd-snapshot-schedule-cron='0 */12 * * *'"
|
"--etcd-snapshot-schedule-cron='0 */12 * * *'"
|
||||||
|
|||||||
@@ -1,71 +0,0 @@
|
|||||||
{
|
|
||||||
pkgs,
|
|
||||||
hostName,
|
|
||||||
networking,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
inherit (networking) proxyGateway proxyGateway6 nameservers;
|
|
||||||
inherit (networking.hostsAddr.${hostName}) iface ipv4;
|
|
||||||
ipv4WithMask = "${ipv4}/24";
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# supported file systems, so we can mount any removable disks with these filesystems
|
|
||||||
boot.supportedFilesystems = [
|
|
||||||
"ext4"
|
|
||||||
"btrfs"
|
|
||||||
"xfs"
|
|
||||||
"fat"
|
|
||||||
"vfat"
|
|
||||||
"exfat"
|
|
||||||
];
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
inherit hostName;
|
|
||||||
|
|
||||||
# we use networkd instead
|
|
||||||
networkmanager.enable = false;
|
|
||||||
useDHCP = false;
|
|
||||||
};
|
|
||||||
networking.useNetworkd = true;
|
|
||||||
systemd.network.enable = true;
|
|
||||||
|
|
||||||
systemd.network.networks."10-${iface}" = {
|
|
||||||
matchConfig.Name = [ iface ];
|
|
||||||
networkConfig = {
|
|
||||||
Address = [ ipv4WithMask ];
|
|
||||||
# DNS = nameservers;
|
|
||||||
DNS = [ proxyGateway ];
|
|
||||||
DHCP = "ipv6"; # enable DHCPv6 only, so we can get a GUA.
|
|
||||||
IPv6AcceptRA = true; # for Stateless IPv6 Autoconfiguraton (SLAAC)
|
|
||||||
LinkLocalAddressing = "ipv6";
|
|
||||||
};
|
|
||||||
routes = [
|
|
||||||
{
|
|
||||||
Destination = "0.0.0.0/0";
|
|
||||||
Gateway = proxyGateway;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
Destination = "::/0";
|
|
||||||
Gateway = proxyGateway6;
|
|
||||||
GatewayOnLink = true; # it's a gateway on local link.
|
|
||||||
}
|
|
||||||
];
|
|
||||||
linkConfig.RequiredForOnline = "routable";
|
|
||||||
};
|
|
||||||
|
|
||||||
# systemd-journal - reduce disk usage
|
|
||||||
# https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html
|
|
||||||
services.journald.settings.Journal = {
|
|
||||||
SystemMaxUse = "2G";
|
|
||||||
RuntimeMaxUse = "256M";
|
|
||||||
};
|
|
||||||
|
|
||||||
# This value determines the NixOS release from which the default
|
|
||||||
# settings for stateful data, like file locations and database versions
|
|
||||||
# on your system were taken. It‘s perfectly fine and recommended to leave
|
|
||||||
# this value at the release version of the first install of this system.
|
|
||||||
# Before changing this value read the documentation for this option
|
|
||||||
# (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
|
|
||||||
system.stateVersion = "24.11"; # Did you read the comment?
|
|
||||||
}
|
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
# (BIOS/q35, virtio disk, bridged NIC, VNC display).
|
# (BIOS/q35, virtio disk, bridged NIC, VNC display).
|
||||||
#
|
#
|
||||||
# The NIC is pinned to q35 PCI bus 0x02 and the disk to bus 0x04 so the guest
|
# The NIC is pinned to q35 PCI bus 0x02 and the disk to bus 0x04 so the guest
|
||||||
# sees the same interface name it did under KubeVirt (enp2s0) and the same
|
# sees the same interface name it did under the former KubeVirt (enp2s0) and the same
|
||||||
# device ordering.
|
# device ordering.
|
||||||
{ lib }:
|
{ lib }:
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
pkgs,
|
pkgs,
|
||||||
hostName,
|
hostName,
|
||||||
networking,
|
networking,
|
||||||
# vCPU count and RAM (MiB) -- mirrors the KubeVirt instancetype
|
# vCPU count and RAM (MiB) -- mirrors the former KubeVirt instancetype
|
||||||
vcpu,
|
vcpu,
|
||||||
mem,
|
mem,
|
||||||
# persistent volume sizes in MiB
|
# persistent volume sizes in MiB
|
||||||
|
|||||||
+1
-1
@@ -57,7 +57,7 @@ modules/
|
|||||||
│ ├── virtualisation.nix # Virtualization support
|
│ ├── virtualisation.nix # Virtualization support
|
||||||
│ └── xdg.nix # XDG base directory settings
|
│ └── xdg.nix # XDG base directory settings
|
||||||
└── server/ # Server-specific modules
|
└── server/ # Server-specific modules
|
||||||
├── kubevirt-hardware-configuration.nix
|
├── qemu-guest-hardware-configuration.nix
|
||||||
├── server-aarch64.nix
|
├── server-aarch64.nix
|
||||||
├── server-riscv64.nix
|
├── server-riscv64.nix
|
||||||
└── server.nix
|
└── server.nix
|
||||||
|
|||||||
@@ -5,12 +5,12 @@
|
|||||||
}:
|
}:
|
||||||
##############################################################################
|
##############################################################################
|
||||||
#
|
#
|
||||||
# Template for KubeVirt's VM, mainly based on:
|
# Template for the VM guests (formerly KubeVirt), mainly based on:
|
||||||
# https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/kubevirt.nix
|
# https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/kubevirt.nix
|
||||||
#
|
#
|
||||||
# We write our hardware-configuration.nix, so that we can do some customization more easily.
|
# We write our hardware-configuration.nix, so that we can do some customization more easily.
|
||||||
#
|
#
|
||||||
# the url above is used by `nixos-generator` to generate the KubeVirt's qcow2 image file.
|
# the url above is used by `nixos-generator` to generate the guest's qcow2 image file.
|
||||||
#
|
#
|
||||||
##############################################################################
|
##############################################################################
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -44,5 +44,4 @@ in
|
|||||||
|
|
||||||
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
||||||
|
|
||||||
packages.${name} = inputs.self.nixosConfigurations.${name}.config.system.build.images.kubevirt;
|
|
||||||
}
|
}
|
||||||
@@ -44,8 +44,6 @@ in
|
|||||||
|
|
||||||
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
||||||
|
|
||||||
packages.${name} = inputs.self.nixosConfigurations.${name}.config.system.build.images.kubevirt;
|
|
||||||
|
|
||||||
# nixos tests
|
# nixos tests
|
||||||
packages."${name}-nixos-tests" = import ../nixos-tests/idols-ruby.nix systemArgs;
|
packages."${name}-nixos-tests" = import ../nixos-tests/idols-ruby.nix systemArgs;
|
||||||
}
|
}
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
{
|
|
||||||
# NOTE: the args not used in this file CAN NOT be removed!
|
|
||||||
# because haumea pass argument lazily,
|
|
||||||
# and these arguments are used in the functions like `mylib.nixosSystem`, `mylib.colmenaSystem`, etc.
|
|
||||||
inputs,
|
|
||||||
lib,
|
|
||||||
mylib,
|
|
||||||
myvars,
|
|
||||||
system,
|
|
||||||
genSpecialArgs,
|
|
||||||
...
|
|
||||||
}@args:
|
|
||||||
let
|
|
||||||
name = "k3s-test-1-worker-1";
|
|
||||||
tags = [ name ];
|
|
||||||
ssh-user = "root";
|
|
||||||
|
|
||||||
modules = {
|
|
||||||
nixos-modules =
|
|
||||||
(map mylib.relativeToRoot [
|
|
||||||
# common
|
|
||||||
"secrets/nixos.nix"
|
|
||||||
"modules/nixos/server/server.nix"
|
|
||||||
"modules/nixos/server/qemu-guest-hardware-configuration.nix"
|
|
||||||
# host specific
|
|
||||||
"hosts/k8s/${name}"
|
|
||||||
])
|
|
||||||
++ [
|
|
||||||
{ modules.secrets.server.kubernetes.enable = true; }
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
systemArgs = modules // args;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
nixosConfigurations.${name} = mylib.nixosSystem systemArgs;
|
|
||||||
|
|
||||||
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
|
||||||
|
|
||||||
packages.${name} = inputs.self.nixosConfigurations.${name}.config.system.build.images.kubevirt;
|
|
||||||
}
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
{
|
|
||||||
# NOTE: the args not used in this file CAN NOT be removed!
|
|
||||||
# because haumea pass argument lazily,
|
|
||||||
# and these arguments are used in the functions like `mylib.nixosSystem`, `mylib.colmenaSystem`, etc.
|
|
||||||
inputs,
|
|
||||||
lib,
|
|
||||||
mylib,
|
|
||||||
myvars,
|
|
||||||
system,
|
|
||||||
genSpecialArgs,
|
|
||||||
...
|
|
||||||
}@args:
|
|
||||||
let
|
|
||||||
name = "k3s-test-1-worker-2";
|
|
||||||
tags = [ name ];
|
|
||||||
ssh-user = "root";
|
|
||||||
|
|
||||||
modules = {
|
|
||||||
nixos-modules =
|
|
||||||
(map mylib.relativeToRoot [
|
|
||||||
# common
|
|
||||||
"secrets/nixos.nix"
|
|
||||||
"modules/nixos/server/server.nix"
|
|
||||||
"modules/nixos/server/qemu-guest-hardware-configuration.nix"
|
|
||||||
# host specific
|
|
||||||
"hosts/k8s/${name}"
|
|
||||||
])
|
|
||||||
++ [
|
|
||||||
{ modules.secrets.server.kubernetes.enable = true; }
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
systemArgs = modules // args;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
nixosConfigurations.${name} = mylib.nixosSystem systemArgs;
|
|
||||||
|
|
||||||
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
|
||||||
|
|
||||||
packages.${name} = inputs.self.nixosConfigurations.${name}.config.system.build.images.kubevirt;
|
|
||||||
}
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
{
|
|
||||||
# NOTE: the args not used in this file CAN NOT be removed!
|
|
||||||
# because haumea pass argument lazily,
|
|
||||||
# and these arguments are used in the functions like `mylib.nixosSystem`, `mylib.colmenaSystem`, etc.
|
|
||||||
inputs,
|
|
||||||
lib,
|
|
||||||
mylib,
|
|
||||||
myvars,
|
|
||||||
system,
|
|
||||||
genSpecialArgs,
|
|
||||||
...
|
|
||||||
}@args:
|
|
||||||
let
|
|
||||||
name = "k3s-test-1-worker-3";
|
|
||||||
tags = [ name ];
|
|
||||||
ssh-user = "root";
|
|
||||||
|
|
||||||
modules = {
|
|
||||||
nixos-modules =
|
|
||||||
(map mylib.relativeToRoot [
|
|
||||||
# common
|
|
||||||
"secrets/nixos.nix"
|
|
||||||
"modules/nixos/server/server.nix"
|
|
||||||
"modules/nixos/server/qemu-guest-hardware-configuration.nix"
|
|
||||||
# host specific
|
|
||||||
"hosts/k8s/${name}"
|
|
||||||
])
|
|
||||||
++ [
|
|
||||||
{ modules.secrets.server.kubernetes.enable = true; }
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
systemArgs = modules // args;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
nixosConfigurations.${name} = mylib.nixosSystem systemArgs;
|
|
||||||
|
|
||||||
colmena.${name} = mylib.colmenaSystem (systemArgs // { inherit tags ssh-user; });
|
|
||||||
|
|
||||||
packages.${name} = inputs.self.nixosConfigurations.${name}.config.system.build.images.kubevirt;
|
|
||||||
}
|
|
||||||
@@ -96,7 +96,7 @@ export def upload-vm [
|
|||||||
# node the VM is pinned to.
|
# node the VM is pinned to.
|
||||||
#
|
#
|
||||||
# Requires the CDI upload proxy to be reachable from here; it is exposed
|
# Requires the CDI upload proxy to be reachable from here; it is exposed
|
||||||
# as a NodePort by the cluster (see infra/configs/base/kubevirt).
|
# as a NodePort by the cluster.
|
||||||
let uploadProxyUrl = "https://192.168.5.181:30443"
|
let uploadProxyUrl = "https://192.168.5.181:30443"
|
||||||
let image = (glob result/nixos-image-*.qcow2 | first)
|
let image = (glob result/nixos-image-*.qcow2 | first)
|
||||||
virtctl image-upload dv $"($name)-disk" -n vms --size $size --image-path $image --uploadproxy-url $uploadProxyUrl --insecure
|
virtctl image-upload dv $"($name)-disk" -n vms --size $size --image-path $image --uploadproxy-url $uploadProxyUrl --insecure
|
||||||
|
|||||||
+1
-1
@@ -37,7 +37,7 @@ Comprehensive network configuration including:
|
|||||||
The networking configuration covers:
|
The networking configuration covers:
|
||||||
|
|
||||||
- **Physical machines**: Desktop PCs, Apple Silicon systems, SBCs
|
- **Physical machines**: Desktop PCs, Apple Silicon systems, SBCs
|
||||||
- **Virtual machines**: KubeVirt guests, K3s nodes
|
- **Virtual machines**: VM guests, K3s nodes
|
||||||
- **Kubernetes clusters**: Production and testing environments
|
- **Kubernetes clusters**: Production and testing environments
|
||||||
- **Network infrastructure**: Routers, gateways, and DNS configuration
|
- **Network infrastructure**: Routers, gateways, and DNS configuration
|
||||||
|
|
||||||
|
|||||||
+6
-6
@@ -105,32 +105,32 @@ rec {
|
|||||||
# Kubernetes Clusters
|
# Kubernetes Clusters
|
||||||
# ============================================
|
# ============================================
|
||||||
k3s-test-1-master-1 = {
|
k3s-test-1-master-1 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.114";
|
ipv4 = "192.168.5.114";
|
||||||
};
|
};
|
||||||
k3s-test-1-master-2 = {
|
k3s-test-1-master-2 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.115";
|
ipv4 = "192.168.5.115";
|
||||||
};
|
};
|
||||||
k3s-test-1-master-3 = {
|
k3s-test-1-master-3 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.116";
|
ipv4 = "192.168.5.116";
|
||||||
};
|
};
|
||||||
k3s-test-1-worker-1 = {
|
k3s-test-1-worker-1 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.111";
|
ipv4 = "192.168.5.111";
|
||||||
};
|
};
|
||||||
k3s-test-1-worker-2 = {
|
k3s-test-1-worker-2 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.112";
|
ipv4 = "192.168.5.112";
|
||||||
};
|
};
|
||||||
k3s-test-1-worker-3 = {
|
k3s-test-1-worker-3 = {
|
||||||
# KubeVirt VM
|
# VM
|
||||||
iface = "enp2s0";
|
iface = "enp2s0";
|
||||||
ipv4 = "192.168.5.113";
|
ipv4 = "192.168.5.113";
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in new issue
Block a user