homelab-infrastructure-as-code/k8s-bootstrap/CLAUDE.md

k8s-bootstrap

Bootstraps a k3s cluster on the 5 Proxmox VMs created by proxmox-infra. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using @pulumi/command remote.Command.

How it works

1. Starts all 5 VMs via POST /api2/json/nodes/{node}/qemu/{vmid}/status/start
2. Waits for port 22 to open on each VM (nc -z)
3. Installs k3s on k3s-master-1 with --cluster-init --tls-san <master1Ip>
4. Joins k3s-master-2 and k3s-master-3 as embedded etcd nodes
5. Joins k3s-worker-1 and k3s-worker-2 as agent nodes
6. Reads /etc/rancher/k3s/k3s.yaml from master-1 via SSH, patches server URL, exports as secret stack output

VM IDs and the CI runner SSH private key are read automatically from the proxmox-infra stack output via StackReference — no manual setup needed for those.

Required Pulumi config

Run in this directory after pulumi stack init dev:

# Same as proxmox-infra
pulumi config set --secret pve1Endpoint "https://192.168.1.x:8006"
pulumi config set --secret pve1ApiToken "root@pam!pulumi=<token>"
pulumi config set --secret pve2Endpoint "https://192.168.1.y:8006"
pulumi config set --secret pve2ApiToken "root@pam!pulumi=<token>"

# Pre-shared k3s token — any strong random string
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"

# Node IPs — static DHCP leases from the router (not secrets)
pulumi config set master1Ip "192.168.1.x"
pulumi config set master2Ip "192.168.1.x"
pulumi config set master3Ip "192.168.1.x"
pulumi config set worker1Ip "192.168.1.x"
pulumi config set worker2Ip "192.168.1.x"

After setting config, re-encode Pulumi.dev.yaml and update the Gitea secret K8S_BOOTSTRAP_DEV_YAML:

base64 -w 0 Pulumi.dev.yaml
# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML

Prerequisites

proxmox-infra must be deployed first. On first pulumi up, it generates an ed25519 SSH keypair via @pulumi/tls, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to ~/.ssh/authorized_keys) before k8s-bootstrap can SSH in.

After first run

The kubeconfig stack output must be propagated to k8s-infra and k8s-apps:

KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
cd ../k8s-apps  && pulumi config set --secret kubeconfig "$KUBECONFIG"

Then re-encode each Pulumi.dev.yaml and update the corresponding Gitea secrets (K8S_INFRA_DEV_YAML, K8S_APPS_DEV_YAML):

base64 -w 0 k8s-infra/Pulumi.dev.yaml
base64 -w 0 k8s-apps/Pulumi.dev.yaml

k8s-infra config changes (democratic-csi → NFS CSI)

The k8s-infra stack no longer needs truenasApiKey or truenasDataset. Replace them with:

cd ../k8s-infra
pulumi config set truenasHost "192.168.1.x"
pulumi config set truenasNfsPath "/mnt/tank/k8s"
# Remove old keys if present:
pulumi config rm truenasApiKey
pulumi config rm truenasDataset

TrueNAS one-time setup (before deploying k8s-infra)

1. Create a dataset: tank/k8s
2. Add an NFS share for that dataset
3. In Network → Allowed Networks, permit 192.168.1.0/24
4. No API key required — the NFS CSI driver connects directly via NFS protocol