k8s-bootstrap

Bootstraps a k3s cluster on the 5 Proxmox VMs created by proxmox-infra. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using @pulumi/command remote.Command.

How it works

Starts all 5 VMs via POST /api2/json/nodes/{node}/qemu/{vmid}/status/start
Waits for port 22 to open on each VM (bash /dev/tcp)
Installs k3s on k3s-master-1 with --cluster-init --tls-san <master1Ip>
Joins k3s-master-2 and k3s-master-3 as embedded etcd nodes
Joins k3s-worker-1 and k3s-worker-2 as agent nodes
Reads /etc/rancher/k3s/k3s.yaml from master-1 via SSH, patches server URL, exports as secret stack output

VM IDs and the CI runner SSH private key are read automatically from the proxmox-infra stack output via StackReference — no manual setup needed for those.

Required Pulumi config

Run in this directory after pulumi stack init dev:

# Pre-shared k3s token — any strong random string
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"

# Node IPs — static DHCP leases from the router (not secrets)
pulumi config set master1Ip "192.168.1.x"
pulumi config set master2Ip "192.168.1.x"
pulumi config set master3Ip "192.168.1.x"
pulumi config set worker1Ip "192.168.1.x"
pulumi config set worker2Ip "192.168.1.x"

Proxmox credentials (pve1Endpoint, pve1ApiToken, pve2Endpoint, pve2ApiToken) are read automatically from the proxmox-infra stack outputs via StackReference — do not set them here.

Deployment order

proxmox-infra must be deployed before k8s-bootstrap. The proxmox-infra stack exports the Proxmox credentials and VM IDs that k8s-bootstrap reads via StackReference. If proxmox-infra hasn't been re-deployed after its latest code changes, those outputs won't exist and k8s-bootstrap will fail.

In CI, both workflows run as previews on PRs (no deploy). Trigger workflow_dispatch on deploy-proxmox-infra first, then run k8s-bootstrap.

After setting config, re-encode Pulumi.dev.yaml and update the Gitea secret K8S_BOOTSTRAP_DEV_YAML:

base64 -w 0 Pulumi.dev.yaml
# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML

Prerequisites

proxmox-infra must be deployed first. On first pulumi up, it generates an ed25519 SSH keypair via @pulumi/tls, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to ~/.ssh/authorized_keys) before k8s-bootstrap can SSH in.

After first run

The kubeconfig stack output must be propagated to k8s-infra and k8s-apps:

KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
cd ../k8s-apps  && pulumi config set --secret kubeconfig "$KUBECONFIG"

Then re-encode each Pulumi.dev.yaml and update the corresponding Gitea secrets (K8S_INFRA_DEV_YAML, K8S_APPS_DEV_YAML):

base64 -w 0 k8s-infra/Pulumi.dev.yaml
base64 -w 0 k8s-apps/Pulumi.dev.yaml

k8s-infra config changes (democratic-csi → NFS CSI)

The k8s-infra stack no longer needs truenasApiKey or truenasDataset. Replace them with:

cd ../k8s-infra
pulumi config set truenasHost "192.168.1.x"
pulumi config set truenasNfsPath "/mnt/tank/k8s"
# Remove old keys if present:
pulumi config rm truenasApiKey
pulumi config rm truenasDataset

TrueNAS one-time setup (before deploying k8s-infra)

Create a dataset: tank/k8s
Add an NFS share for that dataset
In Network → Allowed Networks, permit 192.168.1.0/24
No API key required — the NFS CSI driver connects directly via NFS protocol

3.6 KiB Raw Permalink Blame History