docs: updated documentation to current progress
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 58s
Deploy k8s Apps / Pulumi Preview (push) Has been skipped
Deploy k8s Apps / Pulumi Deploy (push) Successful in 48s
Deploy Monitoring / Pulumi Preview (push) Has been skipped
Deploy Monitoring / Pulumi Deploy (push) Successful in 59s

This commit is contained in:
2026-06-11 22:36:20 +02:00
parent 6b3ed077ee
commit 13549a8d37
9 changed files with 290 additions and 270 deletions
+10 -57
View File
@@ -1,22 +1,22 @@
# k8s-bootstrap
Bootstraps a k3s cluster on the 5 Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` remote.Command.
This file provides guidance to Claude Code when working in `02-k8s-bootstrap`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` `remote.Command`.
## How it works
1. Starts all 5 VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
2. Waits for port 22 to open on each VM (bash `/dev/tcp`)
3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
5. Joins `k3s-worker-1` and `k3s-worker-2` as agent nodes
6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches server URL, exports as secret stack output
6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL, and exports it as the secret stack output `kubeconfig`
VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via `StackReference` — no manual setup needed for those.
VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
## Required Pulumi config
Run in this directory after `pulumi stack init dev`:
```bash
# Pre-shared k3s token — any strong random string
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
@@ -29,59 +29,12 @@ pulumi config set worker1Ip "192.168.1.x"
pulumi config set worker2Ip "192.168.1.x"
```
Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from the `proxmox-infra` stack outputs via StackReference — do **not** set them here.
## Deployment order
`proxmox-infra` must be deployed **before** k8s-bootstrap. The proxmox-infra stack exports the Proxmox credentials and VM IDs that k8s-bootstrap reads via StackReference. If proxmox-infra hasn't been re-deployed after its latest code changes, those outputs won't exist and k8s-bootstrap will fail.
In CI, both workflows run as previews on PRs (no deploy). Trigger `workflow_dispatch` on `deploy-proxmox-infra` first, then run k8s-bootstrap.
After setting config, re-encode `Pulumi.dev.yaml` and update the Gitea secret `K8S_BOOTSTRAP_DEV_YAML`:
```bash
base64 -w 0 Pulumi.dev.yaml
# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML
```
Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
## Prerequisites
`proxmox-infra` must be deployed first. On first `pulumi up`, it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to `~/.ssh/authorized_keys`) before k8s-bootstrap can SSH in.
`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
## After first run
## CI/CD
The `kubeconfig` stack output must be propagated to `k8s-infra` and `k8s-apps`:
```bash
KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
cd ../k8s-apps && pulumi config set --secret kubeconfig "$KUBECONFIG"
```
Then re-encode each `Pulumi.dev.yaml` and update the corresponding Gitea secrets
(`K8S_INFRA_DEV_YAML`, `K8S_APPS_DEV_YAML`):
```bash
base64 -w 0 k8s-infra/Pulumi.dev.yaml
base64 -w 0 k8s-apps/Pulumi.dev.yaml
```
## k8s-infra config changes (democratic-csi → NFS CSI)
The `k8s-infra` stack no longer needs `truenasApiKey` or `truenasDataset`.
Replace them with:
```bash
cd ../k8s-infra
pulumi config set truenasHost "192.168.1.x"
pulumi config set truenasNfsPath "/mnt/tank/k8s"
# Remove old keys if present:
pulumi config rm truenasApiKey
pulumi config rm truenasDataset
```
## TrueNAS one-time setup (before deploying k8s-infra)
1. Create a dataset: `tank/k8s`
2. Add an NFS share for that dataset
3. In Network → Allowed Networks, permit `192.168.1.0/24`
4. No API key required — the NFS CSI driver connects directly via NFS protocol
Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.