kasun
56f0b66f15
Deploy k8s Bootstrap / Bootstrap k3s Cluster (pull_request) Has been skipped
Deploy Proxmox Infra / Pulumi Preview (pull_request) Successful in 48s
Deploy Proxmox Infra / Pulumi Deploy (pull_request) Has been skipped
Deploy k8s Bootstrap / Pulumi Preview (pull_request) Failing after 59s
86 lines
3.3 KiB
Markdown
86 lines
3.3 KiB
Markdown
# k8s-bootstrap
|
|
|
|
Bootstraps a k3s cluster on the 5 Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` remote.Command.
|
|
|
|
## How it works
|
|
|
|
1. Starts all 5 VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
|
|
2. Waits for port 22 to open on each VM (`nc -z`)
|
|
3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
|
|
4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
|
|
5. Joins `k3s-worker-1` and `k3s-worker-2` as agent nodes
|
|
6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches server URL, exports as secret stack output
|
|
|
|
VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via `StackReference` — no manual setup needed for those.
|
|
|
|
## Required Pulumi config
|
|
|
|
Run in this directory after `pulumi stack init dev`:
|
|
|
|
```bash
|
|
# Same as proxmox-infra
|
|
pulumi config set --secret pve1Endpoint "https://192.168.1.x:8006"
|
|
pulumi config set --secret pve1ApiToken "root@pam!pulumi=<token>"
|
|
pulumi config set --secret pve2Endpoint "https://192.168.1.y:8006"
|
|
pulumi config set --secret pve2ApiToken "root@pam!pulumi=<token>"
|
|
|
|
# Pre-shared k3s token — any strong random string
|
|
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
|
|
|
|
# Node IPs — static DHCP leases from the router (not secrets)
|
|
pulumi config set master1Ip "192.168.1.x"
|
|
pulumi config set master2Ip "192.168.1.x"
|
|
pulumi config set master3Ip "192.168.1.x"
|
|
pulumi config set worker1Ip "192.168.1.x"
|
|
pulumi config set worker2Ip "192.168.1.x"
|
|
```
|
|
|
|
After setting config, re-encode `Pulumi.dev.yaml` and update the Gitea secret `K8S_BOOTSTRAP_DEV_YAML`:
|
|
```bash
|
|
base64 -w 0 Pulumi.dev.yaml
|
|
# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
`proxmox-infra` must be deployed first. On first `pulumi up`, it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to `~/.ssh/authorized_keys`) before k8s-bootstrap can SSH in.
|
|
|
|
## After first run
|
|
|
|
The `kubeconfig` stack output must be propagated to `k8s-infra` and `k8s-apps`:
|
|
|
|
```bash
|
|
KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
|
|
cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
|
|
cd ../k8s-apps && pulumi config set --secret kubeconfig "$KUBECONFIG"
|
|
```
|
|
|
|
Then re-encode each `Pulumi.dev.yaml` and update the corresponding Gitea secrets
|
|
(`K8S_INFRA_DEV_YAML`, `K8S_APPS_DEV_YAML`):
|
|
|
|
```bash
|
|
base64 -w 0 k8s-infra/Pulumi.dev.yaml
|
|
base64 -w 0 k8s-apps/Pulumi.dev.yaml
|
|
```
|
|
|
|
## k8s-infra config changes (democratic-csi → NFS CSI)
|
|
|
|
The `k8s-infra` stack no longer needs `truenasApiKey` or `truenasDataset`.
|
|
Replace them with:
|
|
|
|
```bash
|
|
cd ../k8s-infra
|
|
pulumi config set truenasHost "192.168.1.x"
|
|
pulumi config set truenasNfsPath "/mnt/tank/k8s"
|
|
# Remove old keys if present:
|
|
pulumi config rm truenasApiKey
|
|
pulumi config rm truenasDataset
|
|
```
|
|
|
|
## TrueNAS one-time setup (before deploying k8s-infra)
|
|
|
|
1. Create a dataset: `tank/k8s`
|
|
2. Add an NFS share for that dataset
|
|
3. In Network → Allowed Networks, permit `192.168.1.0/24`
|
|
4. No API key required — the NFS CSI driver connects directly via NFS protocol
|