homelab-infrastructure-as-code/02-k8s-bootstrap/CLAUDE.md

k8s-bootstrap

This file provides guidance to Claude Code when working in 02-k8s-bootstrap. See the repo-root CLAUDE.md for shared conventions, deployment order, and CI/CD.

Bootstraps a k3s cluster on the Proxmox VMs created by proxmox-infra. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using @pulumi/command remote.Command.

How it works

1. Starts all VMs via POST /api2/json/nodes/{node}/qemu/{vmid}/status/start
2. Waits for port 22 to open on each VM (bash /dev/tcp), installs k3s on k3s-master-1 with --cluster-init --tls-san <master1Ip> --tls-san <controlPlaneVipAddress>
3. Waits for https://<controlPlaneVipAddress>:6443/healthz to respond before anyone else joins, since every subsequent join uses the VIP, not master1Ip. The VIP itself is a pfSense IP Alias Virtual IP with HAProxy TCP-load-balancing 6443 across all three masters — set up manually, not managed by any Pulumi stack (see "pfSense one-time setup" below). This is deliberately not a second kube-vip/ARP-based VIP: 03-k8s-infra's existing kube-vip DaemonSet already handles Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the cluster, and API access is too critical to depend on the same ARP-failover mechanism
4. Joins k3s-master-2 and k3s-master-3 as embedded etcd nodes via --server https://<controlPlaneVipAddress>:6443
5. Joins k3s-worker-1, k3s-worker-2, and k3s-worker-3 as agent nodes via K3S_URL=https://<controlPlaneVipAddress>:6443
6. Formats the workers' dedicated scsi1 disk (ext4, if not already formatted) and mounts it at /mnt/longhorn-extra (via /etc/fstab) — this is the disk 03-k8s-infra later registers as an extra Longhorn disk, to keep Longhorn off the root filesystem and avoid DiskPressure
7. Reads /etc/rancher/k3s/k3s.yaml from master-1 via SSH, patches the server URL to controlPlaneVipAddress, and exports it as the secret stack output kubeconfig

VM IDs and the CI runner SSH private key are read automatically from the proxmox-infra stack output via StackReference — no manual setup needed for those.

Required Pulumi config

# Pre-shared k3s token — any strong random string
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"

# Node IPs — static DHCP leases from the router (not secrets)
pulumi config set master1Ip "192.168.1.x"
pulumi config set master2Ip "192.168.1.x"
pulumi config set master3Ip "192.168.1.x"
pulumi config set worker1Ip "192.168.1.x"
pulumi config set worker2Ip "192.168.1.x"
pulumi config set worker3Ip "192.168.1.x"

# Control-plane VIP — floating IP for API server access, fronted by pfSense
# + HAProxy (see "pfSense one-time setup" below). Must be a free address
# outside the router's DHCP pool.
pulumi config set --secret controlPlaneVipAddress "192.168.1.147"

Proxmox credentials (pve1Endpoint, pve1ApiToken, pve2Endpoint, pve2ApiToken, pve3Endpoint, pve3ApiToken) are read automatically from proxmox-infra via StackReference — do not set them here.

Prerequisites

proxmox-infra must be deployed first. On its first pulumi up it generates an ed25519 SSH keypair via @pulumi/tls, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to ~/.ssh/authorized_keys) before this stack can SSH in.

pfSense one-time setup (before deploying)

The @pulumi/pfsense provider used by 01-proxmox-infra doesn't expose Virtual IP or HAProxy resources (only DHCP static mappings, DNS overrides, and firewall aliases), so this is manual, not Pulumi-managed:

1. Confirm controlPlaneVipAddress falls outside the DHCP pool (Services → DHCP Server → Range).
2. Firewall → Virtual IPs → Add: Type = IP Alias, Interface = LAN, Address = <controlPlaneVipAddress>/32.
3. System → Package Manager → install haproxy if not already installed.
4. Services → HAProxy → Backend: add a pool (e.g. k3s-apiservers), mode TCP, health check TCP check (or ssl-hello-chk), servers = <master1Ip>:6443, <master2Ip>:6443, <master3Ip>:6443.
5. Services → HAProxy → Frontend: add a frontend listening on <controlPlaneVipAddress>:6443, type TCP (not HTTP — this must stay an opaque passthrough so clients validate k3s's own TLS cert, not one terminated at pfSense), default backend = the pool from step 4.
6. Services → HAProxy → Settings: enable HAProxy, apply.

CI/CD

Workflow: ../.gitea/workflows/02-deploy-k8s-bootstrap.yaml. Secret: K8S_BOOTSTRAP_PULUMI_DEV_YAML. Deploy is manual (workflow_dispatch) — re-running this stack restarts the bootstrap sequence against live VMs.