02-k8s-bootstrap/CLAUDE.md

@@ -7,12 +7,12 @@ Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts V
 ## How it works
 
 1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
-2. Waits for port 22 to open on each VM (bash `/dev/tcp`)
+2. Waits for port 22 to open on each VM (bash `/dev/tcp`), installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip> --tls-san <controlPlaneVipAddress>`
-3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
+3. Waits for `https://<controlPlaneVipAddress>:6443/healthz` to respond before anyone else joins, since every subsequent join uses the VIP, not `master1Ip`. The VIP itself is a pfSense IP Alias Virtual IP with HAProxy TCP-load-balancing 6443 across all three masters — set up manually, not managed by any Pulumi stack (see "pfSense one-time setup" below). This is deliberately *not* a second kube-vip/ARP-based VIP: `03-k8s-infra`'s existing kube-vip DaemonSet already handles Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the cluster, and API access is too critical to depend on the same ARP-failover mechanism
-4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
+4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes via `--server https://<controlPlaneVipAddress>:6443`
-5. Joins `k3s-worker-1`, `k3s-worker-2`, and `k3s-worker-3` as agent nodes
+5. Joins `k3s-worker-1`, `k3s-worker-2`, and `k3s-worker-3` as agent nodes via `K3S_URL=https://<controlPlaneVipAddress>:6443`
 6. Formats the workers' dedicated `scsi1` disk (ext4, if not already formatted) and mounts it at `/mnt/longhorn-extra` (via `/etc/fstab`) — this is the disk `03-k8s-infra` later registers as an extra Longhorn disk, to keep Longhorn off the root filesystem and avoid `DiskPressure`
-7. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL, and exports it as the secret stack output `kubeconfig`
+7. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL to `controlPlaneVipAddress`, and exports it as the secret stack output `kubeconfig`
 
 VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
 
@@ -29,6 +29,11 @@ pulumi config set master3Ip "192.168.1.x"
 pulumi config set worker1Ip "192.168.1.x"
 pulumi config set worker2Ip "192.168.1.x"
 pulumi config set worker3Ip "192.168.1.x"
+
+# Control-plane VIP — floating IP for API server access, fronted by pfSense
+# + HAProxy (see "pfSense one-time setup" below). Must be a free address
+# outside the router's DHCP pool.
+pulumi config set --secret controlPlaneVipAddress "192.168.1.147"
 ```
 
 Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `pve3Endpoint`, `pve3ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
@@ -37,6 +42,17 @@ Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiTok
 
 `proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
 
+## pfSense one-time setup (before deploying)
+
+The `@pulumi/pfsense` provider used by `01-proxmox-infra` doesn't expose Virtual IP or HAProxy resources (only DHCP static mappings, DNS overrides, and firewall aliases), so this is manual, not Pulumi-managed:
+
+1. Confirm `controlPlaneVipAddress` falls outside the DHCP pool (Services → DHCP Server → Range).
+2. Firewall → Virtual IPs → Add: Type = `IP Alias`, Interface = `LAN`, Address = `<controlPlaneVipAddress>/32`.
+3. System → Package Manager → install `haproxy` if not already installed.
+4. Services → HAProxy → Backend: add a pool (e.g. `k3s-apiservers`), mode `TCP`, health check `TCP check` (or `ssl-hello-chk`), servers = `<master1Ip>:6443`, `<master2Ip>:6443`, `<master3Ip>:6443`.
+5. Services → HAProxy → Frontend: add a frontend listening on `<controlPlaneVipAddress>:6443`, type `TCP` (not HTTP — this must stay an opaque passthrough so clients validate k3s's own TLS cert, not one terminated at pfSense), default backend = the pool from step 4.
+6. Services → HAProxy → Settings: enable HAProxy, apply.
+
 ## CI/CD
 
 Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.

02-k8s-bootstrap/index.ts

@@ -37,6 +37,13 @@ const worker3Ip = infraRef.requireOutput("worker3Ip");
 // Pre-shared k3s cluster token
 const k3sToken = config.requireSecret("k3sToken");
 
+// Control-plane VIP — floating IP for API server access, ARP-announced by
+// kube-vip on whichever master is alive. Added as a TLS SAN on master-1's
+// cert below, then used (instead of master1Ip) for every other node's join
+// URL and for the exported kubeconfig, so the cluster stays reachable
+// through the loss of any single master.
+const controlPlaneVipAddress = config.requireSecret("controlPlaneVipAddress");
+
 // VM IDs and CI runner SSH key — read from proxmox-infra stack outputs
 const vmIdsOutput = infraRef.requireOutput("vmIds") as pulumi.Output<
   Record<string, number>
@@ -122,7 +129,7 @@ const installMaster1 = new command.remote.Command(
   "install-k3s-master-1",
   {
     connection: conn(master1Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --cluster-init --tls-san ${master1Ip} --disable servicelb --node-name k3s-master-1`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --cluster-init --tls-san ${master1Ip} --tls-san ${controlPlaneVipAddress} --disable servicelb --node-name k3s-master-1`,
     triggers: [master1VmId],
   },
   { dependsOn: [waitMaster1Ssh] },
@@ -139,7 +146,30 @@ const waitK3sMaster1Ready = new command.remote.Command(
 );
 
 // ---------------------------------------------------------------------------
-// Step 3: Wait for SSH on remaining masters, join cluster
+// Step 3: Confirm the control-plane VIP is actually reachable before anyone
+// else joins, so master-2/master-3 and every worker join through the VIP
+// instead of master1Ip directly. The VIP itself is NOT managed here (or by
+// any Pulumi stack) — it's a pfSense IP Alias Virtual IP with HAProxy
+// TCP-load-balancing 6443 across all three masters, set up manually (see
+// "pfSense one-time setup" in this stack's CLAUDE.md). This intentionally
+// avoids a second kube-vip/ARP-based VIP for something as critical as API
+// access — 03-k8s-infra's existing kube-vip DaemonSet already handles
+// Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the
+// cluster.
+// ---------------------------------------------------------------------------
+
+const waitControlPlaneVipReady = new command.remote.Command(
+  "wait-control-plane-vip-ready",
+  {
+    connection: conn(master1Ip),
+    create: pulumi.interpolate`for i in $(seq 1 60); do curl -sk --max-time 3 https://${controlPlaneVipAddress}:6443/healthz && exit 0; sleep 5; done; exit 1`,
+    triggers: [master1VmId],
+  },
+  { dependsOn: [waitK3sMaster1Ready] },
+);
+
+// ---------------------------------------------------------------------------
+// Step 4: Wait for SSH on remaining masters, join cluster via the VIP
 // ---------------------------------------------------------------------------
 
 const waitMaster2Ssh = new command.local.Command(
@@ -166,24 +196,24 @@ const joinMaster2 = new command.remote.Command(
   "join-k3s-master-2",
   {
     connection: conn(master2Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${master1Ip}:6443 --disable servicelb --node-name k3s-master-2`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${controlPlaneVipAddress}:6443 --disable servicelb --node-name k3s-master-2`,
     triggers: [master2VmId],
   },
-  { dependsOn: [waitMaster2Ssh] },
+  { dependsOn: [waitMaster2Ssh, waitControlPlaneVipReady] },
 );
 
 const joinMaster3 = new command.remote.Command(
   "join-k3s-master-3",
   {
     connection: conn(master3Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${master1Ip}:6443 --disable servicelb --node-name k3s-master-3`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${controlPlaneVipAddress}:6443 --disable servicelb --node-name k3s-master-3`,
     triggers: [master3VmId],
   },
-  { dependsOn: [waitMaster3Ssh, joinMaster2] },
+  { dependsOn: [waitMaster3Ssh, waitControlPlaneVipReady, joinMaster2] },
 );
 
 // ---------------------------------------------------------------------------
-// Step 4: Join worker nodes
+// Step 5: Join worker nodes via the VIP
 // ---------------------------------------------------------------------------
 
 const waitWorker1Ssh = new command.local.Command(
@@ -220,7 +250,7 @@ const joinWorker1 = new command.remote.Command(
   "join-k3s-worker-1",
   {
     connection: conn(worker1Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${master1Ip}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-1`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-1`,
     triggers: [worker1VmId],
   },
   { dependsOn: [waitWorker1Ssh] },
@@ -230,7 +260,7 @@ const joinWorker2 = new command.remote.Command(
   "join-k3s-worker-2",
   {
     connection: conn(worker2Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${master1Ip}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-2`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-2`,
     triggers: [worker2VmId],
   },
   { dependsOn: [waitWorker2Ssh] },
@@ -240,14 +270,14 @@ const joinWorker3 = new command.remote.Command(
   "join-k3s-worker-3",
   {
     connection: conn(worker3Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${master1Ip}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-3`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-3`,
     triggers: [worker3VmId],
   },
   { dependsOn: [waitWorker3Ssh] },
 );
 
 // ---------------------------------------------------------------------------
-// Step 5: Format and mount Longhorn disks
+// Step 6: Format and mount Longhorn disks
 // ---------------------------------------------------------------------------
 
 const workers = [worker1Ip, worker2Ip, worker3Ip];
@@ -272,7 +302,7 @@ workers.forEach((worker, i) => {
 });
 
 // ---------------------------------------------------------------------------
-// Step 6: Read kubeconfig from master-1, patch server URL for external access
+// Step 7: Read kubeconfig from master-1, patch server URL to the VIP
 // ---------------------------------------------------------------------------
 
 const getKubeconfig = new command.remote.Command(
@@ -286,7 +316,7 @@ const getKubeconfig = new command.remote.Command(
 );
 
 // ---------------------------------------------------------------------------
-// Step 7: Install additional packages
+// Step 8: Install additional packages
 // ---------------------------------------------------------------------------
 
 const allVms = [
@@ -342,6 +372,6 @@ allVms.forEach(({ id, ip, name, readiness }) => {
 
 export const kubeconfig = pulumi.secret(
   pulumi
-    .all([getKubeconfig.stdout, master1Ip])
+    .all([getKubeconfig.stdout, controlPlaneVipAddress])
-    .apply(([kc, ip]) => kc.replaceAll("127.0.0.1", ip as string).trim()),
+    .apply(([kc, vip]) => kc.replaceAll("127.0.0.1", vip as string).trim()),
 );