docs(k8s-bootstrap): document control-plane VIP and pfSense HAProxy setup
The VIP and HAProxy frontend/backend aren't exposed by the pfsense provider, so they must be created by hand before this stack is deployed.
This commit is contained in:
1 parent
afac1f338f
commit
19bf92d6bc
1 file changed
+21
-5
@@ -7,12 +7,12 @@ Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts V
|
|||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
|
1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
|
||||||
2. Waits for port 22 to open on each VM (bash `/dev/tcp`)
|
2. Waits for port 22 to open on each VM (bash `/dev/tcp`), installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip> --tls-san <controlPlaneVipAddress>`
|
||||||
3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
|
3. Waits for `https://<controlPlaneVipAddress>:6443/healthz` to respond before anyone else joins, since every subsequent join uses the VIP, not `master1Ip`. The VIP itself is a pfSense IP Alias Virtual IP with HAProxy TCP-load-balancing 6443 across all three masters — set up manually, not managed by any Pulumi stack (see "pfSense one-time setup" below). This is deliberately *not* a second kube-vip/ARP-based VIP: `03-k8s-infra`'s existing kube-vip DaemonSet already handles Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the cluster, and API access is too critical to depend on the same ARP-failover mechanism
|
||||||
4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
|
4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes via `--server https://<controlPlaneVipAddress>:6443`
|
||||||
5. Joins `k3s-worker-1`, `k3s-worker-2`, and `k3s-worker-3` as agent nodes
|
5. Joins `k3s-worker-1`, `k3s-worker-2`, and `k3s-worker-3` as agent nodes via `K3S_URL=https://<controlPlaneVipAddress>:6443`
|
||||||
6. Formats the workers' dedicated `scsi1` disk (ext4, if not already formatted) and mounts it at `/mnt/longhorn-extra` (via `/etc/fstab`) — this is the disk `03-k8s-infra` later registers as an extra Longhorn disk, to keep Longhorn off the root filesystem and avoid `DiskPressure`
|
6. Formats the workers' dedicated `scsi1` disk (ext4, if not already formatted) and mounts it at `/mnt/longhorn-extra` (via `/etc/fstab`) — this is the disk `03-k8s-infra` later registers as an extra Longhorn disk, to keep Longhorn off the root filesystem and avoid `DiskPressure`
|
||||||
7. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL, and exports it as the secret stack output `kubeconfig`
|
7. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL to `controlPlaneVipAddress`, and exports it as the secret stack output `kubeconfig`
|
||||||
|
|
||||||
VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
|
VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
|
||||||
|
|
||||||
@@ -29,6 +29,11 @@ pulumi config set master3Ip "192.168.1.x"
|
|||||||
pulumi config set worker1Ip "192.168.1.x"
|
pulumi config set worker1Ip "192.168.1.x"
|
||||||
pulumi config set worker2Ip "192.168.1.x"
|
pulumi config set worker2Ip "192.168.1.x"
|
||||||
pulumi config set worker3Ip "192.168.1.x"
|
pulumi config set worker3Ip "192.168.1.x"
|
||||||
|
|
||||||
|
# Control-plane VIP — floating IP for API server access, fronted by pfSense
|
||||||
|
# + HAProxy (see "pfSense one-time setup" below). Must be a free address
|
||||||
|
# outside the router's DHCP pool.
|
||||||
|
pulumi config set --secret controlPlaneVipAddress "192.168.1.147"
|
||||||
```
|
```
|
||||||
|
|
||||||
Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `pve3Endpoint`, `pve3ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
|
Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `pve3Endpoint`, `pve3ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
|
||||||
@@ -37,6 +42,17 @@ Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiTok
|
|||||||
|
|
||||||
`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
|
`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
|
||||||
|
|
||||||
|
## pfSense one-time setup (before deploying)
|
||||||
|
|
||||||
|
The `@pulumi/pfsense` provider used by `01-proxmox-infra` doesn't expose Virtual IP or HAProxy resources (only DHCP static mappings, DNS overrides, and firewall aliases), so this is manual, not Pulumi-managed:
|
||||||
|
|
||||||
|
1. Confirm `controlPlaneVipAddress` falls outside the DHCP pool (Services → DHCP Server → Range).
|
||||||
|
2. Firewall → Virtual IPs → Add: Type = `IP Alias`, Interface = `LAN`, Address = `<controlPlaneVipAddress>/32`.
|
||||||
|
3. System → Package Manager → install `haproxy` if not already installed.
|
||||||
|
4. Services → HAProxy → Backend: add a pool (e.g. `k3s-apiservers`), mode `TCP`, health check `TCP check` (or `ssl-hello-chk`), servers = `<master1Ip>:6443`, `<master2Ip>:6443`, `<master3Ip>:6443`.
|
||||||
|
5. Services → HAProxy → Frontend: add a frontend listening on `<controlPlaneVipAddress>:6443`, type `TCP` (not HTTP — this must stay an opaque passthrough so clients validate k3s's own TLS cert, not one terminated at pfSense), default backend = the pool from step 4.
|
||||||
|
6. Services → HAProxy → Settings: enable HAProxy, apply.
|
||||||
|
|
||||||
## CI/CD
|
## CI/CD
|
||||||
|
|
||||||
Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.
|
Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.
|
||||||
Reference in new issue
Block a user