docs(k8s-bootstrap): document control-plane VIP and pfSense HAProxy setup

The VIP and HAProxy frontend/backend aren't exposed by the pfsense
provider, so they must be created by hand before this stack is deployed.

01-proxmox-infra/CLAUDE.md

@@ -1,62 +1,30 @@
-# CLAUDE.md
+# proxmox-infra
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+This file provides guidance to Claude Code when working in `01-proxmox-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
 
 ## Project Overview
 
-This is a Pulumi TypeScript project (`proxmox-infra`) for provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider. The stack name is `dev`.
+Pulumi TypeScript project provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider, plus pfSense DHCPv4 static mappings via a locally-bundled `@pulumi/pfsense` SDK.
 
 ## Common Commands
 
 ```bash
-# Install dependencies
+# Install deps and compile the bundled pfSense SDK
 npm install
 
-# Preview infrastructure changes
-pulumi preview
-
-# Sync Pulumi state with actual Proxmox state (run before up if resources were changed manually)
+# Sync Pulumi state with actual Proxmox state — run before `up` if VMs were
+# changed manually in the Proxmox GUI
 pulumi refresh --yes
 
-# Deploy infrastructure
 pulumi refresh --yes && pulumi up --yes
-
-# Destroy infrastructure
-pulumi destroy
-
-# View current stack outputs
-pulumi stack output
-
-# View stack config
-pulumi config
 ```
 
 ## Architecture
 
-- **Entry point**: `index.ts` — all Pulumi resources are declared here
-- **Provider**: `@muhlba91/pulumi-proxmoxve` v8.x — community Proxmox provider (not an official Pulumi provider)
-- **Stack**: `dev` — configured in `Pulumi.dev.yaml`
-- **Runtime**: Node.js with `npm`, TypeScript compiled to `bin/` (excluded from git)
+- **Entry point**: `index.ts` — all resources declared here.
+- **Providers**: two `@muhlba91/pulumi-proxmoxve` providers — `pveProvider` (node `pve`) and `pveBckpProvider` (node `pve-bckp`). Always pass the correct provider when adding resources.
+- pfSense SDK lives in `sdks/pfsense/` (gitignored, regenerated by `npm install` / CI via `pulumi package add terraform-provider marshallford/pfsense 0.22.0`).
 
-## CI/CD (Gitea Actions)
+## CI/CD
 
-Workflow file: `../.gitea/workflows/deploy-proxmox-infra.yaml`
-
-Triggers: push to `main` and pull requests targeting `main`, scoped to changes under `proxmox-infra/**` or `.gitea/workflows/**`.
-
-- **Pull request** → `pulumi preview` (no changes deployed)
-- **Push to main** → `pulumi refresh` then `pulumi up`
-
-Secrets required in Gitea (`Settings → Actions → Secrets`):
-
-- `PULUMI_BACKEND_URL` — PostgreSQL connection string for the self-hosted state backend
-- `PULUMI_CONFIG_PASSPHRASE` — passphrase used to decrypt secrets in `Pulumi.dev.yaml`
-- `PULUMI_DEV_YAML` — base64-encoded content of `Pulumi.dev.yaml`; update manually whenever the file changes: `base64 -w 0 Pulumi.dev.yaml`
-
-## Key Notes
-
-- Credentials for both Proxmox nodes are stored as encrypted secrets in `Pulumi.dev.yaml` and decrypted at runtime using `PULUMI_CONFIG_PASSPHRASE`. Do not pass Proxmox credentials via environment variables — the code uses `config.requireSecret()`.
-- There are two Proxmox providers: `pveProvider` (main node `pve`) and `pveBckpProvider` (backup node `pve-bckp`). Always pass the correct provider when adding resources.
-- `Pulumi.dev.yaml` contains the encryption salt — never delete it or secrets become unrecoverable.
-- TypeScript is compiled with strict mode, `nodenext` module resolution, and `noImplicitReturns` — all functions must have explicit return types when TypeScript cannot infer them.
-- Don't add a co-author when committing to git.
+Workflow: `../.gitea/workflows/01-deploy-proxmox-infra.yaml`. Secret: `PROXMOX_INFRA_PULUMI_DEV_YAML`. In addition to the shared pattern, this workflow regenerates the pfSense SDK (`pulumi package add terraform-provider marshallford/pfsense 0.22.0`) before preview/deploy.

01-proxmox-infra/index.ts

@@ -6,9 +6,12 @@ import * as pfsense from "@pulumi/pfsense";
 const config = new pulumi.Config();
 
 const pve1Endpoint = config.requireSecret("pve1Endpoint");
-const pve1ApiToken = config.requireSecret("pve1ApiToken");
 const pve2Endpoint = config.requireSecret("pve2Endpoint");
+const pve3Endpoint = config.requireSecret("pve3Endpoint");
+
+const pve1ApiToken = config.requireSecret("pve1ApiToken");
 const pve2ApiToken = config.requireSecret("pve2ApiToken");
+const pve3ApiToken = config.requireSecret("pve3ApiToken");
 
 const pfSenseUrl = config.requireSecret("pfSenseUrl");
 const pfSenseUser = config.requireSecret("pfSenseUser");
@@ -16,25 +19,127 @@ const pfSensePassword = config.requireSecret("pfSensePassword");
 
 const master1Ip = config.requireSecret("master1Ip");
 const master2Ip = config.requireSecret("master2Ip");
-const worker1Ip = config.requireSecret("worker1Ip");
 const master3Ip = config.requireSecret("master3Ip");
+
+const worker1Ip = config.requireSecret("worker1Ip");
 const worker2Ip = config.requireSecret("worker2Ip");
+const worker3Ip = config.requireSecret("worker3Ip");
+
+// ---------------------------------------------------------------------------
+// VM templates — one per node, cloned from the downloaded cloud image.
+// Templates are not started and have no cloud-init config; that is applied
+// per-clone so each node gets its own hostname (derived from VM name).
+// ---------------------------------------------------------------------------
+
+const templateSettings = {
+  template: true,
+  started: false,
+  stopOnDestroy: true,
+  scsiHardware: "virtio-scsi-pci",
+  cdrom: { fileId: "none" },
+  cpu: {
+    cores: 2,
+    sockets: 1,
+    type: "host",
+    numa: true,
+  },
+  memory: {
+    dedicated: 2048,
+    floating: 0,
+  },
+  networkDevices: [{ bridge: "vmbr0", model: "virtio" }],
+  serialDevices: [{}],
+  vga: { type: "serial0" },
+  agent: { enabled: true, timeout: "30s" },
+};
 
 // ---------------------------------------------------------------------------
 // Providers — one per standalone Proxmox machine
 // ---------------------------------------------------------------------------
 
-const pveProvider = new proxmox.Provider("pve", {
-  endpoint: pve1Endpoint,
-  apiToken: pve1ApiToken,
-  insecure: true,
-});
+interface PveNodeConfig {
+  nodeName: string;
+  hostLabel: string;
+  isoStore: string;
+  dataStore: string;
+  endpoint: pulumi.Output<string>;
+  apiToken: pulumi.Output<string>;
+}
 
-const pveBckpProvider = new proxmox.Provider("pve-bckp", {
-  endpoint: pve2Endpoint,
-  apiToken: pve2ApiToken,
-  insecure: true,
-});
+const pveNodeSeeds: PveNodeConfig[] = [
+  {
+    nodeName: "pve",
+    hostLabel: "pve-b550",
+    isoStore: "pve-local-ext1",
+    dataStore: "local-lvm",
+    endpoint: pve1Endpoint,
+    apiToken: pve1ApiToken,
+  },
+  {
+    nodeName: "pve-bckp",
+    hostLabel: "pve-optiplex",
+    isoStore: "local",
+    dataStore: "local",
+    endpoint: pve2Endpoint,
+    apiToken: pve2ApiToken,
+  },
+  {
+    nodeName: "pve",
+    hostLabel: "pve-esprimo",
+    isoStore: "local",
+    dataStore: "local-lvm",
+    endpoint: pve3Endpoint,
+    apiToken: pve3ApiToken,
+  },
+];
+
+const pveNodes = pveNodeSeeds
+  .map((n) => ({
+    ...n,
+    provider: new proxmox.Provider(n.hostLabel, {
+      endpoint: n.endpoint,
+      apiToken: n.apiToken,
+      insecure: true,
+    }),
+  }))
+  .map((n) => ({
+    ...n,
+    ubuntuImagePve: new proxmox.download.File(
+      `ubuntu-noble-${n.hostLabel}`,
+      {
+        nodeName: n.nodeName,
+        datastoreId: n.isoStore,
+        contentType: "import",
+        fileName: "noble-server-cloudimg-amd64.qcow2",
+        url: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img",
+        overwrite: false,
+        overwriteUnmanaged: true,
+      },
+      { provider: n.provider },
+    ),
+  }))
+  .map((n) => ({
+    ...n,
+    pveTemplate: new proxmox.VmLegacy(
+      `k3s-template-${n.hostLabel}`,
+      {
+        ...templateSettings,
+        nodeName: n.nodeName,
+        name: "k3s-ubuntu-noble-template",
+        disks: [
+          {
+            interface: "scsi0",
+            datastoreId: n.dataStore,
+            importFrom: pulumi.interpolate`${n.ubuntuImagePve.datastoreId}:import/${n.ubuntuImagePve.fileName}`,
+            size: 10,
+            ssd: true,
+            discard: "on",
+          },
+        ],
+      },
+      { provider: n.provider, ignoreChanges: ["disks"] },
+    ),
+  }));
 
 // ---------------------------------------------------------------------------
 // Providers — PfSense
@@ -56,173 +161,64 @@ const ciRunnerKey = new tls.PrivateKey("ci-runner-key", {
   algorithm: "ED25519",
 });
 
-// ---------------------------------------------------------------------------
-// Download Ubuntu Noble cloud image to each node's ISO storage
-// ---------------------------------------------------------------------------
-
-const ubuntuNobleUrl =
-  "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img";
-
-const ubuntuImagePve = new proxmox.download.File(
-  "ubuntu-noble-pve",
-  {
-    nodeName: "pve",
-    datastoreId: "pve-local-ext1",
-    contentType: "import",
-    fileName: "noble-server-cloudimg-amd64.qcow2",
-    url: ubuntuNobleUrl,
-    overwrite: false,
-    overwriteUnmanaged: true,
-  },
-  { provider: pveProvider },
-);
-
-const ubuntuImagePveBckp = new proxmox.download.File(
-  "ubuntu-noble-pve-bckp",
-  {
-    nodeName: "pve-bckp",
-    datastoreId: "local",
-    contentType: "import",
-    fileName: "noble-server-cloudimg-amd64.qcow2",
-    url: ubuntuNobleUrl,
-    overwrite: false,
-    overwriteUnmanaged: true,
-  },
-  { provider: pveBckpProvider },
-);
-
-// ---------------------------------------------------------------------------
-// VM templates — one per node, cloned from the downloaded cloud image.
-// Templates are not started and have no cloud-init config; that is applied
-// per-clone so each node gets its own hostname (derived from VM name).
-// ---------------------------------------------------------------------------
-
-const templateSettings = {
-  template: true,
-  started: false,
-  stopOnDestroy: true,
-  scsiHardware: "virtio-scsi-pci",
-  cpu: {
-    cores: 2,
-    sockets: 1,
-    type: "host",
-    numa: true,
-  },
-  memory: {
-    dedicated: 2048,
-    floating: 0,
-  },
-  networkDevices: [{ bridge: "vmbr0", model: "virtio" }],
-  serialDevices: [{}],
-  vga: { type: "serial0" },
-  agent: { enabled: true },
-};
-
-const pveTemplate = new proxmox.VmLegacy(
-  "k3s-template-pve",
-  {
-    ...templateSettings,
-    nodeName: "pve",
-    name: "k3s-ubuntu-noble-template",
-    disks: [
-      {
-        interface: "scsi0",
-        datastoreId: "local-lvm",
-        importFrom: pulumi.interpolate`${ubuntuImagePve.datastoreId}:import/${ubuntuImagePve.fileName}`,
-        size: 10,
-        ssd: true,
-        discard: "on",
-      },
-    ],
-  },
-  { provider: pveProvider, ignoreChanges: ["disks"] },
-);
-
-const pveBckpTemplate = new proxmox.VmLegacy(
-  "k3s-template-pve-bckp",
-  {
-    ...templateSettings,
-    nodeName: "pve-bckp",
-    name: "k3s-ubuntu-noble-template",
-    disks: [
-      {
-        interface: "scsi0",
-        datastoreId: "local",
-        importFrom: pulumi.interpolate`${ubuntuImagePveBckp.datastoreId}:import/${ubuntuImagePveBckp.fileName}`,
-        size: 10,
-        ssd: true,
-        discard: "on",
-      },
-    ],
-  },
-  { provider: pveBckpProvider, ignoreChanges: ["disks"] },
-);
-
 // ---------------------------------------------------------------------------
 // k3s nodes — full clones of their respective templates
 // ---------------------------------------------------------------------------
 
 const k3sVmPassword = config.requireSecret("k3sVmPassword");
-const sshPvePublicKey = config.requireSecret("sshPvePublicKey");
+const pve1SshPublicKey = config.requireSecret("pve1SshPublicKey");
+const pve2SshPublicKey = config.requireSecret("pve2SshPublicKey");
+const pve3SshPublicKey = config.requireSecret("pve3SshPublicKey");
 
 interface NodeConfig {
   name: string;
   role: "master" | "worker";
-  nodeName: string;
-  provider: proxmox.Provider;
-  template: proxmox.VmLegacy;
-  diskDatastore: string;
+  host: (typeof pveNodes)[number]; // one row from the pveNodes chain
   ip: pulumi.Output<string>;
   longhornDiskSize?: number;
 }
 
+const hostFor = (label: string) => pveNodes.find((n) => n.hostLabel === label)!;
+
 const nodeConfigs: NodeConfig[] = [
   {
     name: "k3s-master-1",
     role: "master",
-    nodeName: "pve",
-    provider: pveProvider,
-    template: pveTemplate,
-    diskDatastore: "local-lvm",
+    host: hostFor("pve-b550"),
     ip: master1Ip,
   },
   {
     name: "k3s-master-2",
     role: "master",
-    nodeName: "pve",
-    provider: pveProvider,
-    template: pveTemplate,
-    diskDatastore: "local-lvm",
+    host: hostFor("pve-esprimo"),
     ip: master2Ip,
   },
   {
     name: "k3s-worker-1",
     role: "worker",
-    nodeName: "pve",
-    provider: pveProvider,
-    template: pveTemplate,
-    diskDatastore: "local-lvm",
+    host: hostFor("pve-b550"),
     ip: worker1Ip,
     longhornDiskSize: 50,
   },
   {
     name: "k3s-master-3",
     role: "master",
-    nodeName: "pve-bckp",
-    provider: pveBckpProvider,
-    template: pveBckpTemplate,
-    diskDatastore: "local",
+    host: hostFor("pve-optiplex"),
     ip: master3Ip,
   },
   {
     name: "k3s-worker-2",
     role: "worker",
-    nodeName: "pve-bckp",
-    provider: pveBckpProvider,
-    template: pveBckpTemplate,
-    diskDatastore: "local",
+    host: hostFor("pve-optiplex"),
     ip: worker2Ip,
-    longhornDiskSize: 35,
+    longhornDiskSize: 50,
+  },
+  {
+    name: "k3s-worker-3",
+    role: "worker",
+    host: hostFor("pve-esprimo"),
+    ip: worker3Ip,
+    longhornDiskSize: 50,
   },
 ];
 
@@ -231,14 +227,14 @@ const k3sVms = nodeConfigs.map(
     new proxmox.VmLegacy(
       node.name,
       {
-        nodeName: node.nodeName,
+        nodeName: node.host.nodeName,
         name: node.name,
         description: "k3s " + node.role + " node — managed by Pulumi",
         tags: ["k3s", node.role],
         clone: {
-          vmId: node.template.vmId,
+          vmId: node.host.pveTemplate.vmId,
           full: true,
-          datastoreId: node.diskDatastore,
+          datastoreId: node.host.dataStore,
         },
         cpu: {
           cores: 2,
@@ -253,7 +249,7 @@ const k3sVms = nodeConfigs.map(
         disks: [
           {
             interface: "scsi0",
-            datastoreId: node.diskDatastore,
+            datastoreId: node.host.dataStore,
             size: 35,
             ssd: true,
             discard: "on",
@@ -263,7 +259,7 @@ const k3sVms = nodeConfigs.map(
             : [
                 {
                   interface: "scsi1",
-                  datastoreId: node.diskDatastore,
+                  datastoreId: node.host.dataStore,
                   size: node.longhornDiskSize,
                   ssd: true,
                   discard: "on",
@@ -271,13 +267,15 @@ const k3sVms = nodeConfigs.map(
               ]),
         ],
         initialization: {
-          datastoreId: node.diskDatastore,
+          datastoreId: node.host.dataStore,
           ipConfigs: [{ ipv4: { address: "dhcp" } }],
           userAccount: {
             username: "ubuntu",
             password: k3sVmPassword,
             keys: [
-              sshPvePublicKey.apply((k) => k.trim()),
+              pve1SshPublicKey.apply((k) => k.trim()),
+              pve2SshPublicKey.apply((k) => k.trim()),
+              pve3SshPublicKey.apply((k) => k.trim()),
               ciRunnerKey.publicKeyOpenssh.apply((k) => k.trim()),
             ],
           },
@@ -286,12 +284,13 @@ const k3sVms = nodeConfigs.map(
         scsiHardware: "virtio-scsi-pci",
         serialDevices: [{}],
         vga: { type: "serial0" },
-        agent: { enabled: true },
+        agent: { enabled: true, timeout: "30s" },
+        cdrom: { fileId: "none" },
         started: false,
         stopOnDestroy: true,
       },
       {
-        provider: node.provider,
+        provider: node.host.provider,
         retainOnDelete: true,
         ignoreChanges: ["clone", "started"],
       },
@@ -303,7 +302,7 @@ k3sVms.forEach((vmResource, i) => {
     (nic) => nic[0].macAddress,
   );
 
-  return new pfsense.Dhcpv4Staticmapping(
+  new pfsense.Dhcpv4Staticmapping(
     `${nodeConfigs[i].name}-dhcp`,
     {
       interface: "lan",
@@ -315,21 +314,25 @@ k3sVms.forEach((vmResource, i) => {
   );
 });
 
-// Individual vmId exports — used by k8s-bootstrap to start VMs.
-// Order matches nodeConfigs: master-1, master-2, worker-1, master-3, worker-2.
 export const vmIds = {
   master1: k3sVms[0].vmId,
   master2: k3sVms[1].vmId,
   worker1: k3sVms[2].vmId,
   master3: k3sVms[3].vmId,
   worker2: k3sVms[4].vmId,
+  worker3: k3sVms[5].vmId,
 };
 
 // CI runner SSH private key — consumed by k8s-bootstrap via StackReference.
 export const ciRunnerPrivateKey = pulumi.secret(ciRunnerKey.privateKeyOpenssh);
 
-// Proxmox API credentials — consumed by k8s-bootstrap via StackReference.
-export { pve1Endpoint, pve1ApiToken, pve2Endpoint, pve2ApiToken };
+export {
+  pve1Endpoint,
+  pve1ApiToken,
+  pve2Endpoint,
+  pve2ApiToken,
+  pve3Endpoint,
+  pve3ApiToken,
+};
 
-//k3s instance ips consumed by k8s-bootstrap.
-export { master1Ip, master2Ip, worker1Ip, master3Ip, worker2Ip };
+export { master1Ip, master2Ip, worker1Ip, master3Ip, worker2Ip, worker3Ip };

01-proxmox-infra/package-lock.json

@@ -8,12 +8,12 @@
             "dependencies": {
                 "@muhlba91/pulumi-proxmoxve": "^8.2.1",
                 "@pulumi/pfsense": "file:sdks/pfsense",
-                "@pulumi/pulumi": "^3.113.0",
+                "@pulumi/pulumi": "^3.244.0",
                 "@pulumi/tls": "^5.5.0"
             },
             "devDependencies": {
-                "@types/node": "^18",
-                "typescript": "^5.0.0"
+                "@types/node": "^24",
+                "typescript": "^6.0.0"
             }
         },
         "node_modules/@gar/promise-retry": {
@@ -709,9 +709,9 @@
             "link": true
         },
         "node_modules/@pulumi/pulumi": {
-            "version": "3.243.0",
-            "resolved": "https://registry.npmjs.org/@pulumi/pulumi/-/pulumi-3.243.0.tgz",
-            "integrity": "sha512-KRIBErgCj9+gsEG7fOo3xRvvhq3VGw8gnzVpbFiyW2Zqr84MCIKDDVmj4MZDt/61KOn3OapI4ia1/6dSg3qwzQ==",
+            "version": "3.245.0",
+            "resolved": "https://registry.npmjs.org/@pulumi/pulumi/-/pulumi-3.245.0.tgz",
+            "integrity": "sha512-cRfyKlaoKThewFTRFdvLFSo33FLHhViYQG1xI1WBTS4mrjhJegWl28HmZupVEIQh8aqvBN0Cy0fFNbxbZnOzmA==",
             "license": "Apache-2.0",
             "dependencies": {
                 "@grpc/grpc-js": "^1.10.1",
@@ -870,12 +870,12 @@
             "license": "MIT"
         },
         "node_modules/@types/node": {
-            "version": "18.19.130",
-            "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz",
-            "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==",
+            "version": "24.13.1",
+            "resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.1.tgz",
+            "integrity": "sha512-RSpUJGmvsJ1ZeBehQZFhIdpsz+bIpES0nIQXko4Ybq+N+kX6XvOq3Jo+iJ82FWLdblFq85AsMikd3m35jgezYg==",
             "license": "MIT",
             "dependencies": {
-                "undici-types": "~5.26.4"
+                "undici-types": "~7.18.0"
             }
         },
         "node_modules/@types/semver": {
@@ -2497,9 +2497,9 @@
             }
         },
         "node_modules/typescript": {
-            "version": "5.9.3",
-            "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
-            "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
+            "version": "6.0.3",
+            "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
+            "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
             "devOptional": true,
             "license": "Apache-2.0",
             "bin": {
@@ -2520,9 +2520,9 @@
             }
         },
         "node_modules/undici-types": {
-            "version": "5.26.5",
-            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz",
-            "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==",
+            "version": "7.18.2",
+            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
+            "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
             "license": "MIT"
         },
         "node_modules/upath": {

01-proxmox-infra/package.json

@@ -2,13 +2,13 @@
     "name": "proxmox-infra",
     "main": "index.ts",
     "devDependencies": {
-        "@types/node": "^18",
-        "typescript": "^5.0.0"
+        "@types/node": "^24",
+        "typescript": "^6.0.0"
     },
     "dependencies": {
         "@muhlba91/pulumi-proxmoxve": "^8.2.1",
         "@pulumi/pfsense": "file:sdks/pfsense",
-        "@pulumi/pulumi": "^3.113.0",
+        "@pulumi/pulumi": "^3.244.0",
         "@pulumi/tls": "^5.5.0"
     },
     "imports": {

01-proxmox-infra/tsconfig.json

@@ -2,7 +2,7 @@
     "compilerOptions": {
         "strict": true,
         "outDir": "bin",
-        "target": "es2020",
+        "target": "es2024",
         "module": "nodenext",
         "moduleResolution": "nodenext",
         "sourceMap": true,

02-k8s-bootstrap/CLAUDE.md

@@ -1,22 +1,23 @@
 # k8s-bootstrap
 
-Bootstraps a k3s cluster on the 5 Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` remote.Command.
+This file provides guidance to Claude Code when working in `02-k8s-bootstrap`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` `remote.Command`.
 
 ## How it works
 
-1. Starts all 5 VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
-2. Waits for port 22 to open on each VM (bash `/dev/tcp`)
-3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
-4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
-5. Joins `k3s-worker-1` and `k3s-worker-2` as agent nodes
-6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches server URL, exports as secret stack output
+1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
+2. Waits for port 22 to open on each VM (bash `/dev/tcp`), installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip> --tls-san <controlPlaneVipAddress>`
+3. Waits for `https://<controlPlaneVipAddress>:6443/healthz` to respond before anyone else joins, since every subsequent join uses the VIP, not `master1Ip`. The VIP itself is a pfSense IP Alias Virtual IP with HAProxy TCP-load-balancing 6443 across all three masters — set up manually, not managed by any Pulumi stack (see "pfSense one-time setup" below). This is deliberately *not* a second kube-vip/ARP-based VIP: `03-k8s-infra`'s existing kube-vip DaemonSet already handles Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the cluster, and API access is too critical to depend on the same ARP-failover mechanism
+4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes via `--server https://<controlPlaneVipAddress>:6443`
+5. Joins `k3s-worker-1`, `k3s-worker-2`, and `k3s-worker-3` as agent nodes via `K3S_URL=https://<controlPlaneVipAddress>:6443`
+6. Formats the workers' dedicated `scsi1` disk (ext4, if not already formatted) and mounts it at `/mnt/longhorn-extra` (via `/etc/fstab`) — this is the disk `03-k8s-infra` later registers as an extra Longhorn disk, to keep Longhorn off the root filesystem and avoid `DiskPressure`
+7. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL to `controlPlaneVipAddress`, and exports it as the secret stack output `kubeconfig`
 
-VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via `StackReference` — no manual setup needed for those.
+VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
 
 ## Required Pulumi config
 
-Run in this directory after `pulumi stack init dev`:
-
 ```bash
 # Pre-shared k3s token — any strong random string
 pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
@@ -27,61 +28,31 @@ pulumi config set master2Ip "192.168.1.x"
 pulumi config set master3Ip "192.168.1.x"
 pulumi config set worker1Ip "192.168.1.x"
 pulumi config set worker2Ip "192.168.1.x"
+pulumi config set worker3Ip "192.168.1.x"
+
+# Control-plane VIP — floating IP for API server access, fronted by pfSense
+# + HAProxy (see "pfSense one-time setup" below). Must be a free address
+# outside the router's DHCP pool.
+pulumi config set --secret controlPlaneVipAddress "192.168.1.147"
 ```
 
-Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from the `proxmox-infra` stack outputs via StackReference — do **not** set them here.
-
-## Deployment order
-
-`proxmox-infra` must be deployed **before** k8s-bootstrap. The proxmox-infra stack exports the Proxmox credentials and VM IDs that k8s-bootstrap reads via StackReference. If proxmox-infra hasn't been re-deployed after its latest code changes, those outputs won't exist and k8s-bootstrap will fail.
-
-In CI, both workflows run as previews on PRs (no deploy). Trigger `workflow_dispatch` on `deploy-proxmox-infra` first, then run k8s-bootstrap.
-
-After setting config, re-encode `Pulumi.dev.yaml` and update the Gitea secret `K8S_BOOTSTRAP_DEV_YAML`:
-```bash
-base64 -w 0 Pulumi.dev.yaml
-# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML
-```
+Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `pve3Endpoint`, `pve3ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
 
 ## Prerequisites
 
-`proxmox-infra` must be deployed first. On first `pulumi up`, it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to `~/.ssh/authorized_keys`) before k8s-bootstrap can SSH in.
+`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
 
-## After first run
+## pfSense one-time setup (before deploying)
 
-The `kubeconfig` stack output must be propagated to `k8s-infra` and `k8s-apps`:
+The `@pulumi/pfsense` provider used by `01-proxmox-infra` doesn't expose Virtual IP or HAProxy resources (only DHCP static mappings, DNS overrides, and firewall aliases), so this is manual, not Pulumi-managed:
 
-```bash
-KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
-cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
-cd ../k8s-apps  && pulumi config set --secret kubeconfig "$KUBECONFIG"
-```
+1. Confirm `controlPlaneVipAddress` falls outside the DHCP pool (Services → DHCP Server → Range).
+2. Firewall → Virtual IPs → Add: Type = `IP Alias`, Interface = `LAN`, Address = `<controlPlaneVipAddress>/32`.
+3. System → Package Manager → install `haproxy` if not already installed.
+4. Services → HAProxy → Backend: add a pool (e.g. `k3s-apiservers`), mode `TCP`, health check `TCP check` (or `ssl-hello-chk`), servers = `<master1Ip>:6443`, `<master2Ip>:6443`, `<master3Ip>:6443`.
+5. Services → HAProxy → Frontend: add a frontend listening on `<controlPlaneVipAddress>:6443`, type `TCP` (not HTTP — this must stay an opaque passthrough so clients validate k3s's own TLS cert, not one terminated at pfSense), default backend = the pool from step 4.
+6. Services → HAProxy → Settings: enable HAProxy, apply.
 
-Then re-encode each `Pulumi.dev.yaml` and update the corresponding Gitea secrets
-(`K8S_INFRA_DEV_YAML`, `K8S_APPS_DEV_YAML`):
+## CI/CD
 
-```bash
-base64 -w 0 k8s-infra/Pulumi.dev.yaml
-base64 -w 0 k8s-apps/Pulumi.dev.yaml
-```
-
-## k8s-infra config changes (democratic-csi → NFS CSI)
-
-The `k8s-infra` stack no longer needs `truenasApiKey` or `truenasDataset`.
-Replace them with:
-
-```bash
-cd ../k8s-infra
-pulumi config set truenasHost "192.168.1.x"
-pulumi config set truenasNfsPath "/mnt/tank/k8s"
-# Remove old keys if present:
-pulumi config rm truenasApiKey
-pulumi config rm truenasDataset
-```
-
-## TrueNAS one-time setup (before deploying k8s-infra)
-
-1. Create a dataset: `tank/k8s`
-2. Add an NFS share for that dataset
-3. In Network → Allowed Networks, permit `192.168.1.0/24`
-4. No API key required — the NFS CSI driver connects directly via NFS protocol
+Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.

02-k8s-bootstrap/index.ts

@@ -3,12 +3,10 @@ import * as command from "@pulumi/command";
 
 const config = new pulumi.Config();
 
-//fetch credentials from proxmox-infra
 const infraRef = new pulumi.StackReference(
   `${pulumi.getOrganization()}/proxmox-infra/dev`,
 );
 
-// Proxmox API credentials — same as proxmox-infra stack
 const pve1Endpoint = infraRef.requireOutput(
   "pve1Endpoint",
 ) as pulumi.Output<string>;
@@ -21,6 +19,12 @@ const pve2Endpoint = infraRef.requireOutput(
 const pve2ApiToken = infraRef.requireOutput(
   "pve2ApiToken",
 ) as pulumi.Output<string>;
+const pve3Endpoint = infraRef.requireOutput(
+  "pve3Endpoint",
+) as pulumi.Output<string>;
+const pve3ApiToken = infraRef.requireOutput(
+  "pve3ApiToken",
+) as pulumi.Output<string>;
 
 // Node IPs — static DHCP leases set in the router
 const master1Ip = infraRef.requireOutput("master1Ip");
@@ -28,10 +32,18 @@ const master2Ip = infraRef.requireOutput("master2Ip");
 const master3Ip = infraRef.requireOutput("master3Ip");
 const worker1Ip = infraRef.requireOutput("worker1Ip");
 const worker2Ip = infraRef.requireOutput("worker2Ip");
+const worker3Ip = infraRef.requireOutput("worker3Ip");
 
 // Pre-shared k3s cluster token
 const k3sToken = config.requireSecret("k3sToken");
 
+// Control-plane VIP — floating IP for API server access, ARP-announced by
+// kube-vip on whichever master is alive. Added as a TLS SAN on master-1's
+// cert below, then used (instead of master1Ip) for every other node's join
+// URL and for the exported kubeconfig, so the cluster stays reachable
+// through the loss of any single master.
+const controlPlaneVipAddress = config.requireSecret("controlPlaneVipAddress");
+
 // VM IDs and CI runner SSH key — read from proxmox-infra stack outputs
 const vmIdsOutput = infraRef.requireOutput("vmIds") as pulumi.Output<
   Record<string, number>
@@ -45,6 +57,7 @@ const master2VmId = vmIdsOutput.apply((ids) => String(ids.master2));
 const master3VmId = vmIdsOutput.apply((ids) => String(ids.master3));
 const worker1VmId = vmIdsOutput.apply((ids) => String(ids.worker1));
 const worker2VmId = vmIdsOutput.apply((ids) => String(ids.worker2));
+const worker3VmId = vmIdsOutput.apply((ids) => String(ids.worker3));
 
 // SSH connection helper
 function conn(
@@ -83,6 +96,11 @@ const startWorker2 = new command.local.Command("start-worker-2", {
   triggers: [worker2VmId],
   interpreter: ["/bin/bash", "-c"],
 });
+const startWorker3 = new command.local.Command("start-worker-3", {
+  create: pulumi.interpolate`curl -sf -k -X POST -H "Authorization: PVEAPIToken=${pve3ApiToken}" "${pve3Endpoint}/api2/json/nodes/pve/qemu/${worker3VmId}/status/start" 2>/dev/null || true`,
+  triggers: [worker3VmId],
+  interpreter: ["/bin/bash", "-c"],
+});
 
 const allStarts = [
   startMaster1,
@@ -90,6 +108,7 @@ const allStarts = [
   startMaster3,
   startWorker1,
   startWorker2,
+  startWorker3,
 ];
 
 // ---------------------------------------------------------------------------
@@ -110,7 +129,7 @@ const installMaster1 = new command.remote.Command(
   "install-k3s-master-1",
   {
     connection: conn(master1Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --cluster-init --tls-san ${master1Ip} --disable servicelb --node-name k3s-master-1`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --cluster-init --tls-san ${master1Ip} --tls-san ${controlPlaneVipAddress} --disable servicelb --node-name k3s-master-1`,
     triggers: [master1VmId],
   },
   { dependsOn: [waitMaster1Ssh] },
@@ -127,7 +146,30 @@ const waitK3sMaster1Ready = new command.remote.Command(
 );
 
 // ---------------------------------------------------------------------------
-// Step 3: Wait for SSH on remaining masters, join cluster
+// Step 3: Confirm the control-plane VIP is actually reachable before anyone
+// else joins, so master-2/master-3 and every worker join through the VIP
+// instead of master1Ip directly. The VIP itself is NOT managed here (or by
+// any Pulumi stack) — it's a pfSense IP Alias Virtual IP with HAProxy
+// TCP-load-balancing 6443 across all three masters, set up manually (see
+// "pfSense one-time setup" in this stack's CLAUDE.md). This intentionally
+// avoids a second kube-vip/ARP-based VIP for something as critical as API
+// access — 03-k8s-infra's existing kube-vip DaemonSet already handles
+// Service/LoadBalancer VIPs (e.g. Traefik) separately, entirely inside the
+// cluster.
+// ---------------------------------------------------------------------------
+
+const waitControlPlaneVipReady = new command.remote.Command(
+  "wait-control-plane-vip-ready",
+  {
+    connection: conn(master1Ip),
+    create: pulumi.interpolate`for i in $(seq 1 60); do curl -sk --max-time 3 https://${controlPlaneVipAddress}:6443/healthz && exit 0; sleep 5; done; exit 1`,
+    triggers: [master1VmId],
+  },
+  { dependsOn: [waitK3sMaster1Ready] },
+);
+
+// ---------------------------------------------------------------------------
+// Step 4: Wait for SSH on remaining masters, join cluster via the VIP
 // ---------------------------------------------------------------------------
 
 const waitMaster2Ssh = new command.local.Command(
@@ -154,24 +196,24 @@ const joinMaster2 = new command.remote.Command(
   "join-k3s-master-2",
   {
     connection: conn(master2Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${master1Ip}:6443 --disable servicelb --node-name k3s-master-2`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${controlPlaneVipAddress}:6443 --disable servicelb --node-name k3s-master-2`,
     triggers: [master2VmId],
   },
-  { dependsOn: [waitMaster2Ssh] },
+  { dependsOn: [waitMaster2Ssh, waitControlPlaneVipReady] },
 );
 
 const joinMaster3 = new command.remote.Command(
   "join-k3s-master-3",
   {
     connection: conn(master3Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${master1Ip}:6443 --disable servicelb --node-name k3s-master-3`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_TOKEN='${k3sToken}' sh -s - server --server https://${controlPlaneVipAddress}:6443 --disable servicelb --node-name k3s-master-3`,
     triggers: [master3VmId],
   },
-  { dependsOn: [waitMaster3Ssh, joinMaster2] },
+  { dependsOn: [waitMaster3Ssh, waitControlPlaneVipReady, joinMaster2] },
 );
 
 // ---------------------------------------------------------------------------
-// Step 4: Join worker nodes
+// Step 5: Join worker nodes via the VIP
 // ---------------------------------------------------------------------------
 
 const waitWorker1Ssh = new command.local.Command(
@@ -194,11 +236,21 @@ const waitWorker2Ssh = new command.local.Command(
   { dependsOn: [joinMaster3] },
 );
 
+const waitWorker3Ssh = new command.local.Command(
+  "wait-ssh-worker-3",
+  {
+    create: pulumi.interpolate`for i in $(seq 1 60); do (timeout 5 bash -c "echo > /dev/tcp/${worker3Ip}/22") 2>/dev/null && exit 0; sleep 5; done; exit 1`,
+    triggers: [worker3VmId],
+    interpreter: ["/bin/bash", "-c"],
+  },
+  { dependsOn: [joinMaster3] },
+);
+
 const joinWorker1 = new command.remote.Command(
   "join-k3s-worker-1",
   {
     connection: conn(worker1Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${master1Ip}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-1`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-1`,
     triggers: [worker1VmId],
   },
   { dependsOn: [waitWorker1Ssh] },
@@ -208,14 +260,49 @@ const joinWorker2 = new command.remote.Command(
   "join-k3s-worker-2",
   {
     connection: conn(worker2Ip),
-    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${master1Ip}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-2`,
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-2`,
     triggers: [worker2VmId],
   },
   { dependsOn: [waitWorker2Ssh] },
 );
 
+const joinWorker3 = new command.remote.Command(
+  "join-k3s-worker-3",
+  {
+    connection: conn(worker3Ip),
+    create: pulumi.interpolate`curl -sfL https://get.k3s.io | sudo K3S_URL=https://${controlPlaneVipAddress}:6443 K3S_TOKEN='${k3sToken}' sh -s - --node-name k3s-worker-3`,
+    triggers: [worker3VmId],
+  },
+  { dependsOn: [waitWorker3Ssh] },
+);
+
 // ---------------------------------------------------------------------------
-// Step 5: Read kubeconfig from master-1, patch server URL for external access
+// Step 6: Format and mount Longhorn disks
+// ---------------------------------------------------------------------------
+
+const workers = [worker1Ip, worker2Ip, worker3Ip];
+const workerSshReader = [waitWorker1Ssh, waitWorker2Ssh, waitWorker3Ssh];
+
+workers.forEach((worker, i) => {
+  return new command.remote.Command(
+    `mount-longhorn-disk-${i}`,
+    {
+      connection: conn(worker),
+      create: `set -euo pipefail;
+        DISK_ID=$(ls /dev/disk/by-id/ | grep 'scsi1');
+        DISK_PATH="/dev/disk/by-id/$DISK_ID";
+        sudo blkid "$DISK_PATH" >/dev/null 2>&1 || sudo mkfs.ext4 "$DISK_PATH";
+        sudo mkdir -p /mnt/longhorn-extra;
+        echo ""$DISK_PATH" /mnt/longhorn-extra  ext4  defaults,nofail  0  2" | sudo tee -a /etc/fstab;
+        sudo mount -a;
+        df -h /mnt/longhorn-extra`,
+    },
+    { dependsOn: workerSshReader[i] },
+  );
+});
+
+// ---------------------------------------------------------------------------
+// Step 7: Read kubeconfig from master-1, patch server URL to the VIP
 // ---------------------------------------------------------------------------
 
 const getKubeconfig = new command.remote.Command(
@@ -225,11 +312,66 @@ const getKubeconfig = new command.remote.Command(
     create: `sudo cat /etc/rancher/k3s/k3s.yaml`,
     triggers: [master1VmId],
   },
-  { dependsOn: [joinWorker1, joinWorker2] },
+  { dependsOn: [joinWorker1, joinWorker2, joinWorker3] },
 );
 
+// ---------------------------------------------------------------------------
+// Step 8: Install additional packages
+// ---------------------------------------------------------------------------
+
+const allVms = [
+  {
+    id: master1VmId,
+    ip: master1Ip,
+    name: "master-1",
+    readiness: waitMaster1Ssh,
+  },
+  {
+    id: master2VmId,
+    ip: master2Ip,
+    name: "master-2",
+    readiness: waitMaster2Ssh,
+  },
+  {
+    id: master3VmId,
+    ip: master3Ip,
+    name: "master-3",
+    readiness: waitMaster3Ssh,
+  },
+  {
+    id: worker1VmId,
+    ip: worker1Ip,
+    name: "worker-1",
+    readiness: waitWorker1Ssh,
+  },
+  {
+    id: worker2VmId,
+    ip: worker2Ip,
+    name: "worker-2",
+    readiness: waitWorker2Ssh,
+  },
+  {
+    id: worker3VmId,
+    ip: worker3Ip,
+    name: "worker-3",
+    readiness: waitWorker3Ssh,
+  },
+];
+
+allVms.forEach(({ id, ip, name, readiness }) => {
+  new command.remote.Command(
+    `install-packages-${name}`,
+    {
+      connection: conn(ip),
+      create: pulumi.interpolate`sudo apt update && sudo apt install -y qemu-guest-agent && sudo systemctl enable --now qemu-guest-agent`,
+      triggers: [id],
+    },
+    { dependsOn: readiness },
+  );
+});
+
 export const kubeconfig = pulumi.secret(
   pulumi
-    .all([getKubeconfig.stdout, master1Ip])
-    .apply(([kc, ip]) => kc.replaceAll("127.0.0.1", ip as string).trim()),
+    .all([getKubeconfig.stdout, controlPlaneVipAddress])
+    .apply(([kc, vip]) => kc.replaceAll("127.0.0.1", vip as string).trim()),
 );

02-k8s-bootstrap/package-lock.json

@@ -6,12 +6,13 @@
         "": {
             "name": "k8s-bootstrap",
             "dependencies": {
-                "@pulumi/command": "^0.11.0",
-                "@pulumi/pulumi": "^3.113.0"
+                "@pulumi/command": "^1.2.1",
+                "@pulumi/kubernetes": "^4.31.1",
+                "@pulumi/pulumi": "^3.244.0"
             },
             "devDependencies": {
-                "@types/node": "^18",
-                "typescript": "^5.0.0"
+                "@types/node": "^24.0.0",
+                "typescript": "^6.0.0"
             }
         },
         "node_modules/@gar/promise-retry": {
@@ -54,6 +55,15 @@
                 "node": ">=6"
             }
         },
+        "node_modules/@isaacs/cliui": {
+            "version": "9.0.0",
+            "resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-9.0.0.tgz",
+            "integrity": "sha512-AokJm4tuBHillT+FpMtxQ60n8ObyXBatq7jD2/JA9dxbDDokKQm8KMht5ibGzLVU9IJDIKK4TPKgMHEYMn3lMg==",
+            "license": "BlueOak-1.0.0",
+            "engines": {
+                "node": ">=18"
+            }
+        },
         "node_modules/@isaacs/fs-minipass": {
             "version": "4.0.1",
             "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz",
@@ -694,13 +704,48 @@
             "license": "BSD-3-Clause"
         },
         "node_modules/@pulumi/command": {
-            "version": "0.11.1",
-            "resolved": "https://registry.npmjs.org/@pulumi/command/-/command-0.11.1.tgz",
-            "integrity": "sha512-rKo0fJ+nHP2xPwILZ92Rzpj1fHMXj+AENgSyJ0V0LGjxYCI1WRiJaYmQuFjLbk1YVccHOVR8jcnidMorSdxgeg==",
+            "version": "1.2.1",
+            "resolved": "https://registry.npmjs.org/@pulumi/command/-/command-1.2.1.tgz",
+            "integrity": "sha512-mutNDIUYP67yCBYOVIidQyxuTwZDY9v/sx9EGbgIv4PXfyfolOKGgGLeoHEbI1lxRwaw2wbTZ3VNIynDnA5VKA==",
             "hasInstallScript": true,
             "license": "Apache-2.0",
             "dependencies": {
-                "@pulumi/pulumi": "^3.0.0"
+                "@pulumi/pulumi": "^3.142.0"
+            }
+        },
+        "node_modules/@pulumi/kubernetes": {
+            "version": "4.32.0",
+            "resolved": "https://registry.npmjs.org/@pulumi/kubernetes/-/kubernetes-4.32.0.tgz",
+            "integrity": "sha512-JMzaG8BI+nsHzcOwFEbmueyn2HN4HrLv1znVAKS0vPgLrlSAc+WDsaRUxQnHBE37+3RL3r4mipUemRCIe8Xvww==",
+            "hasInstallScript": true,
+            "license": "Apache-2.0",
+            "dependencies": {
+                "@pulumi/pulumi": "^3.142.0",
+                "glob": "^12.0.0",
+                "shell-quote": "^1.6.1"
+            }
+        },
+        "node_modules/@pulumi/kubernetes/node_modules/glob": {
+            "version": "12.0.0",
+            "resolved": "https://registry.npmjs.org/glob/-/glob-12.0.0.tgz",
+            "integrity": "sha512-5Qcll1z7IKgHr5g485ePDdHcNQY0k2dtv/bjYy0iuyGxQw2qSOiiXUXJ+AYQpg3HNoUMHqAruX478Jeev7UULw==",
+            "license": "BlueOak-1.0.0",
+            "dependencies": {
+                "foreground-child": "^3.3.1",
+                "jackspeak": "^4.1.1",
+                "minimatch": "^10.1.1",
+                "minipass": "^7.1.2",
+                "package-json-from-dist": "^1.0.0",
+                "path-scurry": "^2.0.0"
+            },
+            "bin": {
+                "glob": "dist/esm/bin.mjs"
+            },
+            "engines": {
+                "node": "20 || >=22"
+            },
+            "funding": {
+                "url": "https://github.com/sponsors/isaacs"
             }
         },
         "node_modules/@pulumi/pulumi": {
@@ -856,12 +901,12 @@
             "license": "MIT"
         },
         "node_modules/@types/node": {
-            "version": "18.19.130",
-            "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz",
-            "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==",
+            "version": "24.13.1",
+            "resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.1.tgz",
+            "integrity": "sha512-RSpUJGmvsJ1ZeBehQZFhIdpsz+bIpES0nIQXko4Ybq+N+kX6XvOq3Jo+iJ82FWLdblFq85AsMikd3m35jgezYg==",
             "license": "MIT",
             "dependencies": {
-                "undici-types": "~5.26.4"
+                "undici-types": "~7.18.0"
             }
         },
         "node_modules/@types/semver": {
@@ -1239,6 +1284,34 @@
                 }
             }
         },
+        "node_modules/foreground-child": {
+            "version": "3.3.1",
+            "resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.1.tgz",
+            "integrity": "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==",
+            "license": "ISC",
+            "dependencies": {
+                "cross-spawn": "^7.0.6",
+                "signal-exit": "^4.0.1"
+            },
+            "engines": {
+                "node": ">=14"
+            },
+            "funding": {
+                "url": "https://github.com/sponsors/isaacs"
+            }
+        },
+        "node_modules/foreground-child/node_modules/signal-exit": {
+            "version": "4.1.0",
+            "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz",
+            "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==",
+            "license": "ISC",
+            "engines": {
+                "node": ">=14"
+            },
+            "funding": {
+                "url": "https://github.com/sponsors/isaacs"
+            }
+        },
         "node_modules/fs-minipass": {
             "version": "3.0.3",
             "resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-3.0.3.tgz",
@@ -1479,6 +1552,21 @@
                 "node": ">=20"
             }
         },
+        "node_modules/jackspeak": {
+            "version": "4.2.3",
+            "resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-4.2.3.tgz",
+            "integrity": "sha512-ykkVRwrYvFm1nb2AJfKKYPr0emF6IiXDYUaFx4Zn9ZuIH7MrzEZ3sD5RlqGXNRpHtvUHJyOnCEFxOlNDtGo7wg==",
+            "license": "BlueOak-1.0.0",
+            "dependencies": {
+                "@isaacs/cliui": "^9.0.0"
+            },
+            "engines": {
+                "node": "20 || >=22"
+            },
+            "funding": {
+                "url": "https://github.com/sponsors/isaacs"
+            }
+        },
         "node_modules/js-yaml": {
             "version": "3.14.2",
             "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.2.tgz",
@@ -1962,6 +2050,12 @@
                 "url": "https://github.com/sponsors/sindresorhus"
             }
         },
+        "node_modules/package-json-from-dist": {
+            "version": "1.0.1",
+            "resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz",
+            "integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==",
+            "license": "BlueOak-1.0.0"
+        },
         "node_modules/pacote": {
             "version": "21.5.0",
             "resolved": "https://registry.npmjs.org/pacote/-/pacote-21.5.0.tgz",
@@ -2225,6 +2319,18 @@
                 "node": ">=8"
             }
         },
+        "node_modules/shell-quote": {
+            "version": "1.8.4",
+            "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.4.tgz",
+            "integrity": "sha512-VsC6n6vz1ihYYyZZwX7YZSF5l5x36ca17OC+a69h94YqB7X6XLwf+5MOgynYir2SLFUbl8gIYvBo8K8RoNQ6bQ==",
+            "license": "MIT",
+            "engines": {
+                "node": ">= 0.4"
+            },
+            "funding": {
+                "url": "https://github.com/sponsors/ljharb"
+            }
+        },
         "node_modules/shimmer": {
             "version": "1.2.1",
             "resolved": "https://registry.npmjs.org/shimmer/-/shimmer-1.2.1.tgz",
@@ -2483,9 +2589,9 @@
             }
         },
         "node_modules/typescript": {
-            "version": "5.9.3",
-            "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
-            "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
+            "version": "6.0.3",
+            "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
+            "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
             "devOptional": true,
             "license": "Apache-2.0",
             "bin": {
@@ -2506,9 +2612,9 @@
             }
         },
         "node_modules/undici-types": {
-            "version": "5.26.5",
-            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz",
-            "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==",
+            "version": "7.18.2",
+            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
+            "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
             "license": "MIT"
         },
         "node_modules/upath": {

02-k8s-bootstrap/package.json

@@ -2,11 +2,12 @@
     "name": "k8s-bootstrap",
     "main": "index.ts",
     "devDependencies": {
-        "@types/node": "^18",
-        "typescript": "^5.0.0"
+        "@types/node": "^24.0.0",
+        "typescript": "^6.0.0"
     },
     "dependencies": {
-        "@pulumi/command": "^0.11.0",
-        "@pulumi/pulumi": "^3.113.0"
+        "@pulumi/command": "^1.2.1",
+        "@pulumi/kubernetes": "^4.31.1",
+        "@pulumi/pulumi": "^3.244.0"
     }
 }

03-k8s-infra/CLAUDE.md

@@ -0,0 +1,55 @@
+# k8s-infra
+
+This file provides guidance to Claude Code when working in `03-k8s-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Cluster-level infrastructure for the k3s homelab cluster, deployed via Helm and `@pulumi/kubernetes`. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi refresh --yes && pulumi up --yes
+```
+
+CI installs the `helm` CLI (`azure/setup-helm`) before running Pulumi.
+
+## Architecture
+
+| Component | What it does |
+| --- | --- |
+| NFS CSI Driver | `csi-driver-nfs` Helm chart — enables NFS-backed PVCs |
+| `truenas-nfs` StorageClass | Backed by an NFS share on TrueNAS (`Retain` reclaim policy) |
+| cert-manager | Certificate lifecycle management |
+| `letsencrypt-prod` ClusterIssuer | DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
+| Longhorn | Default block storage (`longhorn` StorageClass, 2 replicas) — used by app/monitoring PVCs. Worker nodes get an extra `longhorn-extra` disk at `/mnt/longhorn-extra` (a dedicated mount set up by `k8s-bootstrap`, patched onto the Longhorn `Node` CR here) so Longhorn data stays off the root disk and doesn't cause `DiskPressure`. `nodeDownPodDeletionPolicy` deletes both the StatefulSet and Deployment pod on node-down so stateful workloads reschedule automatically |
+| kube-vip | DaemonSet that ARP-announces a virtual IP for LoadBalancer Services |
+| kube-vip-cloud-provider | Allocates LoadBalancer IPs from a single-address pool (`kubeVipAddress`) — k3s's built-in ServiceLB is not used |
+| `traefik` ServicePatch | Pins Traefik's LoadBalancer IP to `kubeVipAddress` |
+
+**Export**: `nfsStorageClass = "truenas-nfs"`.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret truenasHost      <truenas-ip>
+pulumi config set --secret truenasNfsPath   /mnt/pool/k8s
+pulumi config set --secret cloudflareApiToken <cf-token>
+pulumi config set --secret letsencryptEmail   <your-email>
+pulumi config set --secret kubeVipAddress   <vip-ip>
+pulumi config set kubeVipInterface          <e.g. eth0>
+```
+
+`kubeconfig` is read automatically via StackReference — do not set it here.
+
+## TrueNAS one-time setup (before deploying)
+
+1. Create dataset `pool/k8s`
+2. Add an NFS share for that dataset:
+   - ACL type: POSIX/generic Unix permissions (not SMB/NFSv4 ACLs) on TrueNAS SCALE
+   - Maproot User / Group: `root` / `root`
+   - Allowed Hosts/Networks: restrict to the k3s node IPs/CIDR
+3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/03-deploy-k8s-infra.yaml`. Secret: `K8S_INFRA_PULUMI_DEV_YAML`.

03-k8s-infra/index.ts

@@ -3,15 +3,19 @@ import * as k8s from "@pulumi/kubernetes";
 
 const config = new pulumi.Config();
 
-const infraRef = new pulumi.StackReference(
+const bootstrapRef = new pulumi.StackReference(
   `${pulumi.getOrganization()}/k8s-bootstrap/dev`,
 );
 
-const kubeconfig = infraRef.requireOutput("kubeconfig");
+const kubeconfig = bootstrapRef.requireOutput("kubeconfig");
+
+export const nfsStorageClass = "truenas-nfs";
 const truenasHost = config.requireSecret("truenasHost");
 const truenasNfsPath = config.requireSecret("truenasNfsPath");
+
 const cloudflareToken = config.requireSecret("cloudflareApiToken");
 const letsencryptEmail = config.requireSecret("letsencryptEmail");
+
 const kubeVipAddress = config.requireSecret("kubeVipAddress");
 const kubeVipInterface = config.require("kubeVipInterface");
 
@@ -41,9 +45,9 @@ const nfsCsiDriver = new k8s.helm.v3.Release(
 );
 
 new k8s.storage.v1.StorageClass(
-  "truenas-nfs",
+  nfsStorageClass,
   {
-    metadata: { name: "truenas-nfs" },
+    metadata: { name: nfsStorageClass },
     provisioner: "nfs.csi.k8s.io",
     parameters: {
       server: truenasHost,
@@ -118,7 +122,7 @@ new k8s.apiextensions.CustomResource(
 
 // ── Longhorn ──────────────────────────────────────────────────────────────
 
-new k8s.helm.v3.Release(
+const longhorn = new k8s.helm.v3.Release(
   "longhorn",
   {
     name: "longhorn",
@@ -129,13 +133,46 @@ new k8s.helm.v3.Release(
     version: "1.12.0",
     values: {
       defaultSettings: {
-        defaultReplicaCount: 3,
+        defaultReplicaCount: 2,
+        nodeDownPodDeletionPolicy: "delete-both-statefulset-and-deployment-pod",
+        createDefaultDiskLabeledNodes: true,
       },
     },
   },
   opts(),
 );
 
+// Patches Longhorn instances to use the mountend Longhorn storage on the worker nodes.
+["k3s-worker-1", "k3s-worker-2", "k3s-worker-3"].forEach(
+  (nodeName) =>
+    new k8s.apiextensions.CustomResourcePatch(
+      `${nodeName}-longhorn-disks`,
+      {
+        apiVersion: "longhorn.io/v1beta2",
+        kind: "Node",
+        metadata: {
+          name: nodeName,
+          namespace: "longhorn-system",
+          annotations: { "pulumi.com/patchForce": "true" },
+        },
+        spec: {
+          disks: {
+            "longhorn-extra": {
+              allowScheduling: true,
+              diskDriver: "",
+              diskType: "filesystem",
+              evictionRequested: false,
+              path: "/mnt/longhorn-extra",
+              storageReserved: 0,
+              tags: [],
+            },
+          },
+        },
+      },
+      opts({ dependsOn: [longhorn] }),
+    ),
+);
+
 // ── kube-vip ────────────
 
 const kubeVipNamespace = "kube-system";
@@ -144,6 +181,22 @@ const controlPlaneTolerations = [
   { key: "node-role.kubernetes.io/control-plane", effect: "NoSchedule" },
 ];
 
+// Keeps regular workloads off the master nodes; kube-vip and
+// kube-vip-cloud-provider below explicitly tolerate these same taints.
+["k3s-master-1", "k3s-master-2", "k3s-master-3"].forEach((nodeName) => {
+  new k8s.core.v1.NodePatch(
+    `${nodeName}-control-plane-taint`,
+    {
+      metadata: {
+        name: nodeName,
+        annotations: { "pulumi.com/patchForce": "true" },
+      },
+      spec: { taints: controlPlaneTolerations },
+    },
+    opts(),
+  );
+});
+
 const kubeVipServiceAccount = new k8s.core.v1.ServiceAccount(
   "kube-vip",
   { metadata: { name: "kube-vip", namespace: kubeVipNamespace } },
@@ -321,13 +374,20 @@ const cloudProviderClusterRoleBinding = new k8s.rbac.v1.ClusterRoleBinding(
   opts(),
 );
 
+const kubeVipCloudProviderLabels = {
+  app: "kube-vip",
+  component: "kube-vip-cloud-provider",
+};
+
 // IP pool the cloud-provider allocates from — scoped to exactly the one VIP we
 // want, so it can never hand the address to some other LoadBalancer Service.
 const kubeVipPool = new k8s.core.v1.ConfigMap(
   "kube-vip-pool",
   {
     metadata: { name: "kubevip", namespace: kubeVipNamespace },
-    data: { "range-global": pulumi.interpolate`${kubeVipAddress}-${kubeVipAddress}` },
+    data: {
+      "range-global": pulumi.interpolate`${kubeVipAddress}-${kubeVipAddress}`,
+    },
   },
   opts(),
 );
@@ -339,11 +399,11 @@ const kubeVipCloudProvider = new k8s.apps.v1.Deployment(
     spec: {
       replicas: 1,
       selector: {
-        matchLabels: { app: "kube-vip", component: "kube-vip-cloud-provider" },
+        matchLabels: kubeVipCloudProviderLabels,
       },
       template: {
         metadata: {
-          labels: { app: "kube-vip", component: "kube-vip-cloud-provider" },
+          labels: kubeVipCloudProviderLabels,
         },
         spec: {
           serviceAccountName: cloudProviderServiceAccount.metadata.name,
@@ -378,5 +438,3 @@ new k8s.core.v1.ServicePatch(
   },
   opts({ dependsOn: [kubeVip, kubeVipCloudProvider] }),
 );
-
-export const storageClass = "truenas-nfs";

04-k8s-apps/CLAUDE.md

@@ -0,0 +1,32 @@
+# k8s-apps
+
+This file provides guidance to Claude Code when working in `04-k8s-apps`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Application workloads for the k3s homelab cluster. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi up --yes
+```
+
+## Architecture
+
+- **heimdall** — application dashboard (`lscr.io/linuxserver/heimdall`), namespace `heimdall`
+  - PVC `heimdall-config` (50Mi, `longhorn` StorageClass)
+  - Deployment + ClusterIP Service
+  - Ingress (`traefik` class, `cert-manager.io/cluster-issuer: letsencrypt-prod`) at `heimdall.<domain>`
+  - deployed with one replica because heimdall is not able to work with multiple replicas. Works with node failure detection and automatic rescheduling.
+
+**Export**: `domain` (secret) — consumed by `monitoring` via StackReference for its own ingress hostnames.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret domain <your-domain>
+```
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/04-deploy-k8s-apps.yaml`. Secret: `K8S_APPS_PULUMI_DEV_YAML`.

04-k8s-apps/index.ts

@@ -3,7 +3,6 @@ import * as k8s from "@pulumi/kubernetes";
 
 const config = new pulumi.Config();
 
-//fetch credentials from k8s-bootstrap
 const bootstrapRef = new pulumi.StackReference(
   `${pulumi.getOrganization()}/k8s-bootstrap/dev`,
 );
@@ -17,7 +16,7 @@ const opts = (extras?: pulumi.ResourceOptions): pulumi.ResourceOptions => ({
   ...extras,
 });
 
-const ns = new k8s.core.v1.Namespace(
+const heimdallNs = new k8s.core.v1.Namespace(
   "heimdall",
   { metadata: { name: "heimdall" } },
   opts(),
@@ -26,7 +25,7 @@ const ns = new k8s.core.v1.Namespace(
 const pvc = new k8s.core.v1.PersistentVolumeClaim(
   "heimdall-config",
   {
-    metadata: { name: "heimdall-config", namespace: ns.metadata.name },
+    metadata: { name: "heimdall-config", namespace: heimdallNs.metadata.name },
     spec: {
       accessModes: ["ReadWriteOnce"],
       storageClassName: "longhorn",
@@ -43,7 +42,7 @@ new k8s.apps.v1.Deployment(
   {
     metadata: {
       name: "heimdall",
-      namespace: ns.metadata.name,
+      namespace: heimdallNs.metadata.name,
       annotations: { "pulumi.com/patchForce": "true" },
     },
     spec: {
@@ -81,7 +80,7 @@ new k8s.apps.v1.Deployment(
 const svc = new k8s.core.v1.Service(
   "heimdall",
   {
-    metadata: { name: "heimdall", namespace: ns.metadata.name },
+    metadata: { name: "heimdall", namespace: heimdallNs.metadata.name },
     spec: {
       selector: labels,
       ports: [{ port: 80, targetPort: 80 }],
@@ -98,7 +97,7 @@ new k8s.networking.v1.Ingress(
   {
     metadata: {
       name: "heimdall",
-      namespace: ns.metadata.name,
+      namespace: heimdallNs.metadata.name,
       annotations: {
         "cert-manager.io/cluster-issuer": "letsencrypt-prod",
       },

04-k8s-apps/package-lock.json

@@ -6,12 +6,12 @@
         "": {
             "name": "k8s-apps",
             "dependencies": {
-                "@pulumi/kubernetes": "^4.0.0",
-                "@pulumi/pulumi": "^3.113.0"
+                "@pulumi/kubernetes": "^4.31.1",
+                "@pulumi/pulumi": "^3.244.0"
             },
             "devDependencies": {
-                "@types/node": "^18",
-                "typescript": "^5.0.0"
+                "@types/node": "^24.0.0",
+                "typescript": "^6.0.0"
             }
         },
         "node_modules/@gar/promise-retry": {
@@ -749,9 +749,9 @@
             }
         },
         "node_modules/@pulumi/pulumi": {
-            "version": "3.243.0",
-            "resolved": "https://registry.npmjs.org/@pulumi/pulumi/-/pulumi-3.243.0.tgz",
-            "integrity": "sha512-KRIBErgCj9+gsEG7fOo3xRvvhq3VGw8gnzVpbFiyW2Zqr84MCIKDDVmj4MZDt/61KOn3OapI4ia1/6dSg3qwzQ==",
+            "version": "3.245.0",
+            "resolved": "https://registry.npmjs.org/@pulumi/pulumi/-/pulumi-3.245.0.tgz",
+            "integrity": "sha512-cRfyKlaoKThewFTRFdvLFSo33FLHhViYQG1xI1WBTS4mrjhJegWl28HmZupVEIQh8aqvBN0Cy0fFNbxbZnOzmA==",
             "license": "Apache-2.0",
             "dependencies": {
                 "@grpc/grpc-js": "^1.10.1",
@@ -901,12 +901,12 @@
             "license": "MIT"
         },
         "node_modules/@types/node": {
-            "version": "18.19.130",
-            "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz",
-            "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==",
+            "version": "24.13.1",
+            "resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.1.tgz",
+            "integrity": "sha512-RSpUJGmvsJ1ZeBehQZFhIdpsz+bIpES0nIQXko4Ybq+N+kX6XvOq3Jo+iJ82FWLdblFq85AsMikd3m35jgezYg==",
             "license": "MIT",
             "dependencies": {
-                "undici-types": "~5.26.4"
+                "undici-types": "~7.18.0"
             }
         },
         "node_modules/@types/semver": {
@@ -2612,9 +2612,9 @@
             }
         },
         "node_modules/typescript": {
-            "version": "5.9.3",
-            "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
-            "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
+            "version": "6.0.3",
+            "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
+            "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
             "devOptional": true,
             "license": "Apache-2.0",
             "bin": {
@@ -2635,9 +2635,9 @@
             }
         },
         "node_modules/undici-types": {
-            "version": "5.26.5",
-            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz",
-            "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==",
+            "version": "7.18.2",
+            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
+            "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
             "license": "MIT"
         },
         "node_modules/upath": {

04-k8s-apps/package.json

@@ -2,11 +2,11 @@
     "name": "k8s-apps",
     "main": "index.ts",
     "devDependencies": {
-        "@types/node": "^18",
-        "typescript": "^5.0.0"
+        "@types/node": "^24.0.0",
+        "typescript": "^6.0.0"
     },
     "dependencies": {
-        "@pulumi/kubernetes": "^4.0.0",
-        "@pulumi/pulumi": "^3.113.0"
+        "@pulumi/kubernetes": "^4.31.1",
+        "@pulumi/pulumi": "^3.244.0"
     }
 }

04-k8s-apps/tsconfig.json

@@ -2,7 +2,7 @@
     "compilerOptions": {
         "strict": true,
         "outDir": "bin",
-        "target": "es2020",
+        "target": "es2024",
         "module": "nodenext",
         "moduleResolution": "nodenext",
         "sourceMap": true,

05-monitoring/CLAUDE.md

@@ -0,0 +1,34 @@
+# monitoring
+
+This file provides guidance to Claude Code when working in `05-monitoring`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Monitoring stack (Prometheus + Grafana) via the `kube-prometheus-stack` Helm chart. Reads `kubeconfig` from `k8s-bootstrap` and `domain` from `k8s-apps`, both via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi up --yes
+```
+
+## Architecture
+
+- Namespace `monitoring`
+- `kube-prometheus-stack` Helm release (chart v86.2.0)
+  - Grafana: ingress at `grafana.<domain>` (`traefik`, `letsencrypt-prod`), `longhorn` PVC (2Gi)
+  - Prometheus: `longhorn` PVC (20Gi) via `volumeClaimTemplate`
+  - Alertmanager disabled
+
+## Key Notes
+
+- The `monitoring` namespace and the Helm release both use the Pulumi `import` resource option (`monitoring`, `monitoring/kube-prometheus-stack`) — they were adopted from pre-existing cluster resources rather than created fresh. Be careful with renames or changes that would force a replace.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret grafanaAdminPassword <password>
+```
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/05-deploy-monitoring.yaml`. Secret: `MONITORING_PULUMI_DEV_YAML`.

05-monitoring/index.ts

@@ -13,28 +13,37 @@ const appsRef = new pulumi.StackReference(
 const kubeconfig = bootstrapRef.requireOutput("kubeconfig");
 const domain = appsRef.requireOutput("domain");
 const grafanaAdminPassword = config.requireSecret("grafanaAdminPassword");
+const grafanaDbUser = config.requireSecret("grafanaDbUser");
+const grafanaDbPassword = config.requireSecret("grafanaDbPassword");
+const grafanaDbHost = config.requireSecret("grafanaDbHost");
+const grafanaDbName = config.require("grafanaDbName");
 
 const k8sProvider = new k8s.Provider("k3s", { kubeconfig });
-const opts = (extras?: pulumi.ResourceOptions): pulumi.ResourceOptions => ({
-  provider: k8sProvider,
-  ...extras,
-});
 
-const ns = new k8s.core.v1.Namespace(
+const monitoringNs = new k8s.core.v1.Namespace(
   "monitoring",
   { metadata: { name: "monitoring" } },
-  opts(),
+  { provider: k8sProvider, import: "monitoring" },
 );
 
 const grafanaHost = pulumi.interpolate`grafana.${domain}`;
 
+const grafanaDbSecret = new k8s.core.v1.Secret(
+  "grafana-db-credentials",
+  {
+    metadata: { namespace: monitoringNs.metadata.name },
+    stringData: { password: grafanaDbPassword },
+  },
+  { provider: k8sProvider },
+);
+
 new k8s.helm.v3.Release(
   "kube-prometheus-stack",
   {
     name: "kube-prometheus-stack",
     chart: "kube-prometheus-stack",
     version: "86.2.0",
-    namespace: ns.metadata.name,
+    namespace: monitoringNs.metadata.name,
     repositoryOpts: {
       repo: "https://prometheus-community.github.io/helm-charts",
     },
@@ -42,6 +51,7 @@ new k8s.helm.v3.Release(
     values: {
       grafana: {
         adminPassword: grafanaAdminPassword,
+        replicas: 2,
         ingress: {
           enabled: true,
           ingressClassName: "traefik",
@@ -49,19 +59,34 @@ new k8s.helm.v3.Release(
           hosts: [grafanaHost],
           tls: [{ secretName: "grafana-tls", hosts: [grafanaHost] }],
         },
-        persistence: {
-          enabled: true,
-          storageClassName: "longhorn",
-          size: "2Gi",
+        env: {
+          GF_DATABASE_TYPE: "postgres",
+          GF_DATABASE_HOST: grafanaDbHost,
+          GF_DATABASE_NAME: grafanaDbName,
+          GF_DATABASE_USER: grafanaDbUser,
+          GF_DATABASE_SSL_MODE: "disable",
+        },
+        envValueFrom: {
+          GF_DATABASE_PASSWORD: {
+            secretKeyRef: {
+              name: grafanaDbSecret.metadata.name,
+              key: "password",
+            },
+          },
         },
       },
       prometheus: {
         prometheusSpec: {
+          retention: "14d",
+          retentionSize: "15GB",
+          scrapeInterval: "60s",
+          evaluationInterval: "60s",
+          replicas: "1",
           storageSpec: {
             volumeClaimTemplate: {
               spec: {
                 storageClassName: "longhorn",
-                resources: { requests: { storage: "20Gi" } },
+                resources: { requests: { storage: "18Gi" } },
               },
             },
           },
@@ -70,5 +95,5 @@ new k8s.helm.v3.Release(
       alertmanager: { enabled: false },
     },
   },
-  opts(),
+  { provider: k8sProvider, import: "monitoring/kube-prometheus-stack" },
 );

05-monitoring/package-lock.json

@@ -1,17 +1,17 @@
 {
-    "name": "k8s-monitoring",
+    "name": "monitoring",
     "lockfileVersion": 3,
     "requires": true,
     "packages": {
         "": {
-            "name": "k8s-monitoring",
+            "name": "monitoring",
             "dependencies": {
-                "@pulumi/kubernetes": "^4.0.0",
-                "@pulumi/pulumi": "^3.113.0"
+                "@pulumi/kubernetes": "^4.31.1",
+                "@pulumi/pulumi": "^3.244.0"
             },
             "devDependencies": {
-                "@types/node": "^18",
-                "typescript": "^5.0.0"
+                "@types/node": "^24.0.0",
+                "typescript": "^6.0.0"
             }
         },
         "node_modules/@gar/promise-retry": {
@@ -901,12 +901,12 @@
             "license": "MIT"
         },
         "node_modules/@types/node": {
-            "version": "18.19.130",
-            "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz",
-            "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==",
+            "version": "24.13.1",
+            "resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.1.tgz",
+            "integrity": "sha512-RSpUJGmvsJ1ZeBehQZFhIdpsz+bIpES0nIQXko4Ybq+N+kX6XvOq3Jo+iJ82FWLdblFq85AsMikd3m35jgezYg==",
             "license": "MIT",
             "dependencies": {
-                "undici-types": "~5.26.4"
+                "undici-types": "~7.18.0"
             }
         },
         "node_modules/@types/semver": {
@@ -2612,9 +2612,9 @@
             }
         },
         "node_modules/typescript": {
-            "version": "5.9.3",
-            "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
-            "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
+            "version": "6.0.3",
+            "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
+            "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
             "devOptional": true,
             "license": "Apache-2.0",
             "bin": {
@@ -2635,9 +2635,9 @@
             }
         },
         "node_modules/undici-types": {
-            "version": "5.26.5",
-            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz",
-            "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==",
+            "version": "7.18.2",
+            "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
+            "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
             "license": "MIT"
         },
         "node_modules/upath": {

05-monitoring/package.json

@@ -1,5 +1,5 @@
 {
-    "name": "k8s-monitoring",
+    "name": "monitoring",
     "main": "index.ts",
     "devDependencies": {
         "@types/node": "^24.0.0",

CLAUDE.md

@@ -0,0 +1,64 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code when working anywhere in this repository. Each stack directory (`01-proxmox-infra`, `02-k8s-bootstrap`, etc.) has its own `CLAUDE.md` with stack-specific commands and architecture — Claude Code loads both this file and the stack's file automatically when working inside a stack directory.
+
+## Project overview
+
+Pulumi (TypeScript) IaC for a Proxmox-based k3s homelab. Independent Pulumi projects ("stacks"), each with its own `Pulumi.yaml`/`package.json`, deployed in order and linked via `pulumi.StackReference`:
+
+| # | Directory | Pulumi stack | Purpose |
+| --- | --- | --- | --- |
+| 1 | `01-proxmox-infra` | `proxmox-infra` | Provisions k3s VMs + DHCP on Proxmox/pfSense |
+| 2 | `02-k8s-bootstrap` | `k8s-bootstrap` | Installs k3s on the VMs over SSH |
+| 3 | `03-k8s-infra` | `k8s-infra` | Cluster infra: storage, cert-manager, kube-vip |
+| 4 | `04-k8s-apps` | `k8s-apps` | Application workloads (e.g. heimdall) |
+| 5 | `05-monitoring` | `monitoring` | kube-prometheus-stack (Prometheus + Grafana) |
+
+All stacks use the Pulumi stack name `dev`.
+
+## Deployment order & StackReferences
+
+```
+proxmox-infra → k8s-bootstrap → k8s-infra
+                              ↘ k8s-apps → monitoring
+```
+
+- `k8s-bootstrap` reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference.
+- `k8s-infra`, `k8s-apps`, and `monitoring` all read `kubeconfig` from `k8s-bootstrap` via StackReference (`requireOutput("kubeconfig")`) — never set `kubeconfig` as Pulumi config in these stacks.
+- `monitoring` additionally reads `domain` from `k8s-apps` via StackReference.
+- StackReference format: `${pulumi.getOrganization()}/<stack-name>/dev`.
+
+A stack's outputs must exist (i.e. it must have been deployed) before any downstream stack referencing it will succeed.
+
+## Shared conventions
+
+- **TypeScript**: `strict: true`, `target: es2024`, `module`/`moduleResolution: nodenext`, `noImplicitReturns`, `noFallthroughCasesInSwitch` — functions need explicit return types where TS can't infer them.
+- **Secrets**: read via `config.requireSecret()`, never via environment variables or hardcoded values.
+- **`Pulumi.dev.yaml`**: gitignored (contains the encryption salt) — never delete it or secrets become unrecoverable. Encrypted with `PULUMI_CONFIG_PASSPHRASE`.
+- **Backend**: self-hosted Postgres state backend, set via `PULUMI_BACKEND_URL`.
+- Don't add a co-author when committing to git.
+
+## Common commands (run from a stack directory)
+
+```bash
+npm install
+pulumi preview
+pulumi up --yes
+pulumi destroy
+pulumi stack output
+pulumi config
+```
+
+Requires `PULUMI_BACKEND_URL` and `PULUMI_CONFIG_PASSPHRASE` in the environment (or `pulumi login`).
+
+## CI/CD (Gitea Actions)
+
+One workflow per stack: `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
+
+- **Pull request → `main`**: `pulumi preview` only.
+- `01`/`02` (VM provisioning / cluster bootstrap — riskier to automate): deploy only via manual `workflow_dispatch`.
+- `03`–`05`: deploy on push to `main` *or* `workflow_dispatch` — `pulumi refresh` then `pulumi up`.
+- Each workflow restores `Pulumi.dev.yaml` from a base64-encoded Gitea secret (`<STACK>_PULUMI_DEV_YAML`). After changing local config, re-encode and update the secret:
+  ```bash
+  base64 -w 0 0N-<stack>/Pulumi.dev.yaml
+  ```

CONVENTIONS.md

@@ -0,0 +1,63 @@
+# Git Conventions
+
+## Branch Naming
+
+```
+<type>/<short-kebab-case-description>
+```
+
+- All lowercase, hyphens only — no camelCase or Title-Case
+- Keep the description short (3–5 words); the branch name is not the place for detail
+- No ticket tracker is used on this project, so branch names don't carry an ID
+
+| Type | Use for |
+|---|---|
+| `feature/` | New functionality (e.g. a new stack, a new resource type) |
+| `bug/` | Bug fixes |
+| `enhancement/` | Improvements or refactors to existing functionality, no new capability |
+| `chore/` | Dependency bumps, tooling, CI/CD, cleanup |
+| `docs/` | Documentation only (README, CLAUDE.md files) |
+
+**Examples:** `feature/add-3rd-proxmox-node`, `bug/fix-longhorn-disk-pressure`, `enhancement/refactor-k8s-bootstrap`
+
+---
+
+## Commit Messages
+
+```
+<type>: <imperative, present-tense summary>
+
+<optional body — explain WHY, not WHAT>
+```
+
+- **Type** — same list as branch types above (`feature`, `bug`, `enhancement`, `chore`, `docs`)
+- **Scope is optional** and, if used, should be one of the stack directories this repo is organized into — `proxmox-infra`, `k8s-bootstrap`, `k8s-infra`, `k8s-apps`, `monitoring` — or `ci` for `.gitea/workflows` changes: `fix(k8s-infra): ...`
+- **Summary** — imperative mood ("add", not "added" or "adds"); no period at the end
+- **Body** — only when the reasoning isn't obvious from the diff (a constraint, a workaround for a Proxmox/k3s quirk, a decision between StackReference outputs). Skip it for simple/self-explanatory changes.
+
+**Examples:**
+```
+feature: add pfsense provider for automating static IP setup
+bug: prune stale kube-vip leases after node replacement
+chore: upgrade Pulumi provider packages across all stacks
+docs: update CLAUDE.md with new Longhorn mount settings
+enhancement(k8s-bootstrap): cache dependencies to speed up deployment
+```
+
+---
+
+## Pull Request Titles
+
+This repo uses Gitea Actions (see root `CLAUDE.md`); Gitea's merge commit reuses the PR title verbatim as the merge summary, so the title should stand on its own in `git log`. Frame it as the outcome, not a task log:
+
+```
+<Type>: <what changes for the cluster/infra>
+```
+
+**Examples:**
+- `Feature: Add 3rd Proxmox node to the cluster`
+- `Bug: Fix Longhorn disk pressure from root disk usage`
+- `Enhancement: Refactor k8s-bootstrap for clearer deployment order`
+- `Chore: Upgrade Pulumi packages across all stacks`
+
+Since `01`/`02` deploys are manual (`workflow_dispatch`) and `03`–`05` deploy on merge to `main`, a clear PR title also doubles as a quick changelog of what just went live.

README.md

@@ -1,239 +1,154 @@
 # Homelab Infrastructure as Code
 
-A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks with version-controlled, reproducible infrastructure — a highly available k3s cluster across multiple Proxmox nodes, bootstrapped and configured end-to-end from code.
+A Pulumi-based IaC project for a Proxmox homelab: a highly-available k3s cluster, its cluster-level infrastructure, and the applications running on top of it — all provisioned and configured from code.
 
 This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab.
 
 ## Why IaC for a homelab?
 
-- **Reproducibility** — rebuild your entire environment from scratch with a single command
+- **Reproducibility** — rebuild the entire environment from scratch from code
 - **Version history** — every change is tracked in git; roll back at any time
-- **Auditability** — diff infrastructure changes before applying them (`pulumi preview`)
-- **Automation** — CI/CD handles deploys; no manual SSH into nodes
-- **Portability** — swap node names, datastores, or credentials without touching logic
+- **Auditability** — `pulumi preview` diffs infrastructure changes before they're applied
+- **Automation** — CI/CD handles deploys; no manual SSH into nodes for routine changes
+- **Portability** — swap node names, datastores, domains, or credentials without touching logic
 
 ## Repository layout
 
 ```
 .
-├── proxmox-infra/          # Stack 1 — VMs & DHCP on Proxmox + pfSense
-│   ├── index.ts
-│   ├── Pulumi.yaml
-│   └── sdks/pfsense/       # Locally bundled @pulumi/pfsense SDK
-├── k8s-bootstrap/          # Stack 2 — Install k3s on the VMs over SSH
-│   ├── index.ts
-│   └── Pulumi.yaml
-├── k8s-infra/              # Stack 3 — Cluster-level infrastructure via Helm
-│   ├── index.ts
-│   └── Pulumi.yaml
-└── .gitea/workflows/       # Gitea Actions — one workflow per stack
+├── 01-proxmox-infra/   # Stack 1 — k3s VMs + DHCP on Proxmox/pfSense
+├── 02-k8s-bootstrap/   # Stack 2 — install k3s on the VMs over SSH
+├── 03-k8s-infra/       # Stack 3 — storage, cert-manager, kube-vip
+├── 04-k8s-apps/        # Stack 4 — application workloads
+├── 05-monitoring/      # Stack 5 — kube-prometheus-stack (Prometheus + Grafana)
+├── .gitea/workflows/   # CI/CD — one workflow per stack
+└── CLAUDE.md           # Conventions & architecture notes (root + one per stack)
 ```
 
-## Stack overview
+Each stack is an independent Pulumi project (own `Pulumi.yaml`/`package.json`), all using the Pulumi stack name `dev`.
 
-The three stacks are deployed in order and linked via Pulumi StackReferences — outputs from one stack flow automatically into the next.
+## Stacks
+
+Stacks are deployed in order and linked via `pulumi.StackReference` — outputs from one stack flow automatically into the next, so a stack must be deployed before anything depending on it will work.
 
 ```
-proxmox-infra  →  k8s-bootstrap  →  k8s-infra
+proxmox-infra → k8s-bootstrap → k8s-infra
+                              ↘ k8s-apps → monitoring
 ```
 
-### Stack 1: `proxmox-infra`
+### 1. proxmox-infra
 
-Provisions a 5-node k3s cluster spread across two Proxmox hosts:
+Provisions a 5-node k3s cluster across two Proxmox hosts, each VM a clone of an Ubuntu Noble (24.04) cloud-image template:
 
-| VM name      | Role   | Proxmox node |
-| ------------ | ------ | ------------ |
-| k3s-master-1 | master | pve          |
-| k3s-master-2 | master | pve          |
-| k3s-worker-1 | worker | pve          |
-| k3s-master-3 | master | pve-bckp     |
-| k3s-worker-2 | worker | pve-bckp     |
+| VM | Role |
+| --- | --- |
+| k3s-master-1 | master |
+| k3s-master-2 | master |
+| k3s-worker-1 | worker |
+| k3s-master-3 | master |
+| k3s-worker-2 | worker |
 
-Each VM is a full clone of an Ubuntu Noble (24.04) cloud-image template. Cloud-init injects the hostname, user credentials, and SSH keys at boot. An ED25519 SSH keypair is generated once and stored in Pulumi state — the public key goes into every VM, the private key is exported as a stack output for `k8s-bootstrap` to consume via StackReference.
+It also generates an ED25519 SSH keypair (stored in Pulumi state, public key injected via cloud-init) and registers each VM's MAC address as a DHCPv4 static mapping in pfSense.
 
-Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so nodes always receive their designated IPs.
+**Providers:** `@muhlba91/pulumi-proxmoxve`, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
 
-**Providers:** `@muhlba91/pulumi-proxmoxve` v8.x, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
+### 2. k8s-bootstrap
 
-**Exports:** `vmIds`, `ciRunnerPrivateKey`, `pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `master1Ip` … `worker2Ip`
+Starts the VMs via the Proxmox API and installs k3s over SSH: `k3s-master-1` initializes the cluster (`--cluster-init`, embedded etcd), `k3s-master-2`/`k3s-master-3` join as additional etcd nodes, and the two workers join as agents. Exports the cluster's `kubeconfig` as a secret for downstream stacks.
 
-### Stack 2: `k8s-bootstrap`
+### 3. k8s-infra
 
-Installs k3s on the Proxmox VMs over SSH using `@pulumi/command`. Reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference — no manual key distribution needed.
+Cluster-level infrastructure via Helm and `@pulumi/kubernetes`:
 
-Deployment sequence:
+| Component | What it does |
+| --- | --- |
+| NFS CSI Driver + `truenas-nfs` StorageClass | Dynamic NFS-backed PVCs from a TrueNAS share |
+| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) |
+| cert-manager + `letsencrypt-prod` ClusterIssuer | TLS certs via Let's Encrypt, DNS-01 through Cloudflare |
+| kube-vip + kube-vip-cloud-provider | Announces a virtual IP and allocates it to LoadBalancer Services (e.g. Traefik) |
 
-1. Start all 5 VMs via the Proxmox REST API
-2. Wait for port 22 to open on each node
-3. Install k3s on `k3s-master-1` with `--cluster-init` (embedded etcd)
-4. Join `k3s-master-2` and `k3s-master-3` as additional etcd nodes
-5. Join `k3s-worker-1` and `k3s-worker-2` as agent nodes
-6. Read `/etc/rancher/k3s/k3s.yaml` from master-1, patch the server URL, export as a secret stack output
+### 4. k8s-apps
 
-**Exports:** `kubeconfig` (secret)
+Application workloads exposed at `app.<domain>` via Traefik with a Let's Encrypt cert.
 
-### Stack 3: `k8s-infra`
+### 5. monitoring
 
-Deploys cluster-level infrastructure via Helm and the `@pulumi/kubernetes` provider. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
-
-| Component            | What it does                                                               |
-| -------------------- | -------------------------------------------------------------------------- |
-| NFS CSI Driver       | `csi-driver-nfs` Helm chart (v4.12.0) — enables dynamic NFS-backed PVCs    |
-| TrueNAS StorageClass | `truenas-nfs` StorageClass backed by an NFS share on TrueNAS               |
-| cert-manager         | `cert-manager` Helm chart — certificate lifecycle management               |
-| ClusterIssuer        | `letsencrypt-prod` — DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
+`kube-prometheus-stack` Helm release — Prometheus (20Gi Longhorn PVC) and Grafana (2Gi PVC, ingress at `grafana.<domain>`).
 
 ## Prerequisites
 
-- [Pulumi CLI](https://www.pulumi.com/docs/install/) installed
-- Node.js 18+ and npm
-- Two Proxmox nodes with API tokens
-- pfSense with REST API credentials (used for DHCPv4 static mapping)
-- TrueNAS with an NFS share (no API key needed — CSI driver connects via NFS protocol directly)
-- A Cloudflare account with an API token scoped to DNS edit on your zone
+- [Pulumi CLI](https://www.pulumi.com/docs/install/), Node.js 24+ and npm
+- One or more Proxmox nodes with API tokens
+- pfSense (DHCPv4 static mappings)
+- TrueNAS with an NFS share (connected directly — no API key needed)
+- A domain with DNS managed by Cloudflare (cert-manager DNS-01 + ingress hostnames)
 - A self-hosted Pulumi state backend (PostgreSQL connection string)
-- Gitea instance for CI/CD (optional for local use)
+- Gitea instance for CI/CD (optional for local-only use)
 
 ## Getting started
 
-Deploy the stacks **in order**. Each stack must be fully deployed before the next one runs.
-
-### Stack 1 — `proxmox-infra`
-
-```bash
-cd proxmox-infra
-npm install      # also compiles the bundled pfSense SDK
-pulumi stack init dev
-```
-
-Set secrets:
-
-```bash
-# Proxmox API credentials
-pulumi config set --secret pve1Endpoint  https://<proxmox-host-1>:8006
-pulumi config set --secret pve1ApiToken  <user>@pam!<token-id>=<uuid>
-pulumi config set --secret pve2Endpoint  https://<proxmox-host-2>:8006
-pulumi config set --secret pve2ApiToken  <user>@pam!<token-id>=<uuid>
-
-# VM credentials
-pulumi config set --secret k3sVmPassword  <vm-password>
-pulumi config set --secret sshPvePublicKey "ssh-ed25519 AAAA..."
-
-# pfSense credentials (used for DHCPv4 static mappings)
-pulumi config set --secret pfSenseUrl      https://<pfsense-host>
-pulumi config set --secret pfSenseUser     <admin-username>
-pulumi config set --secret pfSensePassword <admin-password>
-
-# Static IP addresses assigned to each k3s node
-pulumi config set --secret master1Ip  <ip-for-k3s-master-1>
-pulumi config set --secret master2Ip  <ip-for-k3s-master-2>
-pulumi config set --secret worker1Ip  <ip-for-k3s-worker-1>
-pulumi config set --secret master3Ip  <ip-for-k3s-master-3>
-pulumi config set --secret worker2Ip  <ip-for-k3s-worker-2>
-```
-
-Deploy:
+Deploy stacks **in order** — each must be fully `pulumi up`'d before the next stack's `StackReference` resolves. Set the shared backend env vars once per shell:
 
 ```bash
 export PULUMI_BACKEND_URL=postgresql://<user>:<pass>@<host>/<db>
 export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
-
-pulumi preview    # inspect before touching anything
-pulumi up --yes
 ```
 
-### Stack 2 — `k8s-bootstrap`
+Then for each stack, in order:
 
 ```bash
-cd ../k8s-bootstrap
+cd 0N-<stack-dir>
 npm install
 pulumi stack init dev
-
-pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
-```
-
-Node IPs and Proxmox credentials are read automatically from `proxmox-infra` outputs via StackReference — do not set them here.
-
-```bash
+pulumi config set ...   # required keys and secrets
 pulumi up --yes
 ```
 
-### Stack 3 — `k8s-infra`
+Stack-specific notes:
 
-**TrueNAS one-time setup** (before deploying this stack):
-
-1. Create dataset `tank/k8s`
-2. Add an NFS share for that dataset
-
-    - Acl Type: Set this to POSIX (or generic Unix permissions) rather than SMB/NFSv4 ACLs if you are on TrueNAS SCALE, as Kubernetes handles basic Unix permissions natively.
-    - Maproot User / Maproot Group: Set this to root and root
-    - Allowed Hosts/Networks: Restrict this share specifically to the IP addresses or the CIDR block of your Proxmox K3s nodes (e.g., 192.168.1.50)
-
-3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
-
-```bash
-cd ../k8s-infra
-npm install
-pulumi stack init dev
-
-# kubeconfig from k8s-bootstrap
-KUBECONFIG=$(cd ../k8s-bootstrap && pulumi stack output kubeconfig --show-secrets)
-pulumi config set --secret kubeconfig "$KUBECONFIG"
-
-# TrueNAS NFS
-pulumi config set --secret truenasHost    <truenas-ip>
-pulumi config set --secret truenasNfsPath /mnt/tank/k8s
-
-# cert-manager + Cloudflare DNS-01
-pulumi config set --secret cloudflareApiToken <cf-token>
-pulumi config set --secret letsencryptEmail   <your-email>
-
-pulumi up --yes
-```
+1. **proxmox-infra** — set Proxmox API tokens, pfSense credentials, VM credentials/SSH public key, and a static IP per node.
+2. **k8s-bootstrap** — set `k3sToken` (e.g. `openssl rand -hex 32`) and a static IP per node. Proxmox credentials and the SSH key come automatically from `proxmox-infra`.
+3. **k8s-infra** — before deploying, create an NFS dataset/share on TrueNAS (POSIX ACLs, `root`/`root` maproot, restricted to your k3s nodes). Set the TrueNAS, Cloudflare, and `kubeVipAddress`/`kubeVipInterface` config.
+4. **k8s-apps** — set `domain`, the base domain used for ingress hostnames.
+5. **monitoring** — set `grafanaAdminPassword`.
 
 ## CI/CD (Gitea Actions)
 
-One workflow per stack under `.gitea/workflows/`. Each triggers on changes to its own stack directory.
+One workflow per stack, `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
 
-| Event                 | Action                         |
-| --------------------- | ------------------------------ |
-| Pull request → `main` | `pulumi preview` (read-only)   |
-| Push to `main`        | `pulumi refresh` + `pulumi up` |
-| Manual trigger        | `pulumi refresh` + `pulumi up` |
+| Trigger | Stacks | Action |
+| --- | --- | --- |
+| PR → `main` | all | `pulumi preview` (read-only) |
+| Push to `main` | 03–05 | `pulumi refresh` + `pulumi up` |
+| Manual dispatch | all | `pulumi refresh` + `pulumi up` |
 
-### Required Gitea secrets
+Stacks 01/02 (VM provisioning, cluster bootstrap) deploy **only** via manual dispatch — they're riskier to automate.
 
-Configure under **Settings → Actions → Secrets**:
+### Required secrets
 
-| Secret                          | Used by                     | Description                                    |
-| ------------------------------- | --------------------------- | ---------------------------------------------- |
-| `PULUMI_BACKEND_URL`            | all workflows               | PostgreSQL connection string for state backend |
-| `PULUMI_CONFIG_PASSPHRASE`      | all workflows               | Passphrase to decrypt secrets                  |
-| `PROXMOX_INFRA_PULUMI_DEV_YAML` | `deploy-proxmox-infra.yaml` | Base64-encoded `proxmox-infra/Pulumi.dev.yaml` |
-| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | `deploy-k8s-bootstrap.yaml` | Base64-encoded `k8s-bootstrap/Pulumi.dev.yaml` |
-| `K8S_INFRA_PULUMI_DEV_YAML`     | `deploy-k8s-infra.yaml`     | Base64-encoded `k8s-infra/Pulumi.dev.yaml`     |
+| Secret | Used by |
+| --- | --- |
+| `PULUMI_BACKEND_URL`, `PULUMI_CONFIG_PASSPHRASE` | all workflows |
+| `PROXMOX_INFRA_PULUMI_DEV_YAML` | 01-deploy-proxmox-infra |
+| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | 02-deploy-k8s-bootstrap |
+| `K8S_INFRA_PULUMI_DEV_YAML` | 03-deploy-k8s-infra |
+| `K8S_APPS_PULUMI_DEV_YAML` | 04-deploy-k8s-apps |
+| `MONITORING_PULUMI_DEV_YAML` | 05-deploy-monitoring |
 
-`Pulumi.dev.yaml` files are gitignored (they contain your encryption salt). Whenever one changes, re-encode and paste into the corresponding Gitea secret:
+Each `<STACK>_PULUMI_DEV_YAML` secret is a base64-encoded copy of that stack's gitignored `Pulumi.dev.yaml`. After changing local config, re-encode and update the secret:
 
 ```bash
-base64 -w 0 proxmox-infra/Pulumi.dev.yaml
-base64 -w 0 k8s-bootstrap/Pulumi.dev.yaml
-base64 -w 0 k8s-infra/Pulumi.dev.yaml
+base64 -w 0 0N-<stack>/Pulumi.dev.yaml
 ```
 
-## Adapting this as a template
+## Using this as a template
 
 1. Fork or copy the repo
-2. Update Proxmox node names (`pve`, `pve-bckp`) and datastore IDs in `proxmox-infra/index.ts`
-3. Add or remove VMs from the `nodeConfigs` array
-4. Set your own secrets with `pulumi config set --secret`
-5. Point the CI/CD workflows at your own Gitea instance
+2. Update Proxmox node names/datastores and the `nodeConfigs` array in `01-proxmox-infra/index.ts` for your VM topology
+3. Set your own config/secrets per stack with `pulumi config set [--secret]` — see each stack's `CLAUDE.md` for the full list of required keys
+4. Point the CI/CD workflows at your own Gitea/GitOps instance and update the secrets above
 
 ## Roadmap
 
-- LXC container management
-- `k8s-apps` stack — application deployments on the cluster
-- Docker / Compose stack provisioning
-
 - Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault
-- Add a third bare metal Proxmox node for true 3-node HA parity
+- Add a third bare-metal Proxmox node for true 3-node HA parity