docs: updated documentation to current progress

01-proxmox-infra/CLAUDE.md

@@ -1,62 +1,30 @@
-# CLAUDE.md
+# proxmox-infra
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+This file provides guidance to Claude Code when working in `01-proxmox-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
 
 ## Project Overview
 
-This is a Pulumi TypeScript project (`proxmox-infra`) for provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider. The stack name is `dev`.
+Pulumi TypeScript project provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider, plus pfSense DHCPv4 static mappings via a locally-bundled `@pulumi/pfsense` SDK.
 
 ## Common Commands
 
 ```bash
-# Install dependencies
+# Install deps and compile the bundled pfSense SDK
 npm install
 
-# Preview infrastructure changes
+# Sync Pulumi state with actual Proxmox state — run before `up` if VMs were
-pulumi preview
+# changed manually in the Proxmox GUI
-
-# Sync Pulumi state with actual Proxmox state (run before up if resources were changed manually)
 pulumi refresh --yes
 
-# Deploy infrastructure
 pulumi refresh --yes && pulumi up --yes
-
-# Destroy infrastructure
-pulumi destroy
-
-# View current stack outputs
-pulumi stack output
-
-# View stack config
-pulumi config
 ```
 
 ## Architecture
 
-- **Entry point**: `index.ts` — all Pulumi resources are declared here
+- **Entry point**: `index.ts` — all resources declared here.
-- **Provider**: `@muhlba91/pulumi-proxmoxve` v8.x — community Proxmox provider (not an official Pulumi provider)
+- **Providers**: two `@muhlba91/pulumi-proxmoxve` providers — `pveProvider` (node `pve`) and `pveBckpProvider` (node `pve-bckp`). Always pass the correct provider when adding resources.
-- **Stack**: `dev` — configured in `Pulumi.dev.yaml`
+- pfSense SDK lives in `sdks/pfsense/` (gitignored, regenerated by `npm install` / CI via `pulumi package add terraform-provider marshallford/pfsense 0.22.0`).
-- **Runtime**: Node.js with `npm`, TypeScript compiled to `bin/` (excluded from git)
 
-## CI/CD (Gitea Actions)
+## CI/CD
 
-Workflow file: `../.gitea/workflows/deploy-proxmox-infra.yaml`
+Workflow: `../.gitea/workflows/01-deploy-proxmox-infra.yaml`. Secret: `PROXMOX_INFRA_PULUMI_DEV_YAML`. In addition to the shared pattern, this workflow regenerates the pfSense SDK (`pulumi package add terraform-provider marshallford/pfsense 0.22.0`) before preview/deploy.
-
-Triggers: push to `main` and pull requests targeting `main`, scoped to changes under `proxmox-infra/**` or `.gitea/workflows/**`.
-
-- **Pull request** → `pulumi preview` (no changes deployed)
-- **Push to main** → `pulumi refresh` then `pulumi up`
-
-Secrets required in Gitea (`Settings → Actions → Secrets`):
-
-- `PULUMI_BACKEND_URL` — PostgreSQL connection string for the self-hosted state backend
-- `PULUMI_CONFIG_PASSPHRASE` — passphrase used to decrypt secrets in `Pulumi.dev.yaml`
-- `PULUMI_DEV_YAML` — base64-encoded content of `Pulumi.dev.yaml`; update manually whenever the file changes: `base64 -w 0 Pulumi.dev.yaml`
-
-## Key Notes
-
-- Credentials for both Proxmox nodes are stored as encrypted secrets in `Pulumi.dev.yaml` and decrypted at runtime using `PULUMI_CONFIG_PASSPHRASE`. Do not pass Proxmox credentials via environment variables — the code uses `config.requireSecret()`.
-- There are two Proxmox providers: `pveProvider` (main node `pve`) and `pveBckpProvider` (backup node `pve-bckp`). Always pass the correct provider when adding resources.
-- `Pulumi.dev.yaml` contains the encryption salt — never delete it or secrets become unrecoverable.
-- TypeScript is compiled with strict mode, `nodenext` module resolution, and `noImplicitReturns` — all functions must have explicit return types when TypeScript cannot infer them.
-- Don't add a co-author when committing to git.

02-k8s-bootstrap/CLAUDE.md

@@ -1,22 +1,22 @@
 # k8s-bootstrap
 
-Bootstraps a k3s cluster on the 5 Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` remote.Command.
+This file provides guidance to Claude Code when working in `02-k8s-bootstrap`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` `remote.Command`.
 
 ## How it works
 
-1. Starts all 5 VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
+1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start`
 2. Waits for port 22 to open on each VM (bash `/dev/tcp`)
 3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san <master1Ip>`
 4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes
 5. Joins `k3s-worker-1` and `k3s-worker-2` as agent nodes
-6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches server URL, exports as secret stack output
+6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL, and exports it as the secret stack output `kubeconfig`
 
-VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via `StackReference` — no manual setup needed for those.
+VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those.
 
 ## Required Pulumi config
 
-Run in this directory after `pulumi stack init dev`:
-
 ```bash
 # Pre-shared k3s token — any strong random string
 pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
@@ -29,59 +29,12 @@ pulumi config set worker1Ip "192.168.1.x"
 pulumi config set worker2Ip "192.168.1.x"
 ```
 
-Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from the `proxmox-infra` stack outputs via StackReference — do **not** set them here.
+Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here.
-
-## Deployment order
-
-`proxmox-infra` must be deployed **before** k8s-bootstrap. The proxmox-infra stack exports the Proxmox credentials and VM IDs that k8s-bootstrap reads via StackReference. If proxmox-infra hasn't been re-deployed after its latest code changes, those outputs won't exist and k8s-bootstrap will fail.
-
-In CI, both workflows run as previews on PRs (no deploy). Trigger `workflow_dispatch` on `deploy-proxmox-infra` first, then run k8s-bootstrap.
-
-After setting config, re-encode `Pulumi.dev.yaml` and update the Gitea secret `K8S_BOOTSTRAP_DEV_YAML`:
-```bash
-base64 -w 0 Pulumi.dev.yaml
-# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML
-```
 
 ## Prerequisites
 
-`proxmox-infra` must be deployed first. On first `pulumi up`, it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to `~/.ssh/authorized_keys`) before k8s-bootstrap can SSH in.
+`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in.
 
-## After first run
+## CI/CD
 
-The `kubeconfig` stack output must be propagated to `k8s-infra` and `k8s-apps`:
+Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs.
-
-```bash
-KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets)
-cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG"
-cd ../k8s-apps  && pulumi config set --secret kubeconfig "$KUBECONFIG"
-```
-
-Then re-encode each `Pulumi.dev.yaml` and update the corresponding Gitea secrets
-(`K8S_INFRA_DEV_YAML`, `K8S_APPS_DEV_YAML`):
-
-```bash
-base64 -w 0 k8s-infra/Pulumi.dev.yaml
-base64 -w 0 k8s-apps/Pulumi.dev.yaml
-```
-
-## k8s-infra config changes (democratic-csi → NFS CSI)
-
-The `k8s-infra` stack no longer needs `truenasApiKey` or `truenasDataset`.
-Replace them with:
-
-```bash
-cd ../k8s-infra
-pulumi config set truenasHost "192.168.1.x"
-pulumi config set truenasNfsPath "/mnt/tank/k8s"
-# Remove old keys if present:
-pulumi config rm truenasApiKey
-pulumi config rm truenasDataset
-```
-
-## TrueNAS one-time setup (before deploying k8s-infra)
-
-1. Create a dataset: `tank/k8s`
-2. Add an NFS share for that dataset
-3. In Network → Allowed Networks, permit `192.168.1.0/24`
-4. No API key required — the NFS CSI driver connects directly via NFS protocol

03-k8s-infra/CLAUDE.md

@@ -0,0 +1,55 @@
+# k8s-infra
+
+This file provides guidance to Claude Code when working in `03-k8s-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Cluster-level infrastructure for the k3s homelab cluster, deployed via Helm and `@pulumi/kubernetes`. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi refresh --yes && pulumi up --yes
+```
+
+CI installs the `helm` CLI (`azure/setup-helm`) before running Pulumi.
+
+## Architecture
+
+| Component | What it does |
+| --- | --- |
+| NFS CSI Driver | `csi-driver-nfs` Helm chart — enables NFS-backed PVCs |
+| `truenas-nfs` StorageClass | Backed by an NFS share on TrueNAS (`Retain` reclaim policy) |
+| cert-manager | Certificate lifecycle management |
+| `letsencrypt-prod` ClusterIssuer | DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
+| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) — used by app/monitoring PVCs |
+| kube-vip | DaemonSet that ARP-announces a virtual IP for LoadBalancer Services |
+| kube-vip-cloud-provider | Allocates LoadBalancer IPs from a single-address pool (`kubeVipAddress`) — k3s's built-in ServiceLB is not used |
+| `traefik` ServicePatch | Pins Traefik's LoadBalancer IP to `kubeVipAddress` |
+
+**Export**: `storageClass = "truenas-nfs"`.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret truenasHost      <truenas-ip>
+pulumi config set --secret truenasNfsPath   /mnt/pool/k8s
+pulumi config set --secret cloudflareApiToken <cf-token>
+pulumi config set --secret letsencryptEmail   <your-email>
+pulumi config set --secret kubeVipAddress   <vip-ip>
+pulumi config set kubeVipInterface          <e.g. eth0>
+```
+
+`kubeconfig` is read automatically via StackReference — do not set it here.
+
+## TrueNAS one-time setup (before deploying)
+
+1. Create dataset `pool/k8s`
+2. Add an NFS share for that dataset:
+   - ACL type: POSIX/generic Unix permissions (not SMB/NFSv4 ACLs) on TrueNAS SCALE
+   - Maproot User / Group: `root` / `root`
+   - Allowed Hosts/Networks: restrict to the k3s node IPs/CIDR
+3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/03-deploy-k8s-infra.yaml`. Secret: `K8S_INFRA_PULUMI_DEV_YAML`.

03-k8s-infra/index.ts

@@ -3,11 +3,11 @@ import * as k8s from "@pulumi/kubernetes";
 
 const config = new pulumi.Config();
 
-const infraRef = new pulumi.StackReference(
+const bootstrapRef = new pulumi.StackReference(
   `${pulumi.getOrganization()}/k8s-bootstrap/dev`,
 );
 
-const kubeconfig = infraRef.requireOutput("kubeconfig");
+const kubeconfig = bootstrapRef.requireOutput("kubeconfig");
 const truenasHost = config.requireSecret("truenasHost");
 const truenasNfsPath = config.requireSecret("truenasNfsPath");
 const cloudflareToken = config.requireSecret("cloudflareApiToken");

04-k8s-apps/CLAUDE.md

@@ -0,0 +1,32 @@
+# k8s-apps
+
+This file provides guidance to Claude Code when working in `04-k8s-apps`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Application workloads for the k3s homelab cluster. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi up --yes
+```
+
+## Architecture
+
+- **heimdall** — application dashboard (`lscr.io/linuxserver/heimdall`), namespace `heimdall`
+  - PVC `heimdall-config` (50Mi, `longhorn` StorageClass)
+  - Deployment + ClusterIP Service
+  - Ingress (`traefik` class, `cert-manager.io/cluster-issuer: letsencrypt-prod`) at `heimdall.<domain>`
+  - deployed with one replica because heimdall is not able to work with multiple replicas. Works with node failure detection and automatic rescheduling.
+
+**Export**: `domain` (secret) — consumed by `monitoring` via StackReference for its own ingress hostnames.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret domain <your-domain>
+```
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/04-deploy-k8s-apps.yaml`. Secret: `K8S_APPS_PULUMI_DEV_YAML`.

04-k8s-apps/index.ts

@@ -3,7 +3,6 @@ import * as k8s from "@pulumi/kubernetes";
 
 const config = new pulumi.Config();
 
-//fetch credentials from k8s-bootstrap
 const bootstrapRef = new pulumi.StackReference(
   `${pulumi.getOrganization()}/k8s-bootstrap/dev`,
 );

05-monitoring/CLAUDE.md

@@ -0,0 +1,34 @@
+# monitoring
+
+This file provides guidance to Claude Code when working in `05-monitoring`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD.
+
+Monitoring stack (Prometheus + Grafana) via the `kube-prometheus-stack` Helm chart. Reads `kubeconfig` from `k8s-bootstrap` and `domain` from `k8s-apps`, both via StackReference.
+
+## Common Commands
+
+```bash
+npm install
+pulumi up --yes
+```
+
+## Architecture
+
+- Namespace `monitoring`
+- `kube-prometheus-stack` Helm release (chart v86.2.0)
+  - Grafana: ingress at `grafana.<domain>` (`traefik`, `letsencrypt-prod`), `longhorn` PVC (2Gi)
+  - Prometheus: `longhorn` PVC (20Gi) via `volumeClaimTemplate`
+  - Alertmanager disabled
+
+## Key Notes
+
+- The `monitoring` namespace and the Helm release both use the Pulumi `import` resource option (`monitoring`, `monitoring/kube-prometheus-stack`) — they were adopted from pre-existing cluster resources rather than created fresh. Be careful with renames or changes that would force a replace.
+
+## Required Pulumi config
+
+```bash
+pulumi config set --secret grafanaAdminPassword <password>
+```
+
+## CI/CD
+
+Workflow: `../.gitea/workflows/05-deploy-monitoring.yaml`. Secret: `MONITORING_PULUMI_DEV_YAML`.

CLAUDE.md

@@ -0,0 +1,64 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code when working anywhere in this repository. Each stack directory (`01-proxmox-infra`, `02-k8s-bootstrap`, etc.) has its own `CLAUDE.md` with stack-specific commands and architecture — Claude Code loads both this file and the stack's file automatically when working inside a stack directory.
+
+## Project overview
+
+Pulumi (TypeScript) IaC for a Proxmox-based k3s homelab. Independent Pulumi projects ("stacks"), each with its own `Pulumi.yaml`/`package.json`, deployed in order and linked via `pulumi.StackReference`:
+
+| # | Directory | Pulumi stack | Purpose |
+| --- | --- | --- | --- |
+| 1 | `01-proxmox-infra` | `proxmox-infra` | Provisions k3s VMs + DHCP on Proxmox/pfSense |
+| 2 | `02-k8s-bootstrap` | `k8s-bootstrap` | Installs k3s on the VMs over SSH |
+| 3 | `03-k8s-infra` | `k8s-infra` | Cluster infra: storage, cert-manager, kube-vip |
+| 4 | `04-k8s-apps` | `k8s-apps` | Application workloads (e.g. heimdall) |
+| 5 | `05-monitoring` | `monitoring` | kube-prometheus-stack (Prometheus + Grafana) |
+
+All stacks use the Pulumi stack name `dev`.
+
+## Deployment order & StackReferences
+
+```
+proxmox-infra → k8s-bootstrap → k8s-infra
+                              ↘ k8s-apps → monitoring
+```
+
+- `k8s-bootstrap` reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference.
+- `k8s-infra`, `k8s-apps`, and `monitoring` all read `kubeconfig` from `k8s-bootstrap` via StackReference (`requireOutput("kubeconfig")`) — never set `kubeconfig` as Pulumi config in these stacks.
+- `monitoring` additionally reads `domain` from `k8s-apps` via StackReference.
+- StackReference format: `${pulumi.getOrganization()}/<stack-name>/dev`.
+
+A stack's outputs must exist (i.e. it must have been deployed) before any downstream stack referencing it will succeed.
+
+## Shared conventions
+
+- **TypeScript**: `strict: true`, `target: es2024`, `module`/`moduleResolution: nodenext`, `noImplicitReturns`, `noFallthroughCasesInSwitch` — functions need explicit return types where TS can't infer them.
+- **Secrets**: read via `config.requireSecret()`, never via environment variables or hardcoded values.
+- **`Pulumi.dev.yaml`**: gitignored (contains the encryption salt) — never delete it or secrets become unrecoverable. Encrypted with `PULUMI_CONFIG_PASSPHRASE`.
+- **Backend**: self-hosted Postgres state backend, set via `PULUMI_BACKEND_URL`.
+- Don't add a co-author when committing to git.
+
+## Common commands (run from a stack directory)
+
+```bash
+npm install
+pulumi preview
+pulumi up --yes
+pulumi destroy
+pulumi stack output
+pulumi config
+```
+
+Requires `PULUMI_BACKEND_URL` and `PULUMI_CONFIG_PASSPHRASE` in the environment (or `pulumi login`).
+
+## CI/CD (Gitea Actions)
+
+One workflow per stack: `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
+
+- **Pull request → `main`**: `pulumi preview` only.
+- `01`/`02` (VM provisioning / cluster bootstrap — riskier to automate): deploy only via manual `workflow_dispatch`.
+- `03`–`05`: deploy on push to `main` *or* `workflow_dispatch` — `pulumi refresh` then `pulumi up`.
+- Each workflow restores `Pulumi.dev.yaml` from a base64-encoded Gitea secret (`<STACK>_PULUMI_DEV_YAML`). After changing local config, re-encode and update the secret:
+  ```bash
+  base64 -w 0 0N-<stack>/Pulumi.dev.yaml
+  ```

README.md

@@ -1,239 +1,154 @@
 # Homelab Infrastructure as Code
 
-A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks with version-controlled, reproducible infrastructure — a highly available k3s cluster across multiple Proxmox nodes, bootstrapped and configured end-to-end from code.
+A Pulumi-based IaC project for a Proxmox homelab: a highly-available k3s cluster, its cluster-level infrastructure, and the applications running on top of it — all provisioned and configured from code.
 
 This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab.
 
 ## Why IaC for a homelab?
 
-- **Reproducibility** — rebuild your entire environment from scratch with a single command
+- **Reproducibility** — rebuild the entire environment from scratch from code
 - **Version history** — every change is tracked in git; roll back at any time
-- **Auditability** — diff infrastructure changes before applying them (`pulumi preview`)
+- **Auditability** — `pulumi preview` diffs infrastructure changes before they're applied
-- **Automation** — CI/CD handles deploys; no manual SSH into nodes
+- **Automation** — CI/CD handles deploys; no manual SSH into nodes for routine changes
-- **Portability** — swap node names, datastores, or credentials without touching logic
+- **Portability** — swap node names, datastores, domains, or credentials without touching logic
 
 ## Repository layout
 
 ```
 .
-├── proxmox-infra/          # Stack 1 — VMs & DHCP on Proxmox + pfSense
+├── 01-proxmox-infra/   # Stack 1 — k3s VMs + DHCP on Proxmox/pfSense
-│   ├── index.ts
+├── 02-k8s-bootstrap/   # Stack 2 — install k3s on the VMs over SSH
-│   ├── Pulumi.yaml
+├── 03-k8s-infra/       # Stack 3 — storage, cert-manager, kube-vip
-│   └── sdks/pfsense/       # Locally bundled @pulumi/pfsense SDK
+├── 04-k8s-apps/        # Stack 4 — application workloads
-├── k8s-bootstrap/          # Stack 2 — Install k3s on the VMs over SSH
+├── 05-monitoring/      # Stack 5 — kube-prometheus-stack (Prometheus + Grafana)
-│   ├── index.ts
+├── .gitea/workflows/   # CI/CD — one workflow per stack
-│   └── Pulumi.yaml
+└── CLAUDE.md           # Conventions & architecture notes (root + one per stack)
-├── k8s-infra/              # Stack 3 — Cluster-level infrastructure via Helm
-│   ├── index.ts
-│   └── Pulumi.yaml
-└── .gitea/workflows/       # Gitea Actions — one workflow per stack
 ```
 
-## Stack overview
+Each stack is an independent Pulumi project (own `Pulumi.yaml`/`package.json`), all using the Pulumi stack name `dev`.
 
-The three stacks are deployed in order and linked via Pulumi StackReferences — outputs from one stack flow automatically into the next.
+## Stacks
+
+Stacks are deployed in order and linked via `pulumi.StackReference` — outputs from one stack flow automatically into the next, so a stack must be deployed before anything depending on it will work.
 
 ```
 proxmox-infra → k8s-bootstrap → k8s-infra
+                              ↘ k8s-apps → monitoring
 ```
 
-### Stack 1: `proxmox-infra`
+### 1. proxmox-infra
 
-Provisions a 5-node k3s cluster spread across two Proxmox hosts:
+Provisions a 5-node k3s cluster across two Proxmox hosts, each VM a clone of an Ubuntu Noble (24.04) cloud-image template:
 
-| VM name      | Role   | Proxmox node |
+| VM | Role |
-| ------------ | ------ | ------------ |
+| --- | --- |
-| k3s-master-1 | master | pve          |
+| k3s-master-1 | master |
-| k3s-master-2 | master | pve          |
+| k3s-master-2 | master |
-| k3s-worker-1 | worker | pve          |
+| k3s-worker-1 | worker |
-| k3s-master-3 | master | pve-bckp     |
+| k3s-master-3 | master |
-| k3s-worker-2 | worker | pve-bckp     |
+| k3s-worker-2 | worker |
 
-Each VM is a full clone of an Ubuntu Noble (24.04) cloud-image template. Cloud-init injects the hostname, user credentials, and SSH keys at boot. An ED25519 SSH keypair is generated once and stored in Pulumi state — the public key goes into every VM, the private key is exported as a stack output for `k8s-bootstrap` to consume via StackReference.
+It also generates an ED25519 SSH keypair (stored in Pulumi state, public key injected via cloud-init) and registers each VM's MAC address as a DHCPv4 static mapping in pfSense.
 
-Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so nodes always receive their designated IPs.
+**Providers:** `@muhlba91/pulumi-proxmoxve`, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
 
-**Providers:** `@muhlba91/pulumi-proxmoxve` v8.x, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
+### 2. k8s-bootstrap
 
-**Exports:** `vmIds`, `ciRunnerPrivateKey`, `pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `master1Ip` … `worker2Ip`
+Starts the VMs via the Proxmox API and installs k3s over SSH: `k3s-master-1` initializes the cluster (`--cluster-init`, embedded etcd), `k3s-master-2`/`k3s-master-3` join as additional etcd nodes, and the two workers join as agents. Exports the cluster's `kubeconfig` as a secret for downstream stacks.
 
-### Stack 2: `k8s-bootstrap`
+### 3. k8s-infra
 
-Installs k3s on the Proxmox VMs over SSH using `@pulumi/command`. Reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference — no manual key distribution needed.
+Cluster-level infrastructure via Helm and `@pulumi/kubernetes`:
-
-Deployment sequence:
-
-1. Start all 5 VMs via the Proxmox REST API
-2. Wait for port 22 to open on each node
-3. Install k3s on `k3s-master-1` with `--cluster-init` (embedded etcd)
-4. Join `k3s-master-2` and `k3s-master-3` as additional etcd nodes
-5. Join `k3s-worker-1` and `k3s-worker-2` as agent nodes
-6. Read `/etc/rancher/k3s/k3s.yaml` from master-1, patch the server URL, export as a secret stack output
-
-**Exports:** `kubeconfig` (secret)
-
-### Stack 3: `k8s-infra`
-
-Deploys cluster-level infrastructure via Helm and the `@pulumi/kubernetes` provider. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
 
 | Component | What it does |
-| -------------------- | -------------------------------------------------------------------------- |
+| --- | --- |
-| NFS CSI Driver       | `csi-driver-nfs` Helm chart (v4.12.0) — enables dynamic NFS-backed PVCs    |
+| NFS CSI Driver + `truenas-nfs` StorageClass | Dynamic NFS-backed PVCs from a TrueNAS share |
-| TrueNAS StorageClass | `truenas-nfs` StorageClass backed by an NFS share on TrueNAS               |
+| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) |
-| cert-manager         | `cert-manager` Helm chart — certificate lifecycle management               |
+| cert-manager + `letsencrypt-prod` ClusterIssuer | TLS certs via Let's Encrypt, DNS-01 through Cloudflare |
-| ClusterIssuer        | `letsencrypt-prod` — DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
+| kube-vip + kube-vip-cloud-provider | Announces a virtual IP and allocates it to LoadBalancer Services (e.g. Traefik) |
+
+### 4. k8s-apps
+
+Application workloads exposed at `app.<domain>` via Traefik with a Let's Encrypt cert.
+
+### 5. monitoring
+
+`kube-prometheus-stack` Helm release — Prometheus (20Gi Longhorn PVC) and Grafana (2Gi PVC, ingress at `grafana.<domain>`).
 
 ## Prerequisites
 
-- [Pulumi CLI](https://www.pulumi.com/docs/install/) installed
+- [Pulumi CLI](https://www.pulumi.com/docs/install/), Node.js 24+ and npm
-- Node.js 18+ and npm
+- One or more Proxmox nodes with API tokens
-- Two Proxmox nodes with API tokens
+- pfSense (DHCPv4 static mappings)
-- pfSense with REST API credentials (used for DHCPv4 static mapping)
+- TrueNAS with an NFS share (connected directly — no API key needed)
-- TrueNAS with an NFS share (no API key needed — CSI driver connects via NFS protocol directly)
+- A domain with DNS managed by Cloudflare (cert-manager DNS-01 + ingress hostnames)
-- A Cloudflare account with an API token scoped to DNS edit on your zone
 - A self-hosted Pulumi state backend (PostgreSQL connection string)
-- Gitea instance for CI/CD (optional for local use)
+- Gitea instance for CI/CD (optional for local-only use)
 
 ## Getting started
 
-Deploy the stacks **in order**. Each stack must be fully deployed before the next one runs.
+Deploy stacks **in order** — each must be fully `pulumi up`'d before the next stack's `StackReference` resolves. Set the shared backend env vars once per shell:
-
-### Stack 1 — `proxmox-infra`
-
-```bash
-cd proxmox-infra
-npm install      # also compiles the bundled pfSense SDK
-pulumi stack init dev
-```
-
-Set secrets:
-
-```bash
-# Proxmox API credentials
-pulumi config set --secret pve1Endpoint  https://<proxmox-host-1>:8006
-pulumi config set --secret pve1ApiToken  <user>@pam!<token-id>=<uuid>
-pulumi config set --secret pve2Endpoint  https://<proxmox-host-2>:8006
-pulumi config set --secret pve2ApiToken  <user>@pam!<token-id>=<uuid>
-
-# VM credentials
-pulumi config set --secret k3sVmPassword  <vm-password>
-pulumi config set --secret sshPvePublicKey "ssh-ed25519 AAAA..."
-
-# pfSense credentials (used for DHCPv4 static mappings)
-pulumi config set --secret pfSenseUrl      https://<pfsense-host>
-pulumi config set --secret pfSenseUser     <admin-username>
-pulumi config set --secret pfSensePassword <admin-password>
-
-# Static IP addresses assigned to each k3s node
-pulumi config set --secret master1Ip  <ip-for-k3s-master-1>
-pulumi config set --secret master2Ip  <ip-for-k3s-master-2>
-pulumi config set --secret worker1Ip  <ip-for-k3s-worker-1>
-pulumi config set --secret master3Ip  <ip-for-k3s-master-3>
-pulumi config set --secret worker2Ip  <ip-for-k3s-worker-2>
-```
-
-Deploy:
 
 ```bash
 export PULUMI_BACKEND_URL=postgresql://<user>:<pass>@<host>/<db>
 export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
-
-pulumi preview    # inspect before touching anything
-pulumi up --yes
 ```
 
-### Stack 2 — `k8s-bootstrap`
+Then for each stack, in order:
 
 ```bash
-cd ../k8s-bootstrap
+cd 0N-<stack-dir>
 npm install
 pulumi stack init dev
-
+pulumi config set ...   # required keys and secrets
-pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
-```
-
-Node IPs and Proxmox credentials are read automatically from `proxmox-infra` outputs via StackReference — do not set them here.
-
-```bash
 pulumi up --yes
 ```
 
-### Stack 3 — `k8s-infra`
+Stack-specific notes:
 
-**TrueNAS one-time setup** (before deploying this stack):
+1. **proxmox-infra** — set Proxmox API tokens, pfSense credentials, VM credentials/SSH public key, and a static IP per node.
-
+2. **k8s-bootstrap** — set `k3sToken` (e.g. `openssl rand -hex 32`) and a static IP per node. Proxmox credentials and the SSH key come automatically from `proxmox-infra`.
-1. Create dataset `tank/k8s`
+3. **k8s-infra** — before deploying, create an NFS dataset/share on TrueNAS (POSIX ACLs, `root`/`root` maproot, restricted to your k3s nodes). Set the TrueNAS, Cloudflare, and `kubeVipAddress`/`kubeVipInterface` config.
-2. Add an NFS share for that dataset
+4. **k8s-apps** — set `domain`, the base domain used for ingress hostnames.
-
+5. **monitoring** — set `grafanaAdminPassword`.
-    - Acl Type: Set this to POSIX (or generic Unix permissions) rather than SMB/NFSv4 ACLs if you are on TrueNAS SCALE, as Kubernetes handles basic Unix permissions natively.
-    - Maproot User / Maproot Group: Set this to root and root
-    - Allowed Hosts/Networks: Restrict this share specifically to the IP addresses or the CIDR block of your Proxmox K3s nodes (e.g., 192.168.1.50)
-
-3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
-
-```bash
-cd ../k8s-infra
-npm install
-pulumi stack init dev
-
-# kubeconfig from k8s-bootstrap
-KUBECONFIG=$(cd ../k8s-bootstrap && pulumi stack output kubeconfig --show-secrets)
-pulumi config set --secret kubeconfig "$KUBECONFIG"
-
-# TrueNAS NFS
-pulumi config set --secret truenasHost    <truenas-ip>
-pulumi config set --secret truenasNfsPath /mnt/tank/k8s
-
-# cert-manager + Cloudflare DNS-01
-pulumi config set --secret cloudflareApiToken <cf-token>
-pulumi config set --secret letsencryptEmail   <your-email>
-
-pulumi up --yes
-```
 
 ## CI/CD (Gitea Actions)
 
-One workflow per stack under `.gitea/workflows/`. Each triggers on changes to its own stack directory.
+One workflow per stack, `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
 
-| Event                 | Action                         |
+| Trigger | Stacks | Action |
-| --------------------- | ------------------------------ |
+| --- | --- | --- |
-| Pull request → `main` | `pulumi preview` (read-only)   |
+| PR → `main` | all | `pulumi preview` (read-only) |
-| Push to `main`        | `pulumi refresh` + `pulumi up` |
+| Push to `main` | 03–05 | `pulumi refresh` + `pulumi up` |
-| Manual trigger        | `pulumi refresh` + `pulumi up` |
+| Manual dispatch | all | `pulumi refresh` + `pulumi up` |
 
-### Required Gitea secrets
+Stacks 01/02 (VM provisioning, cluster bootstrap) deploy **only** via manual dispatch — they're riskier to automate.
 
-Configure under **Settings → Actions → Secrets**:
+### Required secrets
 
-| Secret                          | Used by                     | Description                                    |
+| Secret | Used by |
-| ------------------------------- | --------------------------- | ---------------------------------------------- |
+| --- | --- |
-| `PULUMI_BACKEND_URL`            | all workflows               | PostgreSQL connection string for state backend |
+| `PULUMI_BACKEND_URL`, `PULUMI_CONFIG_PASSPHRASE` | all workflows |
-| `PULUMI_CONFIG_PASSPHRASE`      | all workflows               | Passphrase to decrypt secrets                  |
+| `PROXMOX_INFRA_PULUMI_DEV_YAML` | 01-deploy-proxmox-infra |
-| `PROXMOX_INFRA_PULUMI_DEV_YAML` | `deploy-proxmox-infra.yaml` | Base64-encoded `proxmox-infra/Pulumi.dev.yaml` |
+| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | 02-deploy-k8s-bootstrap |
-| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | `deploy-k8s-bootstrap.yaml` | Base64-encoded `k8s-bootstrap/Pulumi.dev.yaml` |
+| `K8S_INFRA_PULUMI_DEV_YAML` | 03-deploy-k8s-infra |
-| `K8S_INFRA_PULUMI_DEV_YAML`     | `deploy-k8s-infra.yaml`     | Base64-encoded `k8s-infra/Pulumi.dev.yaml`     |
+| `K8S_APPS_PULUMI_DEV_YAML` | 04-deploy-k8s-apps |
+| `MONITORING_PULUMI_DEV_YAML` | 05-deploy-monitoring |
 
-`Pulumi.dev.yaml` files are gitignored (they contain your encryption salt). Whenever one changes, re-encode and paste into the corresponding Gitea secret:
+Each `<STACK>_PULUMI_DEV_YAML` secret is a base64-encoded copy of that stack's gitignored `Pulumi.dev.yaml`. After changing local config, re-encode and update the secret:
 
 ```bash
-base64 -w 0 proxmox-infra/Pulumi.dev.yaml
+base64 -w 0 0N-<stack>/Pulumi.dev.yaml
-base64 -w 0 k8s-bootstrap/Pulumi.dev.yaml
-base64 -w 0 k8s-infra/Pulumi.dev.yaml
 ```
 
-## Adapting this as a template
+## Using this as a template
 
 1. Fork or copy the repo
-2. Update Proxmox node names (`pve`, `pve-bckp`) and datastore IDs in `proxmox-infra/index.ts`
+2. Update Proxmox node names/datastores and the `nodeConfigs` array in `01-proxmox-infra/index.ts` for your VM topology
-3. Add or remove VMs from the `nodeConfigs` array
+3. Set your own config/secrets per stack with `pulumi config set [--secret]` — see each stack's `CLAUDE.md` for the full list of required keys
-4. Set your own secrets with `pulumi config set --secret`
+4. Point the CI/CD workflows at your own Gitea/GitOps instance and update the secrets above
-5. Point the CI/CD workflows at your own Gitea instance
 
 ## Roadmap
 
-- LXC container management
-- `k8s-apps` stack — application deployments on the cluster
-- Docker / Compose stack provisioning
-
 - Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault
-- Add a third bare metal Proxmox node for true 3-node HA parity
+- Add a third bare-metal Proxmox node for true 3-node HA parity