From 13549a8d3782b55d55b204256e888e68cf9916c1 Mon Sep 17 00:00:00 2001 From: kasun Date: Thu, 11 Jun 2026 22:36:20 +0200 Subject: [PATCH] docs: updated documentation to current progress --- 01-proxmox-infra/CLAUDE.md | 54 ++------ 02-k8s-bootstrap/CLAUDE.md | 67 ++-------- 03-k8s-infra/CLAUDE.md | 55 ++++++++ 03-k8s-infra/index.ts | 4 +- 04-k8s-apps/CLAUDE.md | 32 +++++ 04-k8s-apps/index.ts | 1 - 05-monitoring/CLAUDE.md | 34 +++++ CLAUDE.md | 64 ++++++++++ README.md | 249 ++++++++++++------------------------- 9 files changed, 290 insertions(+), 270 deletions(-) create mode 100644 03-k8s-infra/CLAUDE.md create mode 100644 04-k8s-apps/CLAUDE.md create mode 100644 05-monitoring/CLAUDE.md create mode 100644 CLAUDE.md diff --git a/01-proxmox-infra/CLAUDE.md b/01-proxmox-infra/CLAUDE.md index 7182cc2..6d10843 100644 --- a/01-proxmox-infra/CLAUDE.md +++ b/01-proxmox-infra/CLAUDE.md @@ -1,62 +1,30 @@ -# CLAUDE.md +# proxmox-infra -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file provides guidance to Claude Code when working in `01-proxmox-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD. ## Project Overview -This is a Pulumi TypeScript project (`proxmox-infra`) for provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider. The stack name is `dev`. +Pulumi TypeScript project provisioning VMs and LXC containers in Proxmox using the `@muhlba91/pulumi-proxmoxve` provider, plus pfSense DHCPv4 static mappings via a locally-bundled `@pulumi/pfsense` SDK. ## Common Commands ```bash -# Install dependencies +# Install deps and compile the bundled pfSense SDK npm install -# Preview infrastructure changes -pulumi preview - -# Sync Pulumi state with actual Proxmox state (run before up if resources were changed manually) +# Sync Pulumi state with actual Proxmox state — run before `up` if VMs were +# changed manually in the Proxmox GUI pulumi refresh --yes -# Deploy infrastructure pulumi refresh --yes && pulumi up --yes - -# Destroy infrastructure -pulumi destroy - -# View current stack outputs -pulumi stack output - -# View stack config -pulumi config ``` ## Architecture -- **Entry point**: `index.ts` — all Pulumi resources are declared here -- **Provider**: `@muhlba91/pulumi-proxmoxve` v8.x — community Proxmox provider (not an official Pulumi provider) -- **Stack**: `dev` — configured in `Pulumi.dev.yaml` -- **Runtime**: Node.js with `npm`, TypeScript compiled to `bin/` (excluded from git) +- **Entry point**: `index.ts` — all resources declared here. +- **Providers**: two `@muhlba91/pulumi-proxmoxve` providers — `pveProvider` (node `pve`) and `pveBckpProvider` (node `pve-bckp`). Always pass the correct provider when adding resources. +- pfSense SDK lives in `sdks/pfsense/` (gitignored, regenerated by `npm install` / CI via `pulumi package add terraform-provider marshallford/pfsense 0.22.0`). -## CI/CD (Gitea Actions) +## CI/CD -Workflow file: `../.gitea/workflows/deploy-proxmox-infra.yaml` - -Triggers: push to `main` and pull requests targeting `main`, scoped to changes under `proxmox-infra/**` or `.gitea/workflows/**`. - -- **Pull request** → `pulumi preview` (no changes deployed) -- **Push to main** → `pulumi refresh` then `pulumi up` - -Secrets required in Gitea (`Settings → Actions → Secrets`): - -- `PULUMI_BACKEND_URL` — PostgreSQL connection string for the self-hosted state backend -- `PULUMI_CONFIG_PASSPHRASE` — passphrase used to decrypt secrets in `Pulumi.dev.yaml` -- `PULUMI_DEV_YAML` — base64-encoded content of `Pulumi.dev.yaml`; update manually whenever the file changes: `base64 -w 0 Pulumi.dev.yaml` - -## Key Notes - -- Credentials for both Proxmox nodes are stored as encrypted secrets in `Pulumi.dev.yaml` and decrypted at runtime using `PULUMI_CONFIG_PASSPHRASE`. Do not pass Proxmox credentials via environment variables — the code uses `config.requireSecret()`. -- There are two Proxmox providers: `pveProvider` (main node `pve`) and `pveBckpProvider` (backup node `pve-bckp`). Always pass the correct provider when adding resources. -- `Pulumi.dev.yaml` contains the encryption salt — never delete it or secrets become unrecoverable. -- TypeScript is compiled with strict mode, `nodenext` module resolution, and `noImplicitReturns` — all functions must have explicit return types when TypeScript cannot infer them. -- Don't add a co-author when committing to git. +Workflow: `../.gitea/workflows/01-deploy-proxmox-infra.yaml`. Secret: `PROXMOX_INFRA_PULUMI_DEV_YAML`. In addition to the shared pattern, this workflow regenerates the pfSense SDK (`pulumi package add terraform-provider marshallford/pfsense 0.22.0`) before preview/deploy. diff --git a/02-k8s-bootstrap/CLAUDE.md b/02-k8s-bootstrap/CLAUDE.md index 19fd66e..f360f92 100644 --- a/02-k8s-bootstrap/CLAUDE.md +++ b/02-k8s-bootstrap/CLAUDE.md @@ -1,22 +1,22 @@ # k8s-bootstrap -Bootstraps a k3s cluster on the 5 Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` remote.Command. +This file provides guidance to Claude Code when working in `02-k8s-bootstrap`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD. + +Bootstraps a k3s cluster on the Proxmox VMs created by `proxmox-infra`. Starts VMs via the Proxmox REST API, then provisions k3s over SSH using `@pulumi/command` `remote.Command`. ## How it works -1. Starts all 5 VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start` +1. Starts all VMs via `POST /api2/json/nodes/{node}/qemu/{vmid}/status/start` 2. Waits for port 22 to open on each VM (bash `/dev/tcp`) 3. Installs k3s on `k3s-master-1` with `--cluster-init --tls-san ` 4. Joins `k3s-master-2` and `k3s-master-3` as embedded etcd nodes 5. Joins `k3s-worker-1` and `k3s-worker-2` as agent nodes -6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches server URL, exports as secret stack output +6. Reads `/etc/rancher/k3s/k3s.yaml` from master-1 via SSH, patches the server URL, and exports it as the secret stack output `kubeconfig` -VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via `StackReference` — no manual setup needed for those. +VM IDs and the CI runner SSH private key are read automatically from the `proxmox-infra` stack output via StackReference — no manual setup needed for those. ## Required Pulumi config -Run in this directory after `pulumi stack init dev`: - ```bash # Pre-shared k3s token — any strong random string pulumi config set --secret k3sToken "$(openssl rand -hex 32)" @@ -29,59 +29,12 @@ pulumi config set worker1Ip "192.168.1.x" pulumi config set worker2Ip "192.168.1.x" ``` -Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from the `proxmox-infra` stack outputs via StackReference — do **not** set them here. - -## Deployment order - -`proxmox-infra` must be deployed **before** k8s-bootstrap. The proxmox-infra stack exports the Proxmox credentials and VM IDs that k8s-bootstrap reads via StackReference. If proxmox-infra hasn't been re-deployed after its latest code changes, those outputs won't exist and k8s-bootstrap will fail. - -In CI, both workflows run as previews on PRs (no deploy). Trigger `workflow_dispatch` on `deploy-proxmox-infra` first, then run k8s-bootstrap. - -After setting config, re-encode `Pulumi.dev.yaml` and update the Gitea secret `K8S_BOOTSTRAP_DEV_YAML`: -```bash -base64 -w 0 Pulumi.dev.yaml -# Copy output → Gitea → Settings → Actions → Secrets → K8S_BOOTSTRAP_DEV_YAML -``` +Proxmox credentials (`pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`) are read automatically from `proxmox-infra` via StackReference — do **not** set them here. ## Prerequisites -`proxmox-infra` must be deployed first. On first `pulumi up`, it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or the public key manually added to `~/.ssh/authorized_keys`) before k8s-bootstrap can SSH in. +`proxmox-infra` must be deployed first. On its first `pulumi up` it generates an ed25519 SSH keypair via `@pulumi/tls`, writes the public key into every VM's cloud-init, and exports the private key as a secret stack output. VMs must be re-provisioned (or have the public key manually added to `~/.ssh/authorized_keys`) before this stack can SSH in. -## After first run +## CI/CD -The `kubeconfig` stack output must be propagated to `k8s-infra` and `k8s-apps`: - -```bash -KUBECONFIG=$(pulumi stack output kubeconfig --show-secrets) -cd ../k8s-infra && pulumi config set --secret kubeconfig "$KUBECONFIG" -cd ../k8s-apps && pulumi config set --secret kubeconfig "$KUBECONFIG" -``` - -Then re-encode each `Pulumi.dev.yaml` and update the corresponding Gitea secrets -(`K8S_INFRA_DEV_YAML`, `K8S_APPS_DEV_YAML`): - -```bash -base64 -w 0 k8s-infra/Pulumi.dev.yaml -base64 -w 0 k8s-apps/Pulumi.dev.yaml -``` - -## k8s-infra config changes (democratic-csi → NFS CSI) - -The `k8s-infra` stack no longer needs `truenasApiKey` or `truenasDataset`. -Replace them with: - -```bash -cd ../k8s-infra -pulumi config set truenasHost "192.168.1.x" -pulumi config set truenasNfsPath "/mnt/tank/k8s" -# Remove old keys if present: -pulumi config rm truenasApiKey -pulumi config rm truenasDataset -``` - -## TrueNAS one-time setup (before deploying k8s-infra) - -1. Create a dataset: `tank/k8s` -2. Add an NFS share for that dataset -3. In Network → Allowed Networks, permit `192.168.1.0/24` -4. No API key required — the NFS CSI driver connects directly via NFS protocol +Workflow: `../.gitea/workflows/02-deploy-k8s-bootstrap.yaml`. Secret: `K8S_BOOTSTRAP_PULUMI_DEV_YAML`. Deploy is manual (`workflow_dispatch`) — re-running this stack restarts the bootstrap sequence against live VMs. diff --git a/03-k8s-infra/CLAUDE.md b/03-k8s-infra/CLAUDE.md new file mode 100644 index 0000000..0015da2 --- /dev/null +++ b/03-k8s-infra/CLAUDE.md @@ -0,0 +1,55 @@ +# k8s-infra + +This file provides guidance to Claude Code when working in `03-k8s-infra`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD. + +Cluster-level infrastructure for the k3s homelab cluster, deployed via Helm and `@pulumi/kubernetes`. Reads `kubeconfig` from `k8s-bootstrap` via StackReference. + +## Common Commands + +```bash +npm install +pulumi refresh --yes && pulumi up --yes +``` + +CI installs the `helm` CLI (`azure/setup-helm`) before running Pulumi. + +## Architecture + +| Component | What it does | +| --- | --- | +| NFS CSI Driver | `csi-driver-nfs` Helm chart — enables NFS-backed PVCs | +| `truenas-nfs` StorageClass | Backed by an NFS share on TrueNAS (`Retain` reclaim policy) | +| cert-manager | Certificate lifecycle management | +| `letsencrypt-prod` ClusterIssuer | DNS-01 via Cloudflare, issues Let's Encrypt TLS certs | +| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) — used by app/monitoring PVCs | +| kube-vip | DaemonSet that ARP-announces a virtual IP for LoadBalancer Services | +| kube-vip-cloud-provider | Allocates LoadBalancer IPs from a single-address pool (`kubeVipAddress`) — k3s's built-in ServiceLB is not used | +| `traefik` ServicePatch | Pins Traefik's LoadBalancer IP to `kubeVipAddress` | + +**Export**: `storageClass = "truenas-nfs"`. + +## Required Pulumi config + +```bash +pulumi config set --secret truenasHost +pulumi config set --secret truenasNfsPath /mnt/pool/k8s +pulumi config set --secret cloudflareApiToken +pulumi config set --secret letsencryptEmail +pulumi config set --secret kubeVipAddress +pulumi config set kubeVipInterface +``` + +`kubeconfig` is read automatically via StackReference — do not set it here. + +## TrueNAS one-time setup (before deploying) + +1. Create dataset `pool/k8s` +2. Add an NFS share for that dataset: + - ACL type: POSIX/generic Unix permissions (not SMB/NFSv4 ACLs) on TrueNAS SCALE + - Maproot User / Group: `root` / `root` + - Allowed Hosts/Networks: restrict to the k3s node IPs/CIDR +3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`) + +## CI/CD + +Workflow: `../.gitea/workflows/03-deploy-k8s-infra.yaml`. Secret: `K8S_INFRA_PULUMI_DEV_YAML`. diff --git a/03-k8s-infra/index.ts b/03-k8s-infra/index.ts index aaf3ba6..d848140 100644 --- a/03-k8s-infra/index.ts +++ b/03-k8s-infra/index.ts @@ -3,11 +3,11 @@ import * as k8s from "@pulumi/kubernetes"; const config = new pulumi.Config(); -const infraRef = new pulumi.StackReference( +const bootstrapRef = new pulumi.StackReference( `${pulumi.getOrganization()}/k8s-bootstrap/dev`, ); -const kubeconfig = infraRef.requireOutput("kubeconfig"); +const kubeconfig = bootstrapRef.requireOutput("kubeconfig"); const truenasHost = config.requireSecret("truenasHost"); const truenasNfsPath = config.requireSecret("truenasNfsPath"); const cloudflareToken = config.requireSecret("cloudflareApiToken"); diff --git a/04-k8s-apps/CLAUDE.md b/04-k8s-apps/CLAUDE.md new file mode 100644 index 0000000..f61881a --- /dev/null +++ b/04-k8s-apps/CLAUDE.md @@ -0,0 +1,32 @@ +# k8s-apps + +This file provides guidance to Claude Code when working in `04-k8s-apps`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD. + +Application workloads for the k3s homelab cluster. Reads `kubeconfig` from `k8s-bootstrap` via StackReference. + +## Common Commands + +```bash +npm install +pulumi up --yes +``` + +## Architecture + +- **heimdall** — application dashboard (`lscr.io/linuxserver/heimdall`), namespace `heimdall` + - PVC `heimdall-config` (50Mi, `longhorn` StorageClass) + - Deployment + ClusterIP Service + - Ingress (`traefik` class, `cert-manager.io/cluster-issuer: letsencrypt-prod`) at `heimdall.` + - deployed with one replica because heimdall is not able to work with multiple replicas. Works with node failure detection and automatic rescheduling. + +**Export**: `domain` (secret) — consumed by `monitoring` via StackReference for its own ingress hostnames. + +## Required Pulumi config + +```bash +pulumi config set --secret domain +``` + +## CI/CD + +Workflow: `../.gitea/workflows/04-deploy-k8s-apps.yaml`. Secret: `K8S_APPS_PULUMI_DEV_YAML`. diff --git a/04-k8s-apps/index.ts b/04-k8s-apps/index.ts index 12071d3..c002a63 100644 --- a/04-k8s-apps/index.ts +++ b/04-k8s-apps/index.ts @@ -3,7 +3,6 @@ import * as k8s from "@pulumi/kubernetes"; const config = new pulumi.Config(); -//fetch credentials from k8s-bootstrap const bootstrapRef = new pulumi.StackReference( `${pulumi.getOrganization()}/k8s-bootstrap/dev`, ); diff --git a/05-monitoring/CLAUDE.md b/05-monitoring/CLAUDE.md new file mode 100644 index 0000000..5a12ed6 --- /dev/null +++ b/05-monitoring/CLAUDE.md @@ -0,0 +1,34 @@ +# monitoring + +This file provides guidance to Claude Code when working in `05-monitoring`. See the [repo-root CLAUDE.md](../CLAUDE.md) for shared conventions, deployment order, and CI/CD. + +Monitoring stack (Prometheus + Grafana) via the `kube-prometheus-stack` Helm chart. Reads `kubeconfig` from `k8s-bootstrap` and `domain` from `k8s-apps`, both via StackReference. + +## Common Commands + +```bash +npm install +pulumi up --yes +``` + +## Architecture + +- Namespace `monitoring` +- `kube-prometheus-stack` Helm release (chart v86.2.0) + - Grafana: ingress at `grafana.` (`traefik`, `letsencrypt-prod`), `longhorn` PVC (2Gi) + - Prometheus: `longhorn` PVC (20Gi) via `volumeClaimTemplate` + - Alertmanager disabled + +## Key Notes + +- The `monitoring` namespace and the Helm release both use the Pulumi `import` resource option (`monitoring`, `monitoring/kube-prometheus-stack`) — they were adopted from pre-existing cluster resources rather than created fresh. Be careful with renames or changes that would force a replace. + +## Required Pulumi config + +```bash +pulumi config set --secret grafanaAdminPassword +``` + +## CI/CD + +Workflow: `../.gitea/workflows/05-deploy-monitoring.yaml`. Secret: `MONITORING_PULUMI_DEV_YAML`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..dbc8201 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,64 @@ +# CLAUDE.md + +This file provides guidance to Claude Code when working anywhere in this repository. Each stack directory (`01-proxmox-infra`, `02-k8s-bootstrap`, etc.) has its own `CLAUDE.md` with stack-specific commands and architecture — Claude Code loads both this file and the stack's file automatically when working inside a stack directory. + +## Project overview + +Pulumi (TypeScript) IaC for a Proxmox-based k3s homelab. Independent Pulumi projects ("stacks"), each with its own `Pulumi.yaml`/`package.json`, deployed in order and linked via `pulumi.StackReference`: + +| # | Directory | Pulumi stack | Purpose | +| --- | --- | --- | --- | +| 1 | `01-proxmox-infra` | `proxmox-infra` | Provisions k3s VMs + DHCP on Proxmox/pfSense | +| 2 | `02-k8s-bootstrap` | `k8s-bootstrap` | Installs k3s on the VMs over SSH | +| 3 | `03-k8s-infra` | `k8s-infra` | Cluster infra: storage, cert-manager, kube-vip | +| 4 | `04-k8s-apps` | `k8s-apps` | Application workloads (e.g. heimdall) | +| 5 | `05-monitoring` | `monitoring` | kube-prometheus-stack (Prometheus + Grafana) | + +All stacks use the Pulumi stack name `dev`. + +## Deployment order & StackReferences + +``` +proxmox-infra → k8s-bootstrap → k8s-infra + ↘ k8s-apps → monitoring +``` + +- `k8s-bootstrap` reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference. +- `k8s-infra`, `k8s-apps`, and `monitoring` all read `kubeconfig` from `k8s-bootstrap` via StackReference (`requireOutput("kubeconfig")`) — never set `kubeconfig` as Pulumi config in these stacks. +- `monitoring` additionally reads `domain` from `k8s-apps` via StackReference. +- StackReference format: `${pulumi.getOrganization()}//dev`. + +A stack's outputs must exist (i.e. it must have been deployed) before any downstream stack referencing it will succeed. + +## Shared conventions + +- **TypeScript**: `strict: true`, `target: es2024`, `module`/`moduleResolution: nodenext`, `noImplicitReturns`, `noFallthroughCasesInSwitch` — functions need explicit return types where TS can't infer them. +- **Secrets**: read via `config.requireSecret()`, never via environment variables or hardcoded values. +- **`Pulumi.dev.yaml`**: gitignored (contains the encryption salt) — never delete it or secrets become unrecoverable. Encrypted with `PULUMI_CONFIG_PASSPHRASE`. +- **Backend**: self-hosted Postgres state backend, set via `PULUMI_BACKEND_URL`. +- Don't add a co-author when committing to git. + +## Common commands (run from a stack directory) + +```bash +npm install +pulumi preview +pulumi up --yes +pulumi destroy +pulumi stack output +pulumi config +``` + +Requires `PULUMI_BACKEND_URL` and `PULUMI_CONFIG_PASSPHRASE` in the environment (or `pulumi login`). + +## CI/CD (Gitea Actions) + +One workflow per stack: `.gitea/workflows/0N-deploy-.yaml`, scoped to changes under that stack's directory. + +- **Pull request → `main`**: `pulumi preview` only. +- `01`/`02` (VM provisioning / cluster bootstrap — riskier to automate): deploy only via manual `workflow_dispatch`. +- `03`–`05`: deploy on push to `main` *or* `workflow_dispatch` — `pulumi refresh` then `pulumi up`. +- Each workflow restores `Pulumi.dev.yaml` from a base64-encoded Gitea secret (`_PULUMI_DEV_YAML`). After changing local config, re-encode and update the secret: + ```bash + base64 -w 0 0N-/Pulumi.dev.yaml + ``` diff --git a/README.md b/README.md index cf3ae3f..a916c44 100644 --- a/README.md +++ b/README.md @@ -1,239 +1,154 @@ # Homelab Infrastructure as Code -A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks with version-controlled, reproducible infrastructure — a highly available k3s cluster across multiple Proxmox nodes, bootstrapped and configured end-to-end from code. +A Pulumi-based IaC project for a Proxmox homelab: a highly-available k3s cluster, its cluster-level infrastructure, and the applications running on top of it — all provisioned and configured from code. This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab. ## Why IaC for a homelab? -- **Reproducibility** — rebuild your entire environment from scratch with a single command +- **Reproducibility** — rebuild the entire environment from scratch from code - **Version history** — every change is tracked in git; roll back at any time -- **Auditability** — diff infrastructure changes before applying them (`pulumi preview`) -- **Automation** — CI/CD handles deploys; no manual SSH into nodes -- **Portability** — swap node names, datastores, or credentials without touching logic +- **Auditability** — `pulumi preview` diffs infrastructure changes before they're applied +- **Automation** — CI/CD handles deploys; no manual SSH into nodes for routine changes +- **Portability** — swap node names, datastores, domains, or credentials without touching logic ## Repository layout ``` . -├── proxmox-infra/ # Stack 1 — VMs & DHCP on Proxmox + pfSense -│ ├── index.ts -│ ├── Pulumi.yaml -│ └── sdks/pfsense/ # Locally bundled @pulumi/pfsense SDK -├── k8s-bootstrap/ # Stack 2 — Install k3s on the VMs over SSH -│ ├── index.ts -│ └── Pulumi.yaml -├── k8s-infra/ # Stack 3 — Cluster-level infrastructure via Helm -│ ├── index.ts -│ └── Pulumi.yaml -└── .gitea/workflows/ # Gitea Actions — one workflow per stack +├── 01-proxmox-infra/ # Stack 1 — k3s VMs + DHCP on Proxmox/pfSense +├── 02-k8s-bootstrap/ # Stack 2 — install k3s on the VMs over SSH +├── 03-k8s-infra/ # Stack 3 — storage, cert-manager, kube-vip +├── 04-k8s-apps/ # Stack 4 — application workloads +├── 05-monitoring/ # Stack 5 — kube-prometheus-stack (Prometheus + Grafana) +├── .gitea/workflows/ # CI/CD — one workflow per stack +└── CLAUDE.md # Conventions & architecture notes (root + one per stack) ``` -## Stack overview +Each stack is an independent Pulumi project (own `Pulumi.yaml`/`package.json`), all using the Pulumi stack name `dev`. -The three stacks are deployed in order and linked via Pulumi StackReferences — outputs from one stack flow automatically into the next. +## Stacks + +Stacks are deployed in order and linked via `pulumi.StackReference` — outputs from one stack flow automatically into the next, so a stack must be deployed before anything depending on it will work. ``` -proxmox-infra → k8s-bootstrap → k8s-infra +proxmox-infra → k8s-bootstrap → k8s-infra + ↘ k8s-apps → monitoring ``` -### Stack 1: `proxmox-infra` +### 1. proxmox-infra -Provisions a 5-node k3s cluster spread across two Proxmox hosts: +Provisions a 5-node k3s cluster across two Proxmox hosts, each VM a clone of an Ubuntu Noble (24.04) cloud-image template: -| VM name | Role | Proxmox node | -| ------------ | ------ | ------------ | -| k3s-master-1 | master | pve | -| k3s-master-2 | master | pve | -| k3s-worker-1 | worker | pve | -| k3s-master-3 | master | pve-bckp | -| k3s-worker-2 | worker | pve-bckp | +| VM | Role | +| --- | --- | +| k3s-master-1 | master | +| k3s-master-2 | master | +| k3s-worker-1 | worker | +| k3s-master-3 | master | +| k3s-worker-2 | worker | -Each VM is a full clone of an Ubuntu Noble (24.04) cloud-image template. Cloud-init injects the hostname, user credentials, and SSH keys at boot. An ED25519 SSH keypair is generated once and stored in Pulumi state — the public key goes into every VM, the private key is exported as a stack output for `k8s-bootstrap` to consume via StackReference. +It also generates an ED25519 SSH keypair (stored in Pulumi state, public key injected via cloud-init) and registers each VM's MAC address as a DHCPv4 static mapping in pfSense. -Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so nodes always receive their designated IPs. +**Providers:** `@muhlba91/pulumi-proxmoxve`, `@pulumi/pfsense` (locally bundled), `@pulumi/tls` -**Providers:** `@muhlba91/pulumi-proxmoxve` v8.x, `@pulumi/pfsense` (locally bundled), `@pulumi/tls` +### 2. k8s-bootstrap -**Exports:** `vmIds`, `ciRunnerPrivateKey`, `pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `master1Ip` … `worker2Ip` +Starts the VMs via the Proxmox API and installs k3s over SSH: `k3s-master-1` initializes the cluster (`--cluster-init`, embedded etcd), `k3s-master-2`/`k3s-master-3` join as additional etcd nodes, and the two workers join as agents. Exports the cluster's `kubeconfig` as a secret for downstream stacks. -### Stack 2: `k8s-bootstrap` +### 3. k8s-infra -Installs k3s on the Proxmox VMs over SSH using `@pulumi/command`. Reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference — no manual key distribution needed. +Cluster-level infrastructure via Helm and `@pulumi/kubernetes`: -Deployment sequence: +| Component | What it does | +| --- | --- | +| NFS CSI Driver + `truenas-nfs` StorageClass | Dynamic NFS-backed PVCs from a TrueNAS share | +| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) | +| cert-manager + `letsencrypt-prod` ClusterIssuer | TLS certs via Let's Encrypt, DNS-01 through Cloudflare | +| kube-vip + kube-vip-cloud-provider | Announces a virtual IP and allocates it to LoadBalancer Services (e.g. Traefik) | -1. Start all 5 VMs via the Proxmox REST API -2. Wait for port 22 to open on each node -3. Install k3s on `k3s-master-1` with `--cluster-init` (embedded etcd) -4. Join `k3s-master-2` and `k3s-master-3` as additional etcd nodes -5. Join `k3s-worker-1` and `k3s-worker-2` as agent nodes -6. Read `/etc/rancher/k3s/k3s.yaml` from master-1, patch the server URL, export as a secret stack output +### 4. k8s-apps -**Exports:** `kubeconfig` (secret) +Application workloads exposed at `app.` via Traefik with a Let's Encrypt cert. -### Stack 3: `k8s-infra` +### 5. monitoring -Deploys cluster-level infrastructure via Helm and the `@pulumi/kubernetes` provider. Reads `kubeconfig` from `k8s-bootstrap` via StackReference. - -| Component | What it does | -| -------------------- | -------------------------------------------------------------------------- | -| NFS CSI Driver | `csi-driver-nfs` Helm chart (v4.12.0) — enables dynamic NFS-backed PVCs | -| TrueNAS StorageClass | `truenas-nfs` StorageClass backed by an NFS share on TrueNAS | -| cert-manager | `cert-manager` Helm chart — certificate lifecycle management | -| ClusterIssuer | `letsencrypt-prod` — DNS-01 via Cloudflare, issues Let's Encrypt TLS certs | +`kube-prometheus-stack` Helm release — Prometheus (20Gi Longhorn PVC) and Grafana (2Gi PVC, ingress at `grafana.`). ## Prerequisites -- [Pulumi CLI](https://www.pulumi.com/docs/install/) installed -- Node.js 18+ and npm -- Two Proxmox nodes with API tokens -- pfSense with REST API credentials (used for DHCPv4 static mapping) -- TrueNAS with an NFS share (no API key needed — CSI driver connects via NFS protocol directly) -- A Cloudflare account with an API token scoped to DNS edit on your zone +- [Pulumi CLI](https://www.pulumi.com/docs/install/), Node.js 24+ and npm +- One or more Proxmox nodes with API tokens +- pfSense (DHCPv4 static mappings) +- TrueNAS with an NFS share (connected directly — no API key needed) +- A domain with DNS managed by Cloudflare (cert-manager DNS-01 + ingress hostnames) - A self-hosted Pulumi state backend (PostgreSQL connection string) -- Gitea instance for CI/CD (optional for local use) +- Gitea instance for CI/CD (optional for local-only use) ## Getting started -Deploy the stacks **in order**. Each stack must be fully deployed before the next one runs. - -### Stack 1 — `proxmox-infra` - -```bash -cd proxmox-infra -npm install # also compiles the bundled pfSense SDK -pulumi stack init dev -``` - -Set secrets: - -```bash -# Proxmox API credentials -pulumi config set --secret pve1Endpoint https://:8006 -pulumi config set --secret pve1ApiToken @pam!= -pulumi config set --secret pve2Endpoint https://:8006 -pulumi config set --secret pve2ApiToken @pam!= - -# VM credentials -pulumi config set --secret k3sVmPassword -pulumi config set --secret sshPvePublicKey "ssh-ed25519 AAAA..." - -# pfSense credentials (used for DHCPv4 static mappings) -pulumi config set --secret pfSenseUrl https:// -pulumi config set --secret pfSenseUser -pulumi config set --secret pfSensePassword - -# Static IP addresses assigned to each k3s node -pulumi config set --secret master1Ip -pulumi config set --secret master2Ip -pulumi config set --secret worker1Ip -pulumi config set --secret master3Ip -pulumi config set --secret worker2Ip -``` - -Deploy: +Deploy stacks **in order** — each must be fully `pulumi up`'d before the next stack's `StackReference` resolves. Set the shared backend env vars once per shell: ```bash export PULUMI_BACKEND_URL=postgresql://:@/ export PULUMI_CONFIG_PASSPHRASE= - -pulumi preview # inspect before touching anything -pulumi up --yes ``` -### Stack 2 — `k8s-bootstrap` +Then for each stack, in order: ```bash -cd ../k8s-bootstrap +cd 0N- npm install pulumi stack init dev - -pulumi config set --secret k3sToken "$(openssl rand -hex 32)" -``` - -Node IPs and Proxmox credentials are read automatically from `proxmox-infra` outputs via StackReference — do not set them here. - -```bash +pulumi config set ... # required keys and secrets pulumi up --yes ``` -### Stack 3 — `k8s-infra` +Stack-specific notes: -**TrueNAS one-time setup** (before deploying this stack): - -1. Create dataset `tank/k8s` -2. Add an NFS share for that dataset - - - Acl Type: Set this to POSIX (or generic Unix permissions) rather than SMB/NFSv4 ACLs if you are on TrueNAS SCALE, as Kubernetes handles basic Unix permissions natively. - - Maproot User / Maproot Group: Set this to root and root - - Allowed Hosts/Networks: Restrict this share specifically to the IP addresses or the CIDR block of your Proxmox K3s nodes (e.g., 192.168.1.50) - -3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`) - -```bash -cd ../k8s-infra -npm install -pulumi stack init dev - -# kubeconfig from k8s-bootstrap -KUBECONFIG=$(cd ../k8s-bootstrap && pulumi stack output kubeconfig --show-secrets) -pulumi config set --secret kubeconfig "$KUBECONFIG" - -# TrueNAS NFS -pulumi config set --secret truenasHost -pulumi config set --secret truenasNfsPath /mnt/tank/k8s - -# cert-manager + Cloudflare DNS-01 -pulumi config set --secret cloudflareApiToken -pulumi config set --secret letsencryptEmail - -pulumi up --yes -``` +1. **proxmox-infra** — set Proxmox API tokens, pfSense credentials, VM credentials/SSH public key, and a static IP per node. +2. **k8s-bootstrap** — set `k3sToken` (e.g. `openssl rand -hex 32`) and a static IP per node. Proxmox credentials and the SSH key come automatically from `proxmox-infra`. +3. **k8s-infra** — before deploying, create an NFS dataset/share on TrueNAS (POSIX ACLs, `root`/`root` maproot, restricted to your k3s nodes). Set the TrueNAS, Cloudflare, and `kubeVipAddress`/`kubeVipInterface` config. +4. **k8s-apps** — set `domain`, the base domain used for ingress hostnames. +5. **monitoring** — set `grafanaAdminPassword`. ## CI/CD (Gitea Actions) -One workflow per stack under `.gitea/workflows/`. Each triggers on changes to its own stack directory. +One workflow per stack, `.gitea/workflows/0N-deploy-.yaml`, scoped to changes under that stack's directory. -| Event | Action | -| --------------------- | ------------------------------ | -| Pull request → `main` | `pulumi preview` (read-only) | -| Push to `main` | `pulumi refresh` + `pulumi up` | -| Manual trigger | `pulumi refresh` + `pulumi up` | +| Trigger | Stacks | Action | +| --- | --- | --- | +| PR → `main` | all | `pulumi preview` (read-only) | +| Push to `main` | 03–05 | `pulumi refresh` + `pulumi up` | +| Manual dispatch | all | `pulumi refresh` + `pulumi up` | -### Required Gitea secrets +Stacks 01/02 (VM provisioning, cluster bootstrap) deploy **only** via manual dispatch — they're riskier to automate. -Configure under **Settings → Actions → Secrets**: +### Required secrets -| Secret | Used by | Description | -| ------------------------------- | --------------------------- | ---------------------------------------------- | -| `PULUMI_BACKEND_URL` | all workflows | PostgreSQL connection string for state backend | -| `PULUMI_CONFIG_PASSPHRASE` | all workflows | Passphrase to decrypt secrets | -| `PROXMOX_INFRA_PULUMI_DEV_YAML` | `deploy-proxmox-infra.yaml` | Base64-encoded `proxmox-infra/Pulumi.dev.yaml` | -| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | `deploy-k8s-bootstrap.yaml` | Base64-encoded `k8s-bootstrap/Pulumi.dev.yaml` | -| `K8S_INFRA_PULUMI_DEV_YAML` | `deploy-k8s-infra.yaml` | Base64-encoded `k8s-infra/Pulumi.dev.yaml` | +| Secret | Used by | +| --- | --- | +| `PULUMI_BACKEND_URL`, `PULUMI_CONFIG_PASSPHRASE` | all workflows | +| `PROXMOX_INFRA_PULUMI_DEV_YAML` | 01-deploy-proxmox-infra | +| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | 02-deploy-k8s-bootstrap | +| `K8S_INFRA_PULUMI_DEV_YAML` | 03-deploy-k8s-infra | +| `K8S_APPS_PULUMI_DEV_YAML` | 04-deploy-k8s-apps | +| `MONITORING_PULUMI_DEV_YAML` | 05-deploy-monitoring | -`Pulumi.dev.yaml` files are gitignored (they contain your encryption salt). Whenever one changes, re-encode and paste into the corresponding Gitea secret: +Each `_PULUMI_DEV_YAML` secret is a base64-encoded copy of that stack's gitignored `Pulumi.dev.yaml`. After changing local config, re-encode and update the secret: ```bash -base64 -w 0 proxmox-infra/Pulumi.dev.yaml -base64 -w 0 k8s-bootstrap/Pulumi.dev.yaml -base64 -w 0 k8s-infra/Pulumi.dev.yaml +base64 -w 0 0N-/Pulumi.dev.yaml ``` -## Adapting this as a template +## Using this as a template 1. Fork or copy the repo -2. Update Proxmox node names (`pve`, `pve-bckp`) and datastore IDs in `proxmox-infra/index.ts` -3. Add or remove VMs from the `nodeConfigs` array -4. Set your own secrets with `pulumi config set --secret` -5. Point the CI/CD workflows at your own Gitea instance +2. Update Proxmox node names/datastores and the `nodeConfigs` array in `01-proxmox-infra/index.ts` for your VM topology +3. Set your own config/secrets per stack with `pulumi config set [--secret]` — see each stack's `CLAUDE.md` for the full list of required keys +4. Point the CI/CD workflows at your own Gitea/GitOps instance and update the secrets above ## Roadmap -- LXC container management -- `k8s-apps` stack — application deployments on the cluster -- Docker / Compose stack provisioning - - Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault -- Add a third bare metal Proxmox node for true 3-node HA parity +- Add a third bare-metal Proxmox node for true 3-node HA parity