15 Commits

Author SHA1 Message Date
kasun 94be23def7 fix: added shutdown logic to deployment yaml to speed up deployment runs
Deploy Proxmox Infra / Pulumi Preview (pull_request) Successful in 1m8s
Deploy Proxmox Infra / Pulumi Deploy (pull_request) Has been skipped
Deploy k8s Infra / Pulumi Preview (pull_request) Successful in 47s
Deploy k8s Infra / Pulumi Deploy (pull_request) Has been skipped
2026-06-01 22:33:43 +02:00
kasun 19af76fd2b Revert "fix: readded pfsense to refresh call since it was not the culprit for the deployment delays"
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 52s
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Successful in 1m4s
This reverts commit 354cf65657.
2026-06-01 21:37:56 +02:00
kasun 354cf65657 fix: readded pfsense to refresh call since it was not the culprit for the deployment delays
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Preview (push) Has been cancelled
Deploy k8s Infra / Pulumi Deploy (push) Has been cancelled
Deploy Proxmox Infra / Pulumi Deploy (push) Has been cancelled
2026-06-01 21:32:26 +02:00
kasun 61758cbb67 fix: further optimizations to increase deployment speed
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Successful in 1m1s
2026-06-01 21:25:50 +02:00
kasun c97bebd327 fix: excluding pfsene from the refresh call because of unknown delay
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 51s
Deploy Proxmox Infra / Pulumi Deploy (push) Has been cancelled
2026-06-01 21:15:30 +02:00
kasun 9ffc1bc94e fix: optimize pfsense refresh
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Has been cancelled
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Has been cancelled
2026-06-01 21:03:02 +02:00
kasun cc40873bb4 fix: optimiized long pulumi refresh calls.
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Has been cancelled
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Has been cancelled
2026-06-01 20:35:47 +02:00
kasun c4a7ea185a fixed wrong syntax
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Successful in 32m32s
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 49s
2026-06-01 20:13:48 +02:00
kasun 3f874e6f97 fixed naming issues
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Failing after 1m30s
Deploy k8s Bootstrap / Pulumi Preview (push) Has been skipped
Deploy k8s Bootstrap / Bootstrap k3s Cluster (push) Successful in 40s
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 52s
2026-06-01 20:08:39 +02:00
kasun 5de2a16b9b changed naming and order of folders and files to represent deployment order
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Failing after 50s
Deploy k8s Bootstrap / Pulumi Preview (push) Has been skipped
Deploy k8s Bootstrap / Bootstrap k3s Cluster (push) Failing after 7s
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Failing after 7s
2026-06-01 20:03:51 +02:00
kasun 54b97fadeb fix: made changes to cache dependencies and lower download processes to make deployments faster
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 51s
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Failing after 2m3s
2026-06-01 19:29:23 +02:00
kasun f22cad1a37 revised README.MD with k8s-infra deployment 2026-06-01 19:17:26 +02:00
kasun 113de6fb46 fix: removed refresh behaviour shutting down running vm
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Successful in 31m24s
2026-06-01 16:55:36 +02:00
kasun 8420c33f69 fix: changed cert-manager params
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 54s
2026-06-01 16:39:25 +02:00
kasun 786974fca3 Merge pull request 'added nfs and cert management' (#5) from feature/add-k3s-infrastructure-for-certs-and-nfs-storage into main
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Failing after 6m11s
Deploy Proxmox Infra / Pulumi Preview (push) Has been skipped
Deploy Proxmox Infra / Pulumi Deploy (push) Successful in 16m15s
Reviewed-on: #5
2026-06-01 16:25:59 +02:00
21 changed files with 209 additions and 94 deletions
@@ -6,13 +6,13 @@ on:
branches:
- main
paths:
- 'proxmox-infra/**'
- '01-proxmox-infra/**'
- '.gitea/workflows/**'
pull_request:
branches:
- main
paths:
- 'proxmox-infra/**'
- '01-proxmox-infra/**'
- '.gitea/workflows/**'
jobs:
@@ -30,25 +30,25 @@ jobs:
node-version: '24'
- name: Restore Stack Config
run: echo "${{ secrets.PROXMOX_INFRA_PULUMI_DEV_YAML }}" | base64 -d > proxmox-infra/Pulumi.dev.yaml
run: echo "${{ secrets.PROXMOX_INFRA_PULUMI_DEV_YAML }}" | base64 -d > 01-proxmox-infra/Pulumi.dev.yaml
- name: Install Dependencies
run: npm ci
working-directory: proxmox-infra
working-directory: 01-proxmox-infra
- name: Install Pulumi CLI
run: curl -fsSL https://get.pulumi.com | sh && echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Generate Local pfSense SDK
run: pulumi package add terraform-provider marshallford/pfsense
working-directory: proxmox-infra
run: pulumi package add terraform-provider marshallford/pfsense 0.22.0
working-directory: 01-proxmox-infra
- name: Preview
uses: pulumi/actions@v5
with:
command: preview
stack-name: dev
work-dir: proxmox-infra
work-dir: 01-proxmox-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -67,25 +67,66 @@ jobs:
node-version: '24'
- name: Restore Stack Config
run: echo "${{ secrets.PROXMOX_INFRA_PULUMI_DEV_YAML }}" | base64 -d > proxmox-infra/Pulumi.dev.yaml
run: echo "${{ secrets.PROXMOX_INFRA_PULUMI_DEV_YAML }}" | base64 -d > 01-proxmox-infra/Pulumi.dev.yaml
- name: Install Dependencies
run: npm ci
working-directory: proxmox-infra
working-directory: 01-proxmox-infra
- name: Install Pulumi CLI
run: curl -fsSL https://get.pulumi.com | sh && echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Generate Local pfSense SDK
run: pulumi package add terraform-provider marshallford/pfsense
working-directory: proxmox-infra
run: pulumi package add terraform-provider marshallford/pfsense 0.22.0
working-directory: 01-proxmox-infra
- name: Shutdown VMs
run: |
pulumi login "$PULUMI_BACKEND_URL"
PVE1=$(pulumi stack output --stack dev --show-secrets pve1Endpoint)
TOKEN1=$(pulumi stack output --stack dev --show-secrets pve1ApiToken)
PVE2=$(pulumi stack output --stack dev --show-secrets pve2Endpoint)
TOKEN2=$(pulumi stack output --stack dev --show-secrets pve2ApiToken)
IDS=$(pulumi stack output --stack dev --json vmIds)
M1=$(echo "$IDS" | jq -r .master1)
M2=$(echo "$IDS" | jq -r .master2)
W1=$(echo "$IDS" | jq -r .worker1)
M3=$(echo "$IDS" | jq -r .master3)
W2=$(echo "$IDS" | jq -r .worker2)
for id in $M1 $M2 $W1; do
curl -sf -k -X POST "$PVE1/api2/json/nodes/pve/qemu/$id/status/shutdown" \
-H "Authorization: PVEAPIToken=$TOKEN1" || true
done
for id in $M3 $W2; do
curl -sf -k -X POST "$PVE2/api2/json/nodes/pve-bckp/qemu/$id/status/shutdown" \
-H "Authorization: PVEAPIToken=$TOKEN2" || true
done
wait_stopped() {
local ep=$1 tok=$2 node=$3 id=$4
for i in $(seq 1 36); do
status=$(curl -sf -k "$ep/api2/json/nodes/$node/qemu/$id/status/current" \
-H "Authorization: PVEAPIToken=$tok" | jq -r .data.status)
[ "$status" = "stopped" ] && return 0
sleep 5
done
echo "Timeout: VM $id did not stop within 3 minutes" && exit 1
}
for id in $M1 $M2 $W1; do wait_stopped "$PVE1" "$TOKEN1" pve "$id"; done
for id in $M3 $W2; do wait_stopped "$PVE2" "$TOKEN2" pve-bckp "$id"; done
working-directory: 01-proxmox-infra
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
PULUMI_BACKEND_URL: ${{ secrets.PULUMI_BACKEND_URL }}
- name: Refresh State
uses: pulumi/actions@v5
with:
command: refresh
stack-name: dev
work-dir: proxmox-infra
work-dir: 01-proxmox-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -95,7 +136,7 @@ jobs:
with:
command: up
stack-name: dev
work-dir: proxmox-infra
work-dir: 01-proxmox-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -6,14 +6,14 @@ on:
branches:
- main
paths:
- "k8s-bootstrap/**"
- ".gitea/workflows/deploy-k8s-bootstrap.yaml"
- "02-k8s-bootstrap/**"
- ".gitea/workflows/02-deploy-k8s-bootstrap.yaml"
pull_request:
branches:
- main
paths:
- "k8s-bootstrap/**"
- ".gitea/workflows/deploy-k8s-bootstrap.yaml"
- "02-k8s-bootstrap/**"
- ".gitea/workflows/02-deploy-k8s-bootstrap.yaml"
jobs:
preview:
@@ -30,18 +30,18 @@ jobs:
node-version: "24"
- name: Restore Stack Config
run: echo "${{ secrets.K8S_BOOTSTRAP_PULUMI_DEV_YAML }}" | base64 -d > k8s-bootstrap/Pulumi.dev.yaml
run: echo "${{ secrets.K8S_BOOTSTRAP_PULUMI_DEV_YAML }}" | base64 -d > 02-k8s-bootstrap/Pulumi.dev.yaml
- name: Install Dependencies
run: npm install
working-directory: k8s-bootstrap
working-directory: 02-k8s-bootstrap
- name: Preview
uses: pulumi/actions@v5
with:
command: preview
stack-name: dev
work-dir: k8s-bootstrap
work-dir: 02-k8s-bootstrap
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -60,18 +60,18 @@ jobs:
node-version: "24"
- name: Restore Stack Config
run: echo "${{ secrets.K8S_BOOTSTRAP_PULUMI_DEV_YAML }}" | base64 -d > k8s-bootstrap/Pulumi.dev.yaml
run: echo "${{ secrets.K8S_BOOTSTRAP_PULUMI_DEV_YAML }}" | base64 -d > 02-k8s-bootstrap/Pulumi.dev.yaml
- name: Install Dependencies
run: npm install
working-directory: k8s-bootstrap
working-directory: 02-k8s-bootstrap
- name: Refresh State
uses: pulumi/actions@v5
with:
command: refresh
stack-name: dev
work-dir: k8s-bootstrap
work-dir: 02-k8s-bootstrap
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -81,7 +81,7 @@ jobs:
with:
command: up
stack-name: dev
work-dir: k8s-bootstrap
work-dir: 02-k8s-bootstrap
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -6,13 +6,13 @@ on:
branches:
- main
paths:
- 'k8s-infra/**'
- '03-k8s-infra/**'
- '.gitea/workflows/**'
pull_request:
branches:
- main
paths:
- 'k8s-infra/**'
- '03-k8s-infra/**'
- '.gitea/workflows/**'
jobs:
@@ -30,21 +30,21 @@ jobs:
node-version: '24'
- name: Restore Stack Config
run: echo "${{ secrets.K8S_INFRA_PULUMI_DEV_YAML }}" | base64 -d > k8s-infra/Pulumi.dev.yaml
run: echo "${{ secrets.K8S_INFRA_PULUMI_DEV_YAML }}" | base64 -d > 03-k8s-infra/Pulumi.dev.yaml
- name: Install Helm
uses: azure/setup-helm@v4
- name: Install Dependencies
run: npm ci
working-directory: k8s-infra
working-directory: 03-k8s-infra
- name: Preview
uses: pulumi/actions@v5
with:
command: preview
stack-name: dev
work-dir: k8s-infra
work-dir: 03-k8s-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -63,21 +63,21 @@ jobs:
node-version: '24'
- name: Restore Stack Config
run: echo "${{ secrets.K8S_INFRA_PULUMI_DEV_YAML }}" | base64 -d > k8s-infra/Pulumi.dev.yaml
run: echo "${{ secrets.K8S_INFRA_PULUMI_DEV_YAML }}" | base64 -d > 03-k8s-infra/Pulumi.dev.yaml
- name: Install Helm
uses: azure/setup-helm@v4
- name: Install Dependencies
run: npm ci
working-directory: k8s-infra
working-directory: 03-k8s-infra
- name: Refresh State
uses: pulumi/actions@v5
with:
command: refresh
stack-name: dev
work-dir: k8s-infra
work-dir: 03-k8s-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -87,7 +87,7 @@ jobs:
with:
command: up
stack-name: dev
work-dir: k8s-infra
work-dir: 03-k8s-infra
cloud-url: ${{ secrets.PULUMI_BACKEND_URL }}
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
@@ -16,9 +16,11 @@ npm install
pulumi preview
# Sync Pulumi state with actual Proxmox state (run before up if resources were changed manually)
# Note: shut down all k3s VMs in Proxmox first — refresh is slow against running VMs
pulumi refresh --yes
# Deploy infrastructure
# Note: shut down all k3s VMs in Proxmox first, then run:
pulumi refresh --yes && pulumi up --yes
# Destroy infrastructure
@@ -71,7 +71,7 @@ const ubuntuImagePve = new proxmox.download.File(
contentType: "import",
fileName: "noble-server-cloudimg-amd64.qcow2",
url: ubuntuNobleUrl,
overwrite: true,
overwrite: false,
overwriteUnmanaged: true,
},
{ provider: pveProvider },
@@ -85,7 +85,7 @@ const ubuntuImagePveBckp = new proxmox.download.File(
contentType: "import",
fileName: "noble-server-cloudimg-amd64.qcow2",
url: ubuntuNobleUrl,
overwrite: true,
overwrite: false,
overwriteUnmanaged: true,
},
{ provider: pveBckpProvider },
@@ -135,7 +135,7 @@ const pveTemplate = new proxmox.VmLegacy(
},
],
},
{ provider: pveProvider },
{ provider: pveProvider, ignoreChanges: ["disks"] },
);
const pveBckpTemplate = new proxmox.VmLegacy(
@@ -155,7 +155,7 @@ const pveBckpTemplate = new proxmox.VmLegacy(
},
],
},
{ provider: pveBckpProvider },
{ provider: pveBckpProvider, ignoreChanges: ["disks"] },
);
// ---------------------------------------------------------------------------
@@ -279,7 +279,7 @@ const k3sVms = nodeConfigs.map(
{
provider: node.provider,
retainOnDelete: true,
ignoreChanges: ["clone"],
ignoreChanges: ["clone", "started", "disks"],
},
),
);
+2 -1
View File
@@ -67,8 +67,9 @@ const certManager = new k8s.helm.v3.Release(
repositoryOpts: { repo: "https://charts.jetstack.io" },
namespace: "cert-manager",
createNamespace: true,
timeout: 600,
values: {
installCRDs: true,
crds: { enabled: true },
},
},
opts(),
+126 -55
View File
@@ -1,6 +1,6 @@
# Homelab Infrastructure as Code
A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks (LXC, VM, Docker, etc.) with version-controlled, reproducible infrastructure — starting with a highly available k3s cluster across multiple Proxmox nodes.
A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks with version-controlled, reproducible infrastructure — a highly available k3s cluster across multiple Proxmox nodes, bootstrapped and configured end-to-end from code.
This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab.
@@ -16,20 +16,30 @@ This repo is intentionally abstract: credentials are never hardcoded, making it
```
.
├── proxmox-infra/ # Pulumi TypeScript stack — VMs & LXC on Proxmox
│ ├── index.ts # All Pulumi resources
│ ├── Pulumi.yaml # Stack project definition
── Pulumi.dev.yaml # Encrypted stack config (gitignored)
│ └── sdks/
└── pfsense/ # Locally bundled @pulumi/pfsense SDK
├── .gitea/
│ └── workflows/
└── deploy-proxmox-infra.yaml # Gitea Actions CI/CD pipeline
├── proxmox-infra/ # Stack 1 — VMs & DHCP on Proxmox + pfSense
│ ├── index.ts
│ ├── Pulumi.yaml
── sdks/pfsense/ # Locally bundled @pulumi/pfsense SDK
├── k8s-bootstrap/ # Stack 2 — Install k3s on the VMs over SSH
├── index.ts
│ └── Pulumi.yaml
├── k8s-infra/ # Stack 3 — Cluster-level infrastructure via Helm
├── index.ts
│ └── Pulumi.yaml
└── .gitea/workflows/ # Gitea Actions — one workflow per stack
```
## Current stack: `proxmox-infra`
## Stack overview
Provisions a 5-node k3s cluster spread across two Proxmox hosts (`pve` and `pve-bckp`, third bare metal host to be added later for actual parity):
The three stacks are deployed in order and linked via Pulumi StackReferences — outputs from one stack flow automatically into the next.
```
proxmox-infra → k8s-bootstrap → k8s-infra
```
### Stack 1: `proxmox-infra`
Provisions a 5-node k3s cluster spread across two Proxmox hosts:
| VM name | Role | Proxmox node |
| ------------ | ------ | ------------ |
@@ -39,43 +49,64 @@ Provisions a 5-node k3s cluster spread across two Proxmox hosts (`pve` and `pve-
| k3s-master-3 | master | pve-bckp |
| k3s-worker-2 | worker | pve-bckp |
Each node is a full clone of an Ubuntu Noble (24.04) cloud-image template, with cloud-init injecting hostname, user credentials, and SSH keys at boot. Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so that nodes always receive their designated IPs.
Each VM is a full clone of an Ubuntu Noble (24.04) cloud-image template. Cloud-init injects the hostname, user credentials, and SSH keys at boot. An ED25519 SSH keypair is generated once and stored in Pulumi state — the public key goes into every VM, the private key is exported as a stack output for `k8s-bootstrap` to consume via StackReference.
An ED25519 SSH key pair is generated once and stored in Pulumi state. The public key is injected into every VM at boot; the private key is exported as a stack output so `k8s-bootstrap` can consume it via StackReference without any manual key distribution.
Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so nodes always receive their designated IPs.
**Tech stack:**
**Providers:** `@muhlba91/pulumi-proxmoxve` v8.x, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
- [Pulumi](https://www.pulumi.com/) with TypeScript
- [`@muhlba91/pulumi-proxmoxve`](https://github.com/muhlba91/pulumi-provider-proxmoxve) v8.x community provider
- [`@pulumi/pfsense`](https://github.com/marshallford/terraform-provider-pfsense) — locally bundled SDK bridged from the Terraform pfSense provider; installed automatically via `npm install`
- [`@pulumi/tls`](https://www.pulumi.com/registry/packages/tls/) — SSH key pair generation
- Self-hosted Pulumi state backend (PostgreSQL)
- Gitea Actions for CI/CD
**Exports:** `vmIds`, `ciRunnerPrivateKey`, `pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `master1Ip``worker2Ip`
### Stack 2: `k8s-bootstrap`
Installs k3s on the Proxmox VMs over SSH using `@pulumi/command`. Reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference — no manual key distribution needed.
Deployment sequence:
1. Start all 5 VMs via the Proxmox REST API
2. Wait for port 22 to open on each node
3. Install k3s on `k3s-master-1` with `--cluster-init` (embedded etcd)
4. Join `k3s-master-2` and `k3s-master-3` as additional etcd nodes
5. Join `k3s-worker-1` and `k3s-worker-2` as agent nodes
6. Read `/etc/rancher/k3s/k3s.yaml` from master-1, patch the server URL, export as a secret stack output
**Exports:** `kubeconfig` (secret)
### Stack 3: `k8s-infra`
Deploys cluster-level infrastructure via Helm and the `@pulumi/kubernetes` provider. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
| Component | What it does |
| -------------------- | -------------------------------------------------------------------------- |
| NFS CSI Driver | `csi-driver-nfs` Helm chart (v4.12.0) — enables dynamic NFS-backed PVCs |
| TrueNAS StorageClass | `truenas-nfs` StorageClass backed by an NFS share on TrueNAS |
| cert-manager | `cert-manager` Helm chart — certificate lifecycle management |
| ClusterIssuer | `letsencrypt-prod` — DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
## Prerequisites
- [Pulumi CLI](https://www.pulumi.com/docs/install/) installed
- Node.js 18+ and npm
- Access to a Proxmox node with an API token
- pfSense instance with API credentials (used for DHCPv4 static mappings)
- Two Proxmox nodes with API tokens
- pfSense with REST API credentials (used for DHCPv4 static mapping)
- TrueNAS with an NFS share (no API key needed — CSI driver connects via NFS protocol directly)
- A Cloudflare account with an API token scoped to DNS edit on your zone
- A self-hosted Pulumi state backend (PostgreSQL connection string)
- Gitea instance for CI/CD (optional for local use)
## Getting started
### 1. Clone and install
Deploy the stacks **in order**. Each stack must be fully deployed before the next one runs.
### Stack 1 — `proxmox-infra`
```bash
git clone <your-repo-url>
cd proxmox-infra
npm install
npm install # also compiles the bundled pfSense SDK
pulumi stack init dev
```
> **pfSense SDK** — The `@pulumi/pfsense` SDK is bundled locally under `sdks/pfsense/` and referenced as a `file:` dependency in `package.json`. Running `npm install` compiles it automatically via its postinstall hook. No separate installation or build step is required.
### 2. Configure credentials
All secrets are stored as encrypted Pulumi config values — never in plain environment variables or committed files.
Set secrets:
```bash
# Proxmox API credentials
@@ -101,31 +132,68 @@ pulumi config set --secret master3Ip <ip-for-k3s-master-3>
pulumi config set --secret worker2Ip <ip-for-k3s-worker-2>
```
Pulumi encrypts these values into `Pulumi.dev.yaml` using your `PULUMI_CONFIG_PASSPHRASE`.
### 3. Set the state backend
Deploy:
```bash
export PULUMI_BACKEND_URL=postgresql://<user>:<pass>@<host>/<db>
export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
pulumi preview # inspect before touching anything
pulumi up --yes
```
### 4. Preview and deploy
### Stack 2 — `k8s-bootstrap`
```bash
# See what will change before touching anything
pulumi preview
cd ../k8s-bootstrap
npm install
pulumi stack init dev
# Sync Pulumi state with actual Proxmox state (run after any manual GUI changes)
pulumi refresh --yes
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
```
# Deploy
pulumi refresh --yes && pulumi up --yes
Node IPs and Proxmox credentials are read automatically from `proxmox-infra` outputs via StackReference — do not set them here.
```bash
pulumi up --yes
```
### Stack 3 — `k8s-infra`
**TrueNAS one-time setup** (before deploying this stack):
1. Create dataset `tank/k8s`
2. Add an NFS share for that dataset
- Acl Type: Set this to POSIX (or generic Unix permissions) rather than SMB/NFSv4 ACLs if you are on TrueNAS SCALE, as Kubernetes handles basic Unix permissions natively.
- Maproot User / Maproot Group: Set this to root and root
- Allowed Hosts/Networks: Restrict this share specifically to the IP addresses or the CIDR block of your Proxmox K3s nodes (e.g., 192.168.1.50)
3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
```bash
cd ../k8s-infra
npm install
pulumi stack init dev
# kubeconfig from k8s-bootstrap
KUBECONFIG=$(cd ../k8s-bootstrap && pulumi stack output kubeconfig --show-secrets)
pulumi config set --secret kubeconfig "$KUBECONFIG"
# TrueNAS NFS
pulumi config set --secret truenasHost <truenas-ip>
pulumi config set --secret truenasNfsPath /mnt/tank/k8s
# cert-manager + Cloudflare DNS-01
pulumi config set --secret cloudflareApiToken <cf-token>
pulumi config set --secret letsencryptEmail <your-email>
pulumi up --yes
```
## CI/CD (Gitea Actions)
The workflow at `.gitea/workflows/deploy-proxmox-infra.yaml` runs automatically:
One workflow per stack under `.gitea/workflows/`. Each triggers on changes to its own stack directory.
| Event | Action |
| --------------------- | ------------------------------ |
@@ -135,34 +203,37 @@ The workflow at `.gitea/workflows/deploy-proxmox-infra.yaml` runs automatically:
### Required Gitea secrets
Configure these under **Settings → Actions → Secrets** in your Gitea repo:
Configure under **Settings → Actions → Secrets**:
| Secret | Description |
| -------------------------- | -------------------------------------------------- |
| `PULUMI_BACKEND_URL` | PostgreSQL connection string for the state backend |
| `PULUMI_CONFIG_PASSPHRASE` | Passphrase to decrypt secrets in `Pulumi.dev.yaml` |
| `PULUMI_DEV_YAML` | Base64-encoded content of `Pulumi.dev.yaml` |
| Secret | Used by | Description |
| ------------------------------- | --------------------------- | ---------------------------------------------- |
| `PULUMI_BACKEND_URL` | all workflows | PostgreSQL connection string for state backend |
| `PULUMI_CONFIG_PASSPHRASE` | all workflows | Passphrase to decrypt secrets |
| `PROXMOX_INFRA_PULUMI_DEV_YAML` | `deploy-proxmox-infra.yaml` | Base64-encoded `proxmox-infra/Pulumi.dev.yaml` |
| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | `deploy-k8s-bootstrap.yaml` | Base64-encoded `k8s-bootstrap/Pulumi.dev.yaml` |
| `K8S_INFRA_PULUMI_DEV_YAML` | `deploy-k8s-infra.yaml` | Base64-encoded `k8s-infra/Pulumi.dev.yaml` |
`Pulumi.dev.yaml` is gitignored because it contains your encryption salt. Whenever it changes (e.g. after adding or rotating a secret), re-encode it and paste the output into the Gitea secret:
`Pulumi.dev.yaml` files are gitignored (they contain your encryption salt). Whenever one changes, re-encode and paste into the corresponding Gitea secret:
```bash
base64 -w 0 proxmox-infra/Pulumi.dev.yaml
base64 -w 0 k8s-bootstrap/Pulumi.dev.yaml
base64 -w 0 k8s-infra/Pulumi.dev.yaml
```
## Adapting this as a template
1. Fork or copy the repo
2. Update node names (`pve`, `pve-bckp`) and datastore IDs in `index.ts` to match your setup
2. Update Proxmox node names (`pve`, `pve-bckp`) and datastore IDs in `proxmox-infra/index.ts`
3. Add or remove VMs from the `nodeConfigs` array
4. Set your own secrets with `pulumi config set --secret`
5. Point the CI/CD workflow at your own Git instance
5. Point the CI/CD workflows at your own Gitea instance
## Roadmap
- LXC container management
- `k8s-apps` stack — application deployments on the cluster
- Docker / Compose stack provisioning
- Firewall rules (pfSense)
- Automated k3s bootstrapping (kubeconfig export)
- Additional worker nodes and storage volumes
- Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault
- Add a third bare metal proxmox instance to create an actual 3 node parity.
- Add a third bare metal Proxmox node for true 3-node HA parity