kasun/homelab-infrastructure-as-code
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

docs: updated documentation to current progress
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Details
Deploy k8s Infra / Pulumi Deploy (push) Successful in 58s
Details
Deploy k8s Apps / Pulumi Preview (push) Has been skipped
Details
Deploy k8s Apps / Pulumi Deploy (push) Successful in 48s
Details
Deploy Monitoring / Pulumi Preview (push) Has been skipped
Details
Deploy Monitoring / Pulumi Deploy (push) Successful in 59s
Details

Browse Source
This commit is contained in:
kasun
2026-06-11 22:36:20 +02:00
parent 6b3ed077ee
commit 13549a8d37
9 changed files with 290 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+82 -167
View File
@@ -1,239 +1,154 @@
# Homelab Infrastructure as Code
A Pulumi-based IaC template for managing a Proxmox homelab. The goal is to replace manual GUI configuration and ad-hoc YAML stacks with version-controlled, reproducible infrastructure — a highly available k3s cluster across multiple Proxmox nodes, bootstrapped and configured end-to-end from code.
A Pulumi-based IaC project for a Proxmox homelab: a highly-available k3s cluster, its cluster-level infrastructure, and the applications running on top of it — all provisioned and configured from code.
This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab.
## Why IaC for a homelab?
- **Reproducibility** — rebuild your entire environment from scratch with a single command
- **Reproducibility** — rebuild the entire environment from scratch from code
- **Version history** — every change is tracked in git; roll back at any time
- **Auditability** — diff infrastructure changes before applying them (`pulumi preview`)
- **Automation** — CI/CD handles deploys; no manual SSH into nodes
- **Portability** — swap node names, datastores, or credentials without touching logic
- **Auditability** — `pulumi preview` diffs infrastructure changes before they're applied
- **Automation** — CI/CD handles deploys; no manual SSH into nodes for routine changes
- **Portability** — swap node names, datastores, domains, or credentials without touching logic
## Repository layout
```
.
├── proxmox-infra/ # Stack 1 — VMs & DHCP on Proxmox + pfSense
│ ├── index.ts
│ ├── Pulumi.yaml
│ └── sdks/pfsense/ # Locally bundled @pulumi/pfsense SDK
├── k8s-bootstrap/ # Stack 2 — Install k3s on the VMs over SSH
│ ├── index.ts
│ └── Pulumi.yaml
├── k8s-infra/ # Stack 3 — Cluster-level infrastructure via Helm
│ ├── index.ts
│ └── Pulumi.yaml
└── .gitea/workflows/ # Gitea Actions — one workflow per stack
├── 01-proxmox-infra/ # Stack 1 — k3s VMs + DHCP on Proxmox/pfSense
├── 02-k8s-bootstrap/ # Stack 2 — install k3s on the VMs over SSH
├── 03-k8s-infra/ # Stack 3 — storage, cert-manager, kube-vip
├── 04-k8s-apps/ # Stack 4 — application workloads
├── 05-monitoring/ # Stack 5 — kube-prometheus-stack (Prometheus + Grafana)
├── .gitea/workflows/ # CI/CD — one workflow per stack
└── CLAUDE.md # Conventions & architecture notes (root + one per stack)
```
## Stack overview
Each stack is an independent Pulumi project (own `Pulumi.yaml`/`package.json`), all using the Pulumi stack name `dev`.
The three stacks are deployed in order and linked via Pulumi StackReferences — outputs from one stack flow automatically into the next.
## Stacks
Stacks are deployed in order and linked via `pulumi.StackReference` — outputs from one stack flow automatically into the next, so a stack must be deployed before anything depending on it will work.
```
proxmox-infra → k8s-bootstrap → k8s-infra
proxmox-infra → k8s-bootstrap → k8s-infra
↘ k8s-apps → monitoring
```
### Stack 1: `proxmox-infra`
### 1. proxmox-infra
Provisions a 5-node k3s cluster spread across two Proxmox hosts:
Provisions a 5-node k3s cluster across two Proxmox hosts, each VM a clone of an Ubuntu Noble (24.04) cloud-image template:
| VM name | Role | Proxmox node |
| ------------ | ------ | ------------ |
| k3s-master-1 | master | pve |
| k3s-master-2 | master | pve |
| k3s-worker-1 | worker | pve |
| k3s-master-3 | master | pve-bckp |
| k3s-worker-2 | worker | pve-bckp |
| VM | Role |
| --- | --- |
| k3s-master-1 | master |
| k3s-master-2 | master |
| k3s-worker-1 | worker |
| k3s-master-3 | master |
| k3s-worker-2 | worker |
Each VM is a full clone of an Ubuntu Noble (24.04) cloud-image template. Cloud-init injects the hostname, user credentials, and SSH keys at boot. An ED25519 SSH keypair is generated once and stored in Pulumi state — the public key goes into every VM, the private key is exported as a stack output for `k8s-bootstrap` to consume via StackReference.
It also generates an ED25519 SSH keypair (stored in Pulumi state, public key injected via cloud-init) and registers each VM's MAC address as a DHCPv4 static mapping in pfSense.
Each VM's MAC address is registered as a DHCPv4 static mapping in pfSense so nodes always receive their designated IPs.
**Providers:** `@muhlba91/pulumi-proxmoxve`, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
**Providers:** `@muhlba91/pulumi-proxmoxve` v8.x, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
### 2. k8s-bootstrap
**Exports:** `vmIds`, `ciRunnerPrivateKey`, `pve1Endpoint`, `pve1ApiToken`, `pve2Endpoint`, `pve2ApiToken`, `master1Ip` … `worker2Ip`
Starts the VMs via the Proxmox API and installs k3s over SSH: `k3s-master-1` initializes the cluster (`--cluster-init`, embedded etcd), `k3s-master-2`/`k3s-master-3` join as additional etcd nodes, and the two workers join as agents. Exports the cluster's `kubeconfig` as a secret for downstream stacks.
### Stack 2: `k8s-bootstrap`
### 3. k8s-infra
Installs k3s on the Proxmox VMs over SSH using `@pulumi/command`. Reads VM IDs, Proxmox credentials, and the SSH private key from `proxmox-infra` via StackReference — no manual key distribution needed.
Cluster-level infrastructure via Helm and `@pulumi/kubernetes`:
Deployment sequence:
| Component | What it does |
| --- | --- |
| NFS CSI Driver + `truenas-nfs` StorageClass | Dynamic NFS-backed PVCs from a TrueNAS share |
| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) |
| cert-manager + `letsencrypt-prod` ClusterIssuer | TLS certs via Let's Encrypt, DNS-01 through Cloudflare |
| kube-vip + kube-vip-cloud-provider | Announces a virtual IP and allocates it to LoadBalancer Services (e.g. Traefik) |
1. Start all 5 VMs via the Proxmox REST API
2. Wait for port 22 to open on each node
3. Install k3s on `k3s-master-1` with `--cluster-init` (embedded etcd)
4. Join `k3s-master-2` and `k3s-master-3` as additional etcd nodes
5. Join `k3s-worker-1` and `k3s-worker-2` as agent nodes
6. Read `/etc/rancher/k3s/k3s.yaml` from master-1, patch the server URL, export as a secret stack output
### 4. k8s-apps
**Exports:** `kubeconfig` (secret)
Application workloads exposed at `app.<domain>` via Traefik with a Let's Encrypt cert.
### Stack 3: `k8s-infra`
### 5. monitoring
Deploys cluster-level infrastructure via Helm and the `@pulumi/kubernetes` provider. Reads `kubeconfig` from `k8s-bootstrap` via StackReference.
| Component | What it does |
| -------------------- | -------------------------------------------------------------------------- |
| NFS CSI Driver | `csi-driver-nfs` Helm chart (v4.12.0) — enables dynamic NFS-backed PVCs |
| TrueNAS StorageClass | `truenas-nfs` StorageClass backed by an NFS share on TrueNAS |
| cert-manager | `cert-manager` Helm chart — certificate lifecycle management |
| ClusterIssuer | `letsencrypt-prod` — DNS-01 via Cloudflare, issues Let's Encrypt TLS certs |
`kube-prometheus-stack` Helm release — Prometheus (20Gi Longhorn PVC) and Grafana (2Gi PVC, ingress at `grafana.<domain>`).
## Prerequisites
- [Pulumi CLI](https://www.pulumi.com/docs/install/) installed
- Node.js 18+ and npm
- Two Proxmox nodes with API tokens
- pfSense with REST API credentials (used for DHCPv4 static mapping)
- TrueNAS with an NFS share (no API key needed — CSI driver connects via NFS protocol directly)
- A Cloudflare account with an API token scoped to DNS edit on your zone
- [Pulumi CLI](https://www.pulumi.com/docs/install/), Node.js 24+ and npm
- One or more Proxmox nodes with API tokens
- pfSense (DHCPv4 static mappings)
- TrueNAS with an NFS share (connected directly — no API key needed)
- A domain with DNS managed by Cloudflare (cert-manager DNS-01 + ingress hostnames)
- A self-hosted Pulumi state backend (PostgreSQL connection string)
- Gitea instance for CI/CD (optional for local use)
- Gitea instance for CI/CD (optional for local-only use)
## Getting started
Deploy the stacks **in order**. Each stack must be fully deployed before the next one runs.
### Stack 1 — `proxmox-infra`
```bash
cd proxmox-infra
npm install # also compiles the bundled pfSense SDK
pulumi stack init dev
```
Set secrets:
```bash
# Proxmox API credentials
pulumi config set --secret pve1Endpoint https://<proxmox-host-1>:8006
pulumi config set --secret pve1ApiToken <user>@pam!<token-id>=<uuid>
pulumi config set --secret pve2Endpoint https://<proxmox-host-2>:8006
pulumi config set --secret pve2ApiToken <user>@pam!<token-id>=<uuid>
# VM credentials
pulumi config set --secret k3sVmPassword <vm-password>
pulumi config set --secret sshPvePublicKey "ssh-ed25519 AAAA..."
# pfSense credentials (used for DHCPv4 static mappings)
pulumi config set --secret pfSenseUrl https://<pfsense-host>
pulumi config set --secret pfSenseUser <admin-username>
pulumi config set --secret pfSensePassword <admin-password>
# Static IP addresses assigned to each k3s node
pulumi config set --secret master1Ip <ip-for-k3s-master-1>
pulumi config set --secret master2Ip <ip-for-k3s-master-2>
pulumi config set --secret worker1Ip <ip-for-k3s-worker-1>
pulumi config set --secret master3Ip <ip-for-k3s-master-3>
pulumi config set --secret worker2Ip <ip-for-k3s-worker-2>
```
Deploy:
Deploy stacks **in order** — each must be fully `pulumi up`'d before the next stack's `StackReference` resolves. Set the shared backend env vars once per shell:
```bash
export PULUMI_BACKEND_URL=postgresql://<user>:<pass>@<host>/<db>
export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
pulumi preview # inspect before touching anything
pulumi up --yes
```
### Stack 2 — `k8s-bootstrap`
Then for each stack, in order:
```bash
cd ../k8s-bootstrap
cd 0N-<stack-dir>
npm install
pulumi stack init dev
pulumi config set --secret k3sToken "$(openssl rand -hex 32)"
```
Node IPs and Proxmox credentials are read automatically from `proxmox-infra` outputs via StackReference — do not set them here.
```bash
pulumi config set ... # required keys and secrets
pulumi up --yes
```
### Stack 3 — `k8s-infra`
Stack-specific notes:
**TrueNAS one-time setup** (before deploying this stack):
1. Create dataset `tank/k8s`
2. Add an NFS share for that dataset
- Acl Type: Set this to POSIX (or generic Unix permissions) rather than SMB/NFSv4 ACLs if you are on TrueNAS SCALE, as Kubernetes handles basic Unix permissions natively.
- Maproot User / Maproot Group: Set this to root and root
- Allowed Hosts/Networks: Restrict this share specifically to the IP addresses or the CIDR block of your Proxmox K3s nodes (e.g., 192.168.1.50)
3. In Network → Allowed Networks, permit your LAN subnet (e.g. `192.168.1.0/24`)
```bash
cd ../k8s-infra
npm install
pulumi stack init dev
# kubeconfig from k8s-bootstrap
KUBECONFIG=$(cd ../k8s-bootstrap && pulumi stack output kubeconfig --show-secrets)
pulumi config set --secret kubeconfig "$KUBECONFIG"
# TrueNAS NFS
pulumi config set --secret truenasHost <truenas-ip>
pulumi config set --secret truenasNfsPath /mnt/tank/k8s
# cert-manager + Cloudflare DNS-01
pulumi config set --secret cloudflareApiToken <cf-token>
pulumi config set --secret letsencryptEmail <your-email>
pulumi up --yes
```
1. **proxmox-infra** — set Proxmox API tokens, pfSense credentials, VM credentials/SSH public key, and a static IP per node.
2. **k8s-bootstrap** — set `k3sToken` (e.g. `openssl rand -hex 32`) and a static IP per node. Proxmox credentials and the SSH key come automatically from `proxmox-infra`.
3. **k8s-infra** — before deploying, create an NFS dataset/share on TrueNAS (POSIX ACLs, `root`/`root` maproot, restricted to your k3s nodes). Set the TrueNAS, Cloudflare, and `kubeVipAddress`/`kubeVipInterface` config.
4. **k8s-apps** — set `domain`, the base domain used for ingress hostnames.
5. **monitoring** — set `grafanaAdminPassword`.
## CI/CD (Gitea Actions)
One workflow per stack under `.gitea/workflows/`. Each triggers on changes to its own stack directory.
One workflow per stack, `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
| Event | Action |
| --------------------- | ------------------------------ |
| Pull request → `main` | `pulumi preview` (read-only) |
| Push to `main` | `pulumi refresh` + `pulumi up` |
| Manual trigger | `pulumi refresh` + `pulumi up` |
| Trigger | Stacks | Action |
| --- | --- | --- |
| PR → `main` | all | `pulumi preview` (read-only) |
| Push to `main` | 03–05 | `pulumi refresh` + `pulumi up` |
| Manual dispatch | all | `pulumi refresh` + `pulumi up` |
### Required Gitea secrets
Stacks 01/02 (VM provisioning, cluster bootstrap) deploy **only** via manual dispatch — they're riskier to automate.
Configure under **Settings → Actions → Secrets**:
### Required secrets
| Secret | Used by | Description |
| ------------------------------- | --------------------------- | ---------------------------------------------- |
| `PULUMI_BACKEND_URL` | all workflows | PostgreSQL connection string for state backend |
| `PULUMI_CONFIG_PASSPHRASE` | all workflows | Passphrase to decrypt secrets |
| `PROXMOX_INFRA_PULUMI_DEV_YAML` | `deploy-proxmox-infra.yaml` | Base64-encoded `proxmox-infra/Pulumi.dev.yaml` |
| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | `deploy-k8s-bootstrap.yaml` | Base64-encoded `k8s-bootstrap/Pulumi.dev.yaml` |
| `K8S_INFRA_PULUMI_DEV_YAML` | `deploy-k8s-infra.yaml` | Base64-encoded `k8s-infra/Pulumi.dev.yaml` |
| Secret | Used by |
| --- | --- |
| `PULUMI_BACKEND_URL`, `PULUMI_CONFIG_PASSPHRASE` | all workflows |
| `PROXMOX_INFRA_PULUMI_DEV_YAML` | 01-deploy-proxmox-infra |
| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | 02-deploy-k8s-bootstrap |
| `K8S_INFRA_PULUMI_DEV_YAML` | 03-deploy-k8s-infra |
| `K8S_APPS_PULUMI_DEV_YAML` | 04-deploy-k8s-apps |
| `MONITORING_PULUMI_DEV_YAML` | 05-deploy-monitoring |
`Pulumi.dev.yaml` files are gitignored (they contain your encryption salt). Whenever one changes, re-encode and paste into the corresponding Gitea secret:
Each `<STACK>_PULUMI_DEV_YAML` secret is a base64-encoded copy of that stack's gitignored `Pulumi.dev.yaml`. After changing local config, re-encode and update the secret:
```bash
base64 -w 0 proxmox-infra/Pulumi.dev.yaml
base64 -w 0 k8s-bootstrap/Pulumi.dev.yaml
base64 -w 0 k8s-infra/Pulumi.dev.yaml
base64 -w 0 0N-<stack>/Pulumi.dev.yaml
```
## Adapting this as a template
## Using this as a template
1. Fork or copy the repo
2. Update Proxmox node names (`pve`, `pve-bckp`) and datastore IDs in `proxmox-infra/index.ts`
3. Add or remove VMs from the `nodeConfigs` array
4. Set your own secrets with `pulumi config set --secret`
5. Point the CI/CD workflows at your own Gitea instance
2. Update Proxmox node names/datastores and the `nodeConfigs` array in `01-proxmox-infra/index.ts` for your VM topology
3. Set your own config/secrets per stack with `pulumi config set [--secret]` — see each stack's `CLAUDE.md` for the full list of required keys
4. Point the CI/CD workflows at your own Gitea/GitOps instance and update the secrets above
## Roadmap
- LXC container management
- `k8s-apps` stack — application deployments on the cluster
- Docker / Compose stack provisioning
- Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault
- Add a third bare metal Proxmox node for true 3-node HA parity
- Add a third bare-metal Proxmox node for true 3-node HA parity