kasun
13549a8d37
Deploy k8s Infra / Pulumi Preview (push) Has been skipped
Deploy k8s Infra / Pulumi Deploy (push) Successful in 58s
Deploy k8s Apps / Pulumi Preview (push) Has been skipped
Deploy k8s Apps / Pulumi Deploy (push) Successful in 48s
Deploy Monitoring / Pulumi Preview (push) Has been skipped
Deploy Monitoring / Pulumi Deploy (push) Successful in 59s
155 lines
6.8 KiB
Markdown
155 lines
6.8 KiB
Markdown
# Homelab Infrastructure as Code
|
||
|
||
A Pulumi-based IaC project for a Proxmox homelab: a highly-available k3s cluster, its cluster-level infrastructure, and the applications running on top of it — all provisioned and configured from code.
|
||
|
||
This repo is intentionally abstract: credentials are never hardcoded, making it easy to fork and adapt as a template for your own homelab.
|
||
|
||
## Why IaC for a homelab?
|
||
|
||
- **Reproducibility** — rebuild the entire environment from scratch from code
|
||
- **Version history** — every change is tracked in git; roll back at any time
|
||
- **Auditability** — `pulumi preview` diffs infrastructure changes before they're applied
|
||
- **Automation** — CI/CD handles deploys; no manual SSH into nodes for routine changes
|
||
- **Portability** — swap node names, datastores, domains, or credentials without touching logic
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
.
|
||
├── 01-proxmox-infra/ # Stack 1 — k3s VMs + DHCP on Proxmox/pfSense
|
||
├── 02-k8s-bootstrap/ # Stack 2 — install k3s on the VMs over SSH
|
||
├── 03-k8s-infra/ # Stack 3 — storage, cert-manager, kube-vip
|
||
├── 04-k8s-apps/ # Stack 4 — application workloads
|
||
├── 05-monitoring/ # Stack 5 — kube-prometheus-stack (Prometheus + Grafana)
|
||
├── .gitea/workflows/ # CI/CD — one workflow per stack
|
||
└── CLAUDE.md # Conventions & architecture notes (root + one per stack)
|
||
```
|
||
|
||
Each stack is an independent Pulumi project (own `Pulumi.yaml`/`package.json`), all using the Pulumi stack name `dev`.
|
||
|
||
## Stacks
|
||
|
||
Stacks are deployed in order and linked via `pulumi.StackReference` — outputs from one stack flow automatically into the next, so a stack must be deployed before anything depending on it will work.
|
||
|
||
```
|
||
proxmox-infra → k8s-bootstrap → k8s-infra
|
||
↘ k8s-apps → monitoring
|
||
```
|
||
|
||
### 1. proxmox-infra
|
||
|
||
Provisions a 5-node k3s cluster across two Proxmox hosts, each VM a clone of an Ubuntu Noble (24.04) cloud-image template:
|
||
|
||
| VM | Role |
|
||
| --- | --- |
|
||
| k3s-master-1 | master |
|
||
| k3s-master-2 | master |
|
||
| k3s-worker-1 | worker |
|
||
| k3s-master-3 | master |
|
||
| k3s-worker-2 | worker |
|
||
|
||
It also generates an ED25519 SSH keypair (stored in Pulumi state, public key injected via cloud-init) and registers each VM's MAC address as a DHCPv4 static mapping in pfSense.
|
||
|
||
**Providers:** `@muhlba91/pulumi-proxmoxve`, `@pulumi/pfsense` (locally bundled), `@pulumi/tls`
|
||
|
||
### 2. k8s-bootstrap
|
||
|
||
Starts the VMs via the Proxmox API and installs k3s over SSH: `k3s-master-1` initializes the cluster (`--cluster-init`, embedded etcd), `k3s-master-2`/`k3s-master-3` join as additional etcd nodes, and the two workers join as agents. Exports the cluster's `kubeconfig` as a secret for downstream stacks.
|
||
|
||
### 3. k8s-infra
|
||
|
||
Cluster-level infrastructure via Helm and `@pulumi/kubernetes`:
|
||
|
||
| Component | What it does |
|
||
| --- | --- |
|
||
| NFS CSI Driver + `truenas-nfs` StorageClass | Dynamic NFS-backed PVCs from a TrueNAS share |
|
||
| Longhorn | Default block storage (`longhorn` StorageClass, 3 replicas) |
|
||
| cert-manager + `letsencrypt-prod` ClusterIssuer | TLS certs via Let's Encrypt, DNS-01 through Cloudflare |
|
||
| kube-vip + kube-vip-cloud-provider | Announces a virtual IP and allocates it to LoadBalancer Services (e.g. Traefik) |
|
||
|
||
### 4. k8s-apps
|
||
|
||
Application workloads exposed at `app.<domain>` via Traefik with a Let's Encrypt cert.
|
||
|
||
### 5. monitoring
|
||
|
||
`kube-prometheus-stack` Helm release — Prometheus (20Gi Longhorn PVC) and Grafana (2Gi PVC, ingress at `grafana.<domain>`).
|
||
|
||
## Prerequisites
|
||
|
||
- [Pulumi CLI](https://www.pulumi.com/docs/install/), Node.js 24+ and npm
|
||
- One or more Proxmox nodes with API tokens
|
||
- pfSense (DHCPv4 static mappings)
|
||
- TrueNAS with an NFS share (connected directly — no API key needed)
|
||
- A domain with DNS managed by Cloudflare (cert-manager DNS-01 + ingress hostnames)
|
||
- A self-hosted Pulumi state backend (PostgreSQL connection string)
|
||
- Gitea instance for CI/CD (optional for local-only use)
|
||
|
||
## Getting started
|
||
|
||
Deploy stacks **in order** — each must be fully `pulumi up`'d before the next stack's `StackReference` resolves. Set the shared backend env vars once per shell:
|
||
|
||
```bash
|
||
export PULUMI_BACKEND_URL=postgresql://<user>:<pass>@<host>/<db>
|
||
export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
|
||
```
|
||
|
||
Then for each stack, in order:
|
||
|
||
```bash
|
||
cd 0N-<stack-dir>
|
||
npm install
|
||
pulumi stack init dev
|
||
pulumi config set ... # required keys and secrets
|
||
pulumi up --yes
|
||
```
|
||
|
||
Stack-specific notes:
|
||
|
||
1. **proxmox-infra** — set Proxmox API tokens, pfSense credentials, VM credentials/SSH public key, and a static IP per node.
|
||
2. **k8s-bootstrap** — set `k3sToken` (e.g. `openssl rand -hex 32`) and a static IP per node. Proxmox credentials and the SSH key come automatically from `proxmox-infra`.
|
||
3. **k8s-infra** — before deploying, create an NFS dataset/share on TrueNAS (POSIX ACLs, `root`/`root` maproot, restricted to your k3s nodes). Set the TrueNAS, Cloudflare, and `kubeVipAddress`/`kubeVipInterface` config.
|
||
4. **k8s-apps** — set `domain`, the base domain used for ingress hostnames.
|
||
5. **monitoring** — set `grafanaAdminPassword`.
|
||
|
||
## CI/CD (Gitea Actions)
|
||
|
||
One workflow per stack, `.gitea/workflows/0N-deploy-<stack>.yaml`, scoped to changes under that stack's directory.
|
||
|
||
| Trigger | Stacks | Action |
|
||
| --- | --- | --- |
|
||
| PR → `main` | all | `pulumi preview` (read-only) |
|
||
| Push to `main` | 03–05 | `pulumi refresh` + `pulumi up` |
|
||
| Manual dispatch | all | `pulumi refresh` + `pulumi up` |
|
||
|
||
Stacks 01/02 (VM provisioning, cluster bootstrap) deploy **only** via manual dispatch — they're riskier to automate.
|
||
|
||
### Required secrets
|
||
|
||
| Secret | Used by |
|
||
| --- | --- |
|
||
| `PULUMI_BACKEND_URL`, `PULUMI_CONFIG_PASSPHRASE` | all workflows |
|
||
| `PROXMOX_INFRA_PULUMI_DEV_YAML` | 01-deploy-proxmox-infra |
|
||
| `K8S_BOOTSTRAP_PULUMI_DEV_YAML` | 02-deploy-k8s-bootstrap |
|
||
| `K8S_INFRA_PULUMI_DEV_YAML` | 03-deploy-k8s-infra |
|
||
| `K8S_APPS_PULUMI_DEV_YAML` | 04-deploy-k8s-apps |
|
||
| `MONITORING_PULUMI_DEV_YAML` | 05-deploy-monitoring |
|
||
|
||
Each `<STACK>_PULUMI_DEV_YAML` secret is a base64-encoded copy of that stack's gitignored `Pulumi.dev.yaml`. After changing local config, re-encode and update the secret:
|
||
|
||
```bash
|
||
base64 -w 0 0N-<stack>/Pulumi.dev.yaml
|
||
```
|
||
|
||
## Using this as a template
|
||
|
||
1. Fork or copy the repo
|
||
2. Update Proxmox node names/datastores and the `nodeConfigs` array in `01-proxmox-infra/index.ts` for your VM topology
|
||
3. Set your own config/secrets per stack with `pulumi config set [--secret]` — see each stack's `CLAUDE.md` for the full list of required keys
|
||
4. Point the CI/CD workflows at your own Gitea/GitOps instance and update the secrets above
|
||
|
||
## Roadmap
|
||
|
||
- Migrate secrets management to [OpenBao](https://openbao.org/) — replace `PULUMI_CONFIG_PASSPHRASE` and manual `Pulumi.dev.yaml` encoding with a self-hosted vault
|
||
- Add a third bare-metal Proxmox node for true 3-node HA parity
|