import * as pulumi from "@pulumi/pulumi"; import * as proxmox from "@muhlba91/pulumi-proxmoxve"; import * as tls from "@pulumi/tls"; import * as pfsense from "@pulumi/pfsense"; const config = new pulumi.Config(); const pve1Endpoint = config.requireSecret("pve1Endpoint"); const pve2Endpoint = config.requireSecret("pve2Endpoint"); const pve3Endpoint = config.requireSecret("pve3Endpoint"); const pve1ApiToken = config.requireSecret("pve1ApiToken"); const pve2ApiToken = config.requireSecret("pve2ApiToken"); const pve3ApiToken = config.requireSecret("pve3ApiToken"); const pfSenseUrl = config.requireSecret("pfSenseUrl"); const pfSenseUser = config.requireSecret("pfSenseUser"); const pfSensePassword = config.requireSecret("pfSensePassword"); const master1Ip = config.requireSecret("master1Ip"); const master2Ip = config.requireSecret("master2Ip"); const master3Ip = config.requireSecret("master3Ip"); const worker1Ip = config.requireSecret("worker1Ip"); const worker2Ip = config.requireSecret("worker2Ip"); const worker3Ip = config.requireSecret("worker3Ip"); // --------------------------------------------------------------------------- // VM templates — one per node, cloned from the downloaded cloud image. // Templates are not started and have no cloud-init config; that is applied // per-clone so each node gets its own hostname (derived from VM name). // --------------------------------------------------------------------------- const templateSettings = { template: true, started: false, stopOnDestroy: true, scsiHardware: "virtio-scsi-pci", cdrom: { fileId: "none" }, cpu: { cores: 2, sockets: 1, type: "host", numa: true, }, memory: { dedicated: 2048, floating: 0, }, networkDevices: [{ bridge: "vmbr0", model: "virtio" }], serialDevices: [{}], vga: { type: "serial0" }, agent: { enabled: true, timeout: "30s" }, }; // --------------------------------------------------------------------------- // Providers — one per standalone Proxmox machine // --------------------------------------------------------------------------- interface PveNodeConfig { nodeName: string; hostLabel: string; isoStore: string; dataStore: string; endpoint: pulumi.Output; apiToken: pulumi.Output; } const pveNodeSeeds: PveNodeConfig[] = [ { nodeName: "pve", hostLabel: "pve-b550", isoStore: "pve-local-ext1", dataStore: "local-lvm", endpoint: pve1Endpoint, apiToken: pve1ApiToken, }, { nodeName: "pve-bckp", hostLabel: "pve-optiplex", isoStore: "local", dataStore: "local", endpoint: pve2Endpoint, apiToken: pve2ApiToken, }, { nodeName: "pve", hostLabel: "pve-esprimo", isoStore: "local", dataStore: "local-lvm", endpoint: pve3Endpoint, apiToken: pve3ApiToken, }, ]; const pveNodes = pveNodeSeeds .map((n) => ({ ...n, provider: new proxmox.Provider(n.hostLabel, { endpoint: n.endpoint, apiToken: n.apiToken, insecure: true, }), })) .map((n) => ({ ...n, ubuntuImagePve: new proxmox.download.File( `ubuntu-noble-${n.hostLabel}`, { nodeName: n.nodeName, datastoreId: n.isoStore, contentType: "import", fileName: "noble-server-cloudimg-amd64.qcow2", url: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img", overwrite: false, overwriteUnmanaged: true, }, { provider: n.provider }, ), })) .map((n) => ({ ...n, pveTemplate: new proxmox.VmLegacy( `k3s-template-${n.hostLabel}`, { ...templateSettings, nodeName: n.nodeName, name: "k3s-ubuntu-noble-template", disks: [ { interface: "scsi0", datastoreId: n.dataStore, importFrom: pulumi.interpolate`${n.ubuntuImagePve.datastoreId}:import/${n.ubuntuImagePve.fileName}`, size: 10, ssd: true, discard: "on", }, ], }, { provider: n.provider, ignoreChanges: ["disks"] }, ), })); // --------------------------------------------------------------------------- // Providers — PfSense // --------------------------------------------------------------------------- const pfSenseProvider = new pfsense.Provider("pfsense", { url: pfSenseUrl, username: pfSenseUser, password: pfSensePassword, tlsSkipVerify: true, }); // --------------------------------------------------------------------------- // CI runner SSH keypair — generated once, stored in Pulumi state backend. // Public key goes into every VM; private key is exported for k8s-bootstrap. // --------------------------------------------------------------------------- const ciRunnerKey = new tls.PrivateKey("ci-runner-key", { algorithm: "ED25519", }); // --------------------------------------------------------------------------- // k3s nodes — full clones of their respective templates // --------------------------------------------------------------------------- const k3sVmPassword = config.requireSecret("k3sVmPassword"); const pve1SshPublicKey = config.requireSecret("pve1SshPublicKey"); const pve2SshPublicKey = config.requireSecret("pve2SshPublicKey"); const pve3SshPublicKey = config.requireSecret("pve3SshPublicKey"); interface NodeConfig { name: string; role: "master" | "worker"; host: (typeof pveNodes)[number]; // one row from the pveNodes chain ip: pulumi.Output; longhornDiskSize?: number; } const hostFor = (label: string) => pveNodes.find((n) => n.hostLabel === label)!; const nodeConfigs: NodeConfig[] = [ { name: "k3s-master-1", role: "master", host: hostFor("pve-b550"), ip: master1Ip, }, { name: "k3s-master-2", role: "master", host: hostFor("pve-esprimo"), ip: master2Ip, }, { name: "k3s-worker-1", role: "worker", host: hostFor("pve-b550"), ip: worker1Ip, longhornDiskSize: 50, }, { name: "k3s-master-3", role: "master", host: hostFor("pve-optiplex"), ip: master3Ip, }, { name: "k3s-worker-2", role: "worker", host: hostFor("pve-optiplex"), ip: worker2Ip, longhornDiskSize: 50, }, { name: "k3s-worker-3", role: "worker", host: hostFor("pve-esprimo"), ip: worker3Ip, longhornDiskSize: 50, }, ]; const k3sVms = nodeConfigs.map( (node) => new proxmox.VmLegacy( node.name, { nodeName: node.host.nodeName, name: node.name, description: "k3s " + node.role + " node — managed by Pulumi", tags: ["k3s", node.role], clone: { vmId: node.host.pveTemplate.vmId, full: true, datastoreId: node.host.dataStore, }, cpu: { cores: 2, sockets: 1, type: "host", numa: true, }, memory: { dedicated: node.role === "master" ? 4096 : 8192, floating: 0, }, disks: [ { interface: "scsi0", datastoreId: node.host.dataStore, size: 35, ssd: true, discard: "on", }, ...(node.longhornDiskSize == null ? [] : [ { interface: "scsi1", datastoreId: node.host.dataStore, size: node.longhornDiskSize, ssd: true, discard: "on", }, ]), ], initialization: { datastoreId: node.host.dataStore, ipConfigs: [{ ipv4: { address: "dhcp" } }], userAccount: { username: "ubuntu", password: k3sVmPassword, keys: [ pve1SshPublicKey.apply((k) => k.trim()), pve2SshPublicKey.apply((k) => k.trim()), pve3SshPublicKey.apply((k) => k.trim()), ciRunnerKey.publicKeyOpenssh.apply((k) => k.trim()), ], }, }, networkDevices: [{ bridge: "vmbr0", model: "virtio" }], scsiHardware: "virtio-scsi-pci", serialDevices: [{}], vga: { type: "serial0" }, agent: { enabled: true, timeout: "30s" }, cdrom: { fileId: "none" }, started: false, stopOnDestroy: true, }, { provider: node.host.provider, retainOnDelete: true, ignoreChanges: ["clone", "started"], }, ), ); k3sVms.forEach((vmResource, i) => { const assignedMac = vmResource.networkDevices.apply( (nic) => nic[0].macAddress, ); new pfsense.Dhcpv4Staticmapping( `${nodeConfigs[i].name}-dhcp`, { interface: "lan", macAddress: assignedMac, ipAddress: nodeConfigs[i].ip, hostname: nodeConfigs[i].name, }, { dependsOn: vmResource, provider: pfSenseProvider }, ); }); export const vmIds = { master1: k3sVms[0].vmId, master2: k3sVms[1].vmId, worker1: k3sVms[2].vmId, master3: k3sVms[3].vmId, worker2: k3sVms[4].vmId, worker3: k3sVms[5].vmId, }; // CI runner SSH private key — consumed by k8s-bootstrap via StackReference. export const ciRunnerPrivateKey = pulumi.secret(ciRunnerKey.privateKeyOpenssh); export { pve1Endpoint, pve1ApiToken, pve2Endpoint, pve2ApiToken, pve3Endpoint, pve3ApiToken, }; export { master1Ip, master2Ip, worker1Ip, master3Ip, worker2Ip, worker3Ip };