Starflinger
← All posts

· 7 min read

Setting up a Proxmox cluster with Claude Code, CLI-only

Four mini PCs, twelve VMs, zero clicks in the web UI. How a hands-on AI-in-the-terminal workflow turns ad-hoc cluster admin into idempotent scripts you actually keep.

  • proxmox
  • claude-code
  • infrastructure
  • cli

The Proxmox web UI is fine. The buttons are where you’d expect, the wizards work, and you’ll never lose a Tuesday because someone fat-fingered a qm set flag.

I just don’t open it.

Four Bee-Link mini PCs sat on my desk last December. They had to become a 4-node Proxmox 9.1 cluster: ZFS mirrors, a cloud-init template per VM type, a Tailscale subnet router, a GPU server next to them, twelve VMs across the lot. I could plug a monitor into each node and click through the wizards — or write the answer once, in scripts, and stamp it onto every node identically. I picked the second, with Claude Code as the way to write it.

Why CLI over UI for cluster work

The web UI is great at single moves. It’s terrible at the same move four times.

When you’re pointing-and-clicking, your record of what happened is your memory. If node 2 ends up subtly different from node 1 because you scrolled past one checkbox during the install — and you will — you find out three months later when one host can’t replicate a VM and you can’t remember whether you set swappiness on this one or that one. (I have a 99-proxmox.conf on node 1 that’s just vm.swappiness=10 repeated nine times. I don’t know how. Neither does the UI.)

CLI changes the contract: every command is text, every text is reviewable, every review is committable. The cluster ends up describing itself in a directory of shell scripts, and the next time I add a node, I run the scripts.

What I wanted was infrastructure-as-code without the upfront tax of Terraform — no resource graph to declare, no provider plugins to bisect, no state file. I wanted to write bash and have someone smarter than me catch when I was about to write something dumb.

The bootstrap: an autoinstall ISO that knows which node it is

Proxmox 9 ships an autoinstall TOML you bake into the install ISO. One ISO per node, each pinning a hostname (pve-node1, pve-node2, …, pve-gpu). The TOML drops a first-boot.sh onto the new system; that’s the whole orchestration spine.

The first thing first-boot.sh does is figure out which node it just booted on:

HOSTNAME=$(hostname -s)
if [[ "$HOSTNAME" =~ ^pve-node([0-9]+)$ ]]; then
    NODE_NUM="${BASH_REMATCH[1]}"
    NODE_TYPE="standard"
elif [[ "$HOSTNAME" == "pve-gpu" ]]; then
    NODE_NUM=99
    NODE_TYPE="gpu"
else
    echo "ERROR: hostname '$HOSTNAME' doesn't match pattern" >&2
    exit 1
fi

Everything downstream — IP, ZFS layout, GPU drivers or not, create the cluster vs join it — falls out of those four lines. Node 1 creates the cluster. Nodes 2–4 wait for node 1 to be reachable and call pvecm add. The GPU node skips Proxmox entirely (Ubuntu 24.04 with NVIDIA Container Toolkit, exporting a ZFS pool over NFS as the cluster’s backup target).

The first version of first-boot.sh was a spaghetti of ifs. I asked Claude Code to break it into numbered phases — 00-config.sh, 02-cluster.sh, 02-storage.sh, 06-vms.sh, 07-monitoring.sh, 08-backup.sh — each idempotent, each safe to re-run. It refactored. I read the diff. I ran it on node 4 first because that node had no live VMs. It didn’t break.

The storage gotcha that cost me an evening

If you set zfs.hdsize=100 in the autoinstall (you should — 100 GB is plenty for rpool), the installer creates partitions 1–3 and leaves the rest of each NVMe unpartitioned. The 900 GB the docs imply is yours to use is yours to first carve out. The Proxmox UI eventually gets there with point-and-click. The CLI gets there with sgdisk:

for dev in "${RPOOL_DEVS[@]}"; do
    p4=$(get_part_path "$dev" 4)
    if [[ ! -b "$p4" ]]; then
        sgdisk -n 4:0:0 -t 4:bf01 "$dev"
    fi
done
partprobe; udevadm settle

-n 4:0:0 means partition 4, from the next available sector to the end of the disk. -t 4:bf01 types it Solaris/ZFS. Run on both NVMe drives, then zpool create vmpool mirror /dev/.../p4 /dev/.../p4 and you have a 900 GB mirrored pool for VM storage.

The bug I shipped first: the helper that turned a partition path back into a base device used sed 's/-part[0-9]*$//'. That works for /dev/disk/by-id/...-part3 (strips -part3), but on /dev/nvme0n1p3 it does nothing — there’s no -part. The script worked perfectly on the test path and silently failed on real NVMe-by-path lookups. Fixing it took five minutes; finding it took an hour, because the failure mode was “everything looks fine” until you read zpool status very carefully.

The first VM, declaratively

Once vmpool exists, every VM in this cluster is the same five-step recipe:

qm create $VMID \
  --name $VM_NAME \
  --memory $MEM --cores $CORES \
  --net0 virtio,bridge=vmbr0 \
  --ostype l26 --machine q35 --bios ovmf \
  --efidisk0 vmpool:0,format=raw,efitype=4m,pre-enrolled-keys=1 \
  --scsihw virtio-scsi-single

qm importdisk $VMID /var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img vmpool
qm set $VMID --scsi0 vmpool:vm-$VMID-disk-1,iothread=1,ssd=1,discard=on
qm resize $VMID scsi0 ${DISK_GB}G

qm set $VMID --ide2 vmpool:cloudinit
qm set $VMID --boot order=scsi0 --serial0 socket --vga serial0
qm set $VMID --cicustom "user=local:snippets/${VM_TYPE}-user-data.yaml" \
            --ipconfig0 "ip=$VM_IP/24,gw=$GW"

qm start $VMID

Five qm calls. Idempotent if I script the create-or-skip logic. The cloud-init user-data lives at local:snippets/<type>-user-data.yaml — one per VM role. Add a new VM type, drop a new YAML, point one variable. No clicks, no wizard, no “I think I left the cloud-init checkbox unticked.”

The Proxmox UI can do all of this. It just can’t do it the same way every time.

The cloud-init / sshd_config trap

Here’s the one that ran silently for three months before I noticed.

Cloud-init seeds a fresh Ubuntu image with /etc/ssh/sshd_config.d/50-cloud-init.conf containing PasswordAuthentication yes — that’s how it lets you log in on first boot. My hardening cloud-init then writes a second file: /etc/ssh/sshd_config.d/99-hardening.conf with PasswordAuthentication no. The standard last-word-wins reflex, like sudoers.d or sysctl.d.

It’s wrong. sshd processes drop-ins alphabetically with first-match-wins semantics. 50-cloud-init.conf comes before 99-hardening.conf, so the first file sets PasswordAuthentication yes and the second is silently ignored. The fix is one character:

 write_files:
-  - path: /etc/ssh/sshd_config.d/99-hardening.conf
+  - path: /etc/ssh/sshd_config.d/10-hardening.conf
     permissions: '0644'
     content: |
       PasswordAuthentication no
       PermitRootLogin no
       KbdInteractiveAuthentication no

You won’t notice from the outside. SSH still works. Public-key auth still works. The login banner says hardening is enabled. The only way to find it is to probe what’s not supposed to work:

ssh -o PreferredAuthentications=password \
    -o PubkeyAuthentication=no \
    ubuntu@vm-ip
# password prompt that should not exist

I caught it on a GitHub Actions runner whose job logs are public on Gitea, whose IP is reachable on the LAN, and whose sshd_config.d had had the (dud) 99-hardening.conf since November.

Two lessons:

  1. When configuring a daemon via drop-in directories, check the merge semantics. Sshd, nginx.conf, and sysctl.d behave differently on duplicate keys; “the first obtained value will be used” is buried in sshd_config(5), not on the line that introduces Include.
  2. Test the negative case. “SSH still works” is not the same as “hardening is in effect.” A ssh -o PreferredAuthentications=password probe in your post-provisioning smoke tests catches this in one line.

Where Claude Code actually closes the last mile

I want to be careful here, because “AI agent runs your infra” is a sentence that should make any sensible operator nervous. The way I use it in practice is closer to a weird, helpful pair than to an autopilot:

Iterative discovery. “Cloud-init isn’t applying — can you look?” → It runs qm cloudinit dump, notices the snippet path is wrong, proposes a fix, waits for me to OK it. Faster than the Stack Overflow tab. Same commands.

Fan-out. “Snapshot every VM on node3 before I reboot it.” → It generates the loop, shows it, runs it. Same task in the UI is six clicks per VM — the kind of thing where I’d skip the snapshot and regret it.

State auditing. “What’s running on .82 and what’s it consuming?” → qm list, qm status, pvesh get /nodes/pve-node3/status, formatted. The UI gives you the same answer across three screens.

Catching bugs. It’s not magic. It still has to read the script. But it reads faster than I do, without the prior of “this worked yesterday.” When I asked why RPOOL_DEVS was empty, the first thing it noticed was that the sed didn’t match the input.

What it is not: a thing I let drive long-running destructive operations on its own. Every command that does something irreversible — deleting a VM, destroying a pool, force-pushing — pauses for confirmation. That’s a setting, not an instinct. Don’t run a coding agent against your cluster without it.

Caveats

  • The web UI is still useful for one-offs. VM console, cloud-init drive menus, anything you’d Google otherwise. CLI-first doesn’t mean CLI-only — it means CLI is where the durable state lives.
  • You will write helper functions you regret. Idempotent scripts are harder than non-idempotent ones, and Claude Code happily wrote me a “robust” function that was robust against the wrong thing. Read every diff.
  • For a five-person team with real change-management, the right answer is Terraform + Ansible. For a one-person consultancy with four nodes, four hundred lines of bash is genuinely simpler.
  • The cluster ends up describing itself in your CLAUDE.md. That’s the point and a hostage to fortune: when docs drift from reality, the agent gets confidently wrong. Re-run audit prompts against live state periodically.

The meta-loop

The same cluster runs my LLM inference: vLLM and SGLang on a GPU server with two RTX PRO 6000s, fronted by LiteLLM. The Claude API calls that wrote half the bootstrap scripts could, in principle, go through that local stack.

The loop doesn’t close today — Claude Code still hits Anthropic’s API, and probably should: my single GPU server isn’t going to beat their fleet on latency or model quality. But the local stack is what I show clients when pitching sovereign LLM deployments. This is what running AI on your own hardware looks like. I drove it here from a terminal. Here is the script.

That’s a different pitch from “trust me, I’ve used vLLM before.” Closer to “here’s the script, here’s the cluster, here’s the bill of materials, here’s the failure I had at 11 p.m. last Tuesday and what I changed to fix it.” For clients weighing self-hosted AI against cloud lock-in, the more honest one.

The cluster doesn’t need a UI. It needs scripts a stranger could read. The agent in the terminal is the strangest pair-programmer I’ve ever had — and the most patient — willing to remind me that sed 's/-part[0-9]*$//' doesn’t match what I think it matches.

— @starflinger.eu · Vienna, May 2026