A two node Proxmox cluster, and the quorum problem

A two node Proxmox cluster has two votes and needs a majority, so losing either node leaves the other one without quorum. I fixed that on my cluster with corosync’s two_node and wait_for_all options. This post covers what each one does, the split brain risk it accepts, and the RAM and storage limits that show up next.

What happens when one node goes down?

Each node gets one vote. A majority of two is two. So one node down is one vote short, and the survivor loses quorum.

Without quorum the cluster file system goes read only. /etc/pve cannot be changed, and starting a guest fails with cluster not ready - no quorum?. Half the cluster is up and it can do nothing.

In a rack, a node being down is an incident. In my lab it is routine: the second node dual boots, so it leaves the cluster every time I play a game.

Is a QDevice the better fix?

On paper, yes. The Proxmox docs recommend a QDevice for two node clusters. It is corosync-qnetd on a third machine, which gives the cluster one extra vote and keeps almost no state. It talks to the nodes over TCP and does not even need to be on the same LAN.

There is a Pi in this lab that could do it, theoretically. I have not set one up yet, so the rest of this post is the fix I actually run.

two_node and wait_for_all

Both options go in the quorum section of /etc/pve/corosync.conf:

quorum {
  provider: corosync_votequorum
  two_node: 1
  wait_for_all: 0
}

two_node: 1 sets quorum to 1, so a single surviving node stays quorate. The catch is in the votequorum man page: two_node: 1 turns on wait_for_all by itself.

With wait_for_all on, the cluster only becomes quorate for the first time after every node has been seen at once. A node that survives an outage keeps quorum. A node that cold boots alone waits for a machine that is not coming. wait_for_all: 0 overrides it.

How to edit corosync.conf on Proxmox

Each node has two copies, /etc/pve/corosync.conf and /etc/corosync/corosync.conf. Edit the one in /etc/pve. It propagates to the local copy on every node, and not the other way round. Changes apply as soon as the file is saved, so work on a copy:

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new
nano /etc/pve/corosync.conf.new        # add the two lines, increment config_version
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak
mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf
journalctl -b -u corosync              # check it applied
pvecm status

If it did not apply on its own, systemctl restart corosync. I also kept a copy outside /etc/pve. Copies are free.

The earlier version of this post reloaded with corosync-cfgtool -R and backed up the file under /etc/corosync. That is not the procedure the Proxmox docs describe, and a change made only under /etc/corosync does not propagate.

What does it cost?

Split brain becomes possible. With quorum at 1, both halves of a partition are quorate. If both nodes are up and cannot see each other, each one can change its own copy of /etc/pve and start guests.

For that, both nodes have to be running and the link between them broken. Mine sit on one switch, so that means a cable, a NIC or the switch itself. A dead switch cuts both nodes off from everything anyway. Against a second node that leaves every time I play a game, I take that trade.

RAM, not cores

Cores were never the limit on either node. RAM was.

Ballooning makes that survivable and confusing. Proxmox guarantees each guest its minimum, and once host usage passes the target (80% by default) guests give memory back. A guest with 2 GB and a 1 GB floor showed about 940 MiB usable on a busy host. A build that assumed 2 GB got OOM killed, and the message said nothing about ballooning.

The GPU guest made it worse. VFIO pins the memory it maps for DMA, so that guest cannot balloon at all, and its locked memory read about 16 GB. Every other guest sat on its floor. The floors added up to 32,768 MiB, on a host with 31,186 MiB.

Two habits fix most of it. Give every guest some swap, so a spike slows it down instead of killing it. And check free -m inside the guest, because the config is a request, not a promise.

Thin pools deserve the same suspicion. Thin provisioning lets the disks you hand out add up to more than the pool holds. When the pool fills, LVM queues writes for 60 seconds by default, then fails them, and the lvmthin man page warns that this can damage file systems.

How to move a guest between nodes

Live migration with local disks is one command:

qm migrate <vmid> <target-node> --online --with-local-disks

The snag I hit was an install ISO still attached to ide2 from the day the VM was built. The ISO sat on local storage, and migration refuses that with local cdrom image. Empty the drive and it runs:

qm set <vmid> --ide2 none,media=cdrom

What this does not cover

• HA. The Proxmox HA docs ask for at least three nodes, and two_node is not a way around that.
• QDevice setup, because I have not done it yet.