GPU passthrough on Proxmox, and why not an LXC

There are two ways to give a virtual machine a GPU on Proxmox, and most homelab guides pick the wrong one for the wrong reason. The usual advice is to run the workload in an LXC container, bind mount the device nodes, and skip the whole VFIO ritual. It is lighter, it shares the card between containers, and it avoids a page of kernel flags. All of that is true. It also did not work here, for a reason that has nothing to do with containers.

The reason it has to be a VM

A GPU in an LXC container is not virtualised. The container uses the host’s driver, through the host’s kernel. So the host has to have a working NVIDIA driver.

Proxmox ships a very new kernel. The NVIDIA DKMS package in Debian stable is built against a much older one. The module does not compile, and no amount of persuading DKMS changes that: the driver source predates the kernel’s internal API. The host cannot have a driver, so no container on that host can have a GPU.

Passthrough sidesteps the problem entirely. The host never loads a graphics driver at all. It binds the card to vfio-pci, hands the whole device to a guest, and the guest runs an ordinary distribution kernel that the vendor driver has supported for a year. Same card, same driver package, and it builds on the first try.

This is worth stating plainly because the internet frames it as containers versus VMs, light versus heavy. That is not the decision. The decision is whose kernel has to load the driver, and on Proxmox the answer had better not be the host’s 🙂

Binding the card

Three pieces. IOMMU on at the bootloader, the device bound to vfio-pci by ID, and the open source driver kept away from it.

# /etc/default/grub, on AMD. Intel uses intel_iommu=on.
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
update-grub

# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1f91,10de:10fa

Both IDs, not one. A discrete NVIDIA card presents as at least two functions, the VGA controller and its HDMI audio device, and they live in the same IOMMU group. Bind only the graphics function and the guest gets a card whose audio half is still owned by the host, which fails in a way that looks like a driver bug and is not. 🐛

# /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0

Then update-initramfs -u, reboot, and check the card actually landed in the right driver:

lspci -nnk -d 10de:
# Kernel driver in use: vfio-pci

If that line still says nouveau, nothing else in this post will work. Fix it there.

The reset bug decides placement

Both nodes have an NVIDIA card. They do not behave the same way, and the difference decided which workloads live where.

The laptop’s GTX 1650 resets cleanly. Stop the guest, start it again, the card comes back, and the host never notices. That guest can be rebooted all day.

Cards that do not reset cleanly hit the well known GPU reset bug: the device is left in a state the kernel cannot recover without a full host power cycle. The second guest start fails, and the fix is rebooting the hypervisor. Theoretically that is fine on a lab machine. In practice a hypervisor that needs a reboot to restart one guest is not where you put a guest you expect to restart. 🔌

So the split is by behaviour, not by size. Work that gets restarted often goes on the card that resets. Work that starts once and runs goes on the bigger card.

What actually runs on it

The small always on side is a 4 GB card, which sounds like nothing and is enough for a 3B parameter model to run entirely on the GPU. Fully resident is the thing that matters. A model that fits in VRAM answers in a second, and the same model spilling into system RAM takes twenty. 4 GB is not a compromised version of a big GPU, it is a hard boundary with a very different experience on each side. 🧱

The guest is Ubuntu 24.04 with q35 and OVMF, the vendor driver, and the NVIDIA container toolkit so Docker can see the card. From there it is a normal compose file.

nvidia-smi                 # inside the guest, not the host
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If the second command works, everything above it is correct.

One thing to check before you start

The backup story changes the moment a GPU is passed through, and it changes silently. A stop mode vzdump has to start a second QEMU process to read the disk, and that process cannot acquire a card the guest still owns. So the backup fails while every other guest on the host backs up fine, which is the kind of failure that hides for weeks. 💾

Check it the day you set passthrough up, not the day you need a restore. qm config <id> will not tell you; the backup job log will.

Next: VLANs, and a config flag that reads as broken.