Configuring GPU Passthrough in Proxmox for LLMs

Introduction to Virtualization of AI Hardware

When you set up a local AI infrastructure within a virtual environment, you quickly run into the limitations of hypervisor abstraction. By default, Proxmox VE does not share the graphics card's compute power directly with virtual machines (VMs). To achieve maximum inference speed for large language models, direct hardware access is necessary. This process, known as PCIe Passthrough or VFIO (Virtual Function I/O), ensures that a virtual machine gets exclusive control over the physical GPU.

Setting up this functionality correctly requires in-depth configuration of the underlying Linux kernel, the motherboard's BIOS/UEFI settings, and the specific parameters of the virtual machine. Although this is more complex than an installation on a dedicated operating system, it offers enormous benefits in terms of resource isolation and server management. Anyone who wants to understand more deeply how the compute models process data internally during these processes can read the theoretical background on the attention mechanism in AI explained clearly.

An important weak point of this architecture is its inflexibility. Once a GPU is assigned to one specific VM via passthrough, the Proxmox host can no longer use that card for other containers or GUI tasks. This means you need to carefully plan which workloads run on the host and which resources remain reserved for the inference machine. Moreover, setting up a stable production server often requires that later steps such as Configuring vLLM for high throughput on Linux connect seamlessly to this virtualization layer.

Step 1: Activating IOMMU and BIOS Virtualization

The fundamental technology behind PCIe passthrough is IOMMU (Input-Output Memory Management Unit). This system allows the kernel to map memory addresses of physical devices to the virtual addresses of a specific VM. Without a correctly configured IOMMU group, direct hardware access is impossible.

Intel systems: Enable Intel VT-d (Virtualization Technology for Directed I/O).
AMD systems: Enable AMD-Vi (AMD IOMMU).
General: Make sure Above 4G Decoding and SR-IOV (if supported by the motherboard) are enabled.

After saving these settings in the motherboard, you need to instruct the Proxmox Linux kernel to initialize IOMMU via the bootloader (GRUB or systemd-boot). Edit the configuration file depending on the boot system used. For systems with GRUB, edit /etc/default/grub and add the correct parameters to the kernel lines. A common obstacle here is that certain older BIOS versions hide options, which may require a firmware update before the IOMMU functionality responds flawlessly to the kernel parameters.

Step 2: Configuring Kernel Parameters and VFIO Modules

To prevent the Proxmox host itself from claiming the graphics card during boot, we need to isolate the GPU and its associated audio controller using VFIO drivers. This forces the host to ignore the standard open-source or proprietary drivers for these specific PCIe addresses.

Open the terminal of your Proxmox host and add the required kernel modules to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Then identify the Vendor ID and Device ID of your graphics card using the command lspci -nnk. Note the IDs of both the GPU and the built-in HDMI/DisplayPort audio controller, as these are often linked in the same IOMMU group. Create a configuration file in /etc/modprobe.d/vfio.conf and specify the IDs:

options vfio-pci ids=10de:2820,10de:22af

Don't forget to refresh the initramfs after these changes with the command update-initramfs -u -k all and restart the server. A common weak point at this stage is that users forget to include the audio controller in the VFIO configuration, which results in IOMMU conflicts and VM boot errors.

Step 3: Checking IOMMU Groups and Verifying Isolation

Before configuring a virtual machine, it's crucial to verify whether the GPU is in an isolated IOMMU group. If the GPU shares the same IOMMU group with essential motherboard components or the primary network card, you cannot safely pass through the GPU without jeopardizing the host's stability.

You can run a bash script to inspect the IOMMU groups:

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done

If this output shows that your graphics card shares a group with other indispensable devices, you should look for a different PCIe slot option in the BIOS. Moving the graphics card to the top PCIe x16 slot often resolves isolation issues directly. For detailed steps on setting up a clean Linux environment, you can consult the guide on running local LLMs on Linux.

Step 4: Creating and Setting Up the Virtual Machine

Now that the host side is ready, you can create the virtual machine within Proxmox VE. For running heavy language models, a Linux-based guest environment (such as Ubuntu Server or Debian) is preferable due to efficient memory allocation and broad support for AI frameworks.

When setting up the VM's hardware properties, pay attention to the following conditions:

OS Type: Set this to the correct Linux distribution.
BIOS: Choose OVMF (UEFI) instead of SeaBIOS, because modern GPUs require UEFI for initialization.
Machine: Preferably use the q35 chipset, as it supports modern PCIe topologies better than the outdated i440fx standard.
CPU Cores: Assign a realistic number of vCPUs and set CPU emulation to host to pass through instruction sets such as AVX-512 directly.

Also make sure you reserve enough virtual RAM, and when creating the hard disk, check the option for Discard and SSD emulation to minimize I/O bottlenecks.

Step 5: Adding a PCIe Device to the Proxmox VM

After the base VM has been created, you can physically attach the graphics hardware via the Proxmox web interface or via the configuration files in /etc/pve/qemu-server/[VM-ID].conf.

In the web interface, go to the relevant VM, select Hardware and click Add -> PCI Device. Select your graphics card here. Check the following options for maximum compatibility:

All Functions: Make sure both the GPU and its associated audio chip are passed through together.
Primary GPU: Check this if this is the VM's only or primary GPU (required for consoles and direct output, though most LLM servers run headless).
PCI-Express: Enable this checkbox so the VM correctly recognizes the PCIe generation.

If you want to set this manually in the configuration file, the line typically looks like this:

hostpci0: 0000:01:00,pcie=1,x-vga=1

A weak point of this configuration is that certain consumer GPUs (particularly specific NVIDIA GeForce cards) can generate an 'Error 43' under Windows VMs, or exhibit complex behavior when the host is restarted. Under Linux VMs, this issue occurs considerably less often, provided the hypervisor signatures are correctly hidden in the VM configuration.

Step 6: Installing Drivers and Verifying Operation in the VM

Start the virtual machine and log in via SSH or the console. Now that the hardware is physically visible to the guest operating system, the correct drivers and runtime libraries need to be installed to drive the AI models.

For NVIDIA hardware, this means installing the official drivers and the CUDA toolkit. Perform the following steps on an Ubuntu guest:

sudo apt update && sudo apt upgrade -y
sudo apt install build-essential nvidia-driver-550 nvidia-cuda-toolkit -y

After restarting the virtual machine, you can verify whether the GPU is correctly recognized by the kernel by running the command nvidia-smi . This should give an overview of your graphics card, the available VRAM, and the active driver version. If this command fails or indicates that no NVIDIA devices were found, the VFIO binding on the host was not passed through correctly, or a conflict with open-source drivers (nouveau) is blocking initialization.

Step 7: Monitoring Power Consumption and Operational Costs

Deploying a dedicated GPU within a virtual environment brings continuous operational costs. Because the graphics card in a passthrough setup remains directly under the host's physical power, power consumption keeps running in the background, even when no active inference tasks are being performed.

It's therefore necessary to closely monitor energy usage to prevent an unexpected rise in the energy bill. A comprehensive analysis of power costs and practical measurement methods can be found in the guide on what the power consumption of local AI costs. For continuous server use, expect a significant baseline wattage just for maintaining the PCIe connection and the cooling system.