I started this blog four years ago to document my highly technical adventures – mainly so I could reproduce them later. One of my first articles dealt with GPU passthrough / virtualization. It was a complicated ordeal with Xen. Now that I’ve switched to KVM (ProxMox) I thought I’d give it another go. It’s still complicated but not nearly as much this time.

To get my Nvidia GTX 1070 GPU properly passed through to a Windows VM hosted by ProxMox 5 I simply followed this excellent guide written by sshaikh. I will summarize what I took from his guide to get my setup to work.

1. Ensure VT-d is supported and enabled in the BIOS
2. Enable IOMMU on the host
   1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub

      intel_iommu=on

   2. Save your changes by running

      update-grub

3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot

   1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

   2. Save your changes by running

      update-initramfs -u

4. Add the following lines to /etc/modules

   vfio
   vfio_iommu_type1
   vfio_pci
   vfio_virqfd

5. Determine the PCI address of your GPU
   1. Run

      lspci -v

      and look for your card. Mine was 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

   2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :

      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)

6. Assign your GPU to vfio driver using the IDs obtained above. Example:

   echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

7. Reboot the host
8. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet. Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.

   bios: ovmf
   machine: q35
   cpu: host,hidden=1
   numa: 1

9. Install Windows, including VirtIO drivers. Be sure to enable Remote desktop.
10. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
       hostpci0: <device address>,x-vga=on,pcie=1. Example

       hostpci0: 01:00,x-vga=on,pcie=1

11. Profit.

Troubleshooting

Code 43

I received the dreaded code 43 error after installing CUDA drivers. The workaround was to add hidden=1 to the CPU option of the VM:

cpu: host,hidden=1

Blue screening when launching certain games

Heroes of the Storm and Starcraft II would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

GPU optimization:

Give as many CPUs as the host (in my case 8) and then enable NUMA for the CPU. This appeared to make my GTX 1070 perform better in the VM – near native performance.