Windows VM with GTX 1070 GPU passthrough in ProxMox 5

I started this blog four years ago to document my highly technical adventures – mainly so I could reproduce them later. One of my first articles dealt with GPU passthrough / virtualization. It was a complicated ordeal with Xen. Now that I’ve switched to KVM (ProxMox) I thought I’d give it another go. It’s still complicated but not nearly as much this time.

To get my Nvidia GTX 1070 GPU properly passed through to a Windows VM hosted by ProxMox 5 I simply followed this excellent guide written by sshaikh. I will summarize what I took from his guide to get my setup to work.

  1. Ensure VT-d is supported and enabled in the BIOS
  2. Enable IOMMU on the host
    1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
      intel_iommu=on
    2. Save your changes by running
      update-grub
  3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
    1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
    2. Save your changes by running
      update-initramfs -u
  4. Add the following lines to /etc/modules
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  5. Determine the PCI address of your GPU
    1. Run
      lspci -v

      and look for your card. Mine was 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

    2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  6. Assign your GPU to vfio driver using the IDs obtained above. Example:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf
  7. Reboot the host
  8. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet. Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
    bios: ovmf
    machine: q35
    cpu: host,hidden=1
    numa: 1
  9. Install Windows, including VirtIO drivers. Be sure to enable Remote desktop.
  10. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
      hostpci0: <device address>,x-vga=on,pcie=1. Example

      hostpci0: 01:00,x-vga=on,pcie=1
  11. Profit.

Troubleshooting

Code 43

I received the dreaded code 43 error after installing CUDA drivers. The workaround was to add hidden=1 to the CPU option of the VM:

cpu: host,hidden=1

Blue screening when launching certain games

Heroes of the Storm and Starcraft II would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

GPU optimization:

Give as many CPUs as the host (in my case 8) and then enable NUMA for the CPU. This appeared to make my GTX 1070 perform better in the VM – near native performance.

20 thoughts on “Windows VM with GTX 1070 GPU passthrough in ProxMox 5”

  1. Hi,

    Do you run into weird audio issues? Like the sound is slowed down. Haven’t been able to pinpoint the problem and the only way to fix is to restart the whole server..

    Running win 10 with a 970 pass through

    1. Yes I have!

      In my case, if the microphone was accessed by anything, all sound would disappear on my Windows VM. I could get it to come back by going into audio properties of my playback device, go to the Enhancements tab, and either check or uncheck “Disable all enhancements” and hit apply. It would magically bring sound back until the mic was accessed again.

      I’ve tracked it down to whatever setup of USB device passthrough ProxMox is using. I’ve actually abandoned ProxMox because of it. I’m working on publishing a followup article on how to successfully get everything working in Arch Linux. No sound issues with that setup so far.

    2. I finally figured out what my particular audio problem was. I had a Sabrent USB sound card passed through to the guest. The USB controller was set to USB3, but this is a USB2 device. If I changed the controller to be USB2 only, the problem went away. So if you’re passing USB through, make sure to go with the lowest common denominator as far as USB speeds go on your VM’s USB controller.

  2. Just want to leave a note for myself: if the whole system freeze when start passthrough enabled guest, check “cat /proc/iomem” to see if efifb or legacy frame buffer (depending on BIOS type) is enabled. Disable it in grub kernel command by “video=efifb:off”.

    See “BAR 3: can’t reserve [mem] error” in https://pve.proxmox.com/wiki/Pci_passthrough

  3. Unfortunately this doesn’t work for me. I’m using Proxmox 6, maybe something changed? I have a EVGA branded GTX 1050 Ti. I’m concerned because while the card shows up in Windows, it doesn’t display ANYTHING. Not even the Proxmox EFI boot splash.

  4. Hi,
    I followed these steps and everything seems ok, but after installing NVIDIA Driver, my vm keeps running into blue screen. My video card (1070) is recognized in windows, any idea how to solve the problem?
    Thanks.

    1. Hi, blue screens after installing drivers are usually caused by one of two things. 1: NVIDIA drivers itself detects a virtulized environment. The fix is to hide the fact that it’s a VM by using this config line in your vm config file: cpu: host,hidden=1

      2: Windows 10 specifically has an issue with blue screening in proxmox sometimes, the fix is to add “ignore_msrs=1” to the VM config file.

      Likely you will need to have both in place. Hope this helps.

  5. Love you, man…
    worked on 6.2-4 with z270 chipset (acs patched)
    Been haunted by esxi for so long after switching from a dell r710 to a desktop and just couldn’t get gpu passthrough working, until I decided to switch to proxmox and saw your helpful post (and more importantly, concise).

    Will soon upgrade my lab gears with a z390 board and probably get two gpus installed and see how it goes.

    Two notes from me: 1.machine q35 is essential, since default i440fx won’t work; 2. either `cpu: hidden=1` or `cpu: kvm=off` works for me

  6. Great article and follow ups.

    Just also watched a similar description for Windows by TechnoTim, March 26, ’20. He wasn’t clear if it was multiplayer gaming …

    Questions:

    1. Does this allow multiplayer gaming with the gpu machine as gaming server?

    How many players can it support with a PC server with, say, AMD RX580 gpu running Minecraft with 4GB RAM?

    Should Minecraft run on host machine or within the guest OS in Proxmox?

    2. Or can this gpu passthu only support one client machine ?

    3. Re:#1, if we use parsecgaming.com or Steam Remote Play Together like screen sharing, will the answer change?

    4. Your article says to do this to avoid Code 43 error ( i.e., hide VM install from Nvidia driver):

    cpu: host,hidden=1
    Do we need to do this for AMD and Intel cards as well?

    Does this mean anti-cheat is over ridden?

    1. The way it works is 1 GPU per VM (you can’t share a GPU to multiple VMs.) So if you wanted one host machine to power multiple games at once, you’d need multiple GPUs and a different VM for each GPU. The Code 43 error only applies to NVIDIA cards (because NVIDIA deliberately wrote this into their drivers) I’m not aware of needing to do it for other GPUs.
      I have no idea about anything minecraft or anti-cheat related.
      Hope this helps!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.