Windows VM with GTX 1070 GPU passthrough in ProxMox 5

March 13, 2018 nicholas 20 Comments

I started this blog four years ago to document my highly technical adventures – mainly so I could reproduce them later. One of my first articles dealt with GPU passthrough / virtualization. It was a complicated ordeal with Xen. Now that I’ve switched to KVM (ProxMox) I thought I’d give it another go. It’s still complicated but not nearly as much this time.

To get my Nvidia GTX 1070 GPU properly passed through to a Windows VM hosted by ProxMox 5 I simply followed this excellent guide written by sshaikh. I will summarize what I took from his guide to get my setup to work.

Ensure VT-d is supported and enabled in the BIOS
Enable IOMMU on the host
1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
```
intel_iommu=on
```
2. Save your changes by running
```
update-grub
```

Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Save your changes by running
```
update-initramfs -u
```

Add the following lines to /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Determine the PCI address of your GPU
1. Run
```
lspci -v
```
  and look for your card. Mine was 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00
2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
```
lspci -n -s 01:00
01:00.0 0300: 10de:1b81 (rev a1)
01:00.1 0403: 10de:10f0 (rev a1)
```

Assign your GPU to vfio driver using the IDs obtained above. Example:

echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

Reboot the host
Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet. Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
```
bios: ovmf
machine: q35
cpu: host,hidden=1
numa: 1
```
Install Windows, including VirtIO drivers. Be sure to enable Remote desktop.
Pass through the GPU.
1. Modify /etc/pve/qemu-server/<vmid>.conf and add
  hostpci0: <device address>,x-vga=on,pcie=1. Example
```
hostpci0: 01:00,x-vga=on,pcie=1
```
Profit.

Troubleshooting

Code 43

I received the dreaded code 43 error after installing CUDA drivers. The workaround was to add hidden=1 to the CPU option of the VM:

cpu: host,hidden=1

Blue screening when launching certain games

Heroes of the Storm and Starcraft II would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

GPU optimization:

Give as many CPUs as the host (in my case 8) and then enable NUMA for the CPU. This appeared to make my GTX 1070 perform better in the VM – near native performance.

20 thoughts on “Windows VM with GTX 1070 GPU passthrough in ProxMox 5”

David says:

July 4, 2018 at 2:13 am

Hi,

Do you run into weird audio issues? Like the sound is slowed down. Haven’t been able to pinpoint the problem and the only way to fix is to restart the whole server..

Running win 10 with a 970 pass through

Reply
1. nicholas says:
  
  July 5, 2018 at 8:50 am
  
  Yes I have!
  
  In my case, if the microphone was accessed by anything, all sound would disappear on my Windows VM. I could get it to come back by going into audio properties of my playback device, go to the Enhancements tab, and either check or uncheck “Disable all enhancements” and hit apply. It would magically bring sound back until the mic was accessed again.
  
  I’ve tracked it down to whatever setup of USB device passthrough ProxMox is using. I’ve actually abandoned ProxMox because of it. I’m working on publishing a followup article on how to successfully get everything working in Arch Linux. No sound issues with that setup so far.
  
  Reply
  1. chris says:
    
    October 29, 2018 at 10:36 am
    
    you think you can share your experience with Arch Linux installation vm PCI pass through?
    
    Reply
    1. nicholas says:
      
      October 29, 2018 at 11:09 am
      
      I actually have switched away from Arch back to ProxMox. When I did do this in Arch, I mostly followed the ArchWiki entry for it: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
      
      I did make a rough draft of notes that I jotted down when I was doing this with Arch. Here is the link: https://techblog.jeppson.org/2018/10/gaming-vm-with-graphics-passthrough-in-arch-linux/
      
      Hope this helps, good luck!
      
      Reply
2. nicholas says:
  
  July 7, 2018 at 2:18 pm
  
  I finally figured out what my particular audio problem was. I had a Sabrent USB sound card passed through to the guest. The USB controller was set to USB3, but this is a USB2 device. If I changed the controller to be USB2 only, the problem went away. So if you’re passing USB through, make sure to go with the lowest common denominator as far as USB speeds go on your VM’s USB controller.
  
  Reply
Yuntian Zhang says:

July 7, 2018 at 3:07 pm

Just want to leave a note for myself: if the whole system freeze when start passthrough enabled guest, check “cat /proc/iomem” to see if efifb or legacy frame buffer (depending on BIOS type) is enabled. Disable it in grub kernel command by “video=efifb:off”.

See “BAR 3: can’t reserve [mem] error” in https://pve.proxmox.com/wiki/Pci_passthrough

Reply
1. Yuntian Zhang says:
  
  July 7, 2018 at 7:47 pm
  
  Also no balloon
  
  Reply
2. Brent says:
  
  January 27, 2020 at 7:26 pm
  
  More than just a note to yourself! Solved my problem as well – thanks!
  
  Reply
Pingback: Proxmox GPU Passthrough Guide | Tomas' Laboratory
Pingback: ProxMox – GPU Passthrough – Seabios PCI EXPRESS – Bujakiewicz Paweł
Pingback: Sabrent USB AU-MMSA microphone not working in Windows 10 - Technicus
Kavus Kazian says:

August 16, 2019 at 9:22 pm

Unfortunately this doesn’t work for me. I’m using Proxmox 6, maybe something changed? I have a EVGA branded GTX 1050 Ti. I’m concerned because while the card shows up in Windows, it doesn’t display ANYTHING. Not even the Proxmox EFI boot splash.

Reply
1. nicholas says:
  
  October 22, 2019 at 8:00 am
  
  Late reply, but I finally upgraded to Proxmox 6 and it is indeed the issue. This all broke once I upgraded. The fix is to set your machine type to q35-3.1 “qm set VM_ID -machine pc-q35-3.1”
  
  I outlined it here: https://techblog.jeppson.org/2019/10/proxmox-6-nvidia-gpu-passthrough-fix/
  
  Reply
Pingback: Primary VGA passthrough in ProxMox - Technicus
Zheng says:

October 19, 2019 at 3:10 am

Hi,
I followed these steps and everything seems ok, but after installing NVIDIA Driver, my vm keeps running into blue screen. My video card (1070) is recognized in windows, any idea how to solve the problem?
Thanks.

Reply
1. nicholas says:
  
  October 19, 2019 at 12:36 pm
  
  Hi, blue screens after installing drivers are usually caused by one of two things. 1: NVIDIA drivers itself detects a virtulized environment. The fix is to hide the fact that it’s a VM by using this config line in your vm config file: cpu: host,hidden=1
  
  2: Windows 10 specifically has an issue with blue screening in proxmox sometimes, the fix is to add “ignore_msrs=1” to the VM config file.
  
  Likely you will need to have both in place. Hope this helps.
  
  Reply
Frank Zhang says:

August 27, 2020 at 8:47 pm

Love you, man…
worked on 6.2-4 with z270 chipset (acs patched)
Been haunted by esxi for so long after switching from a dell r710 to a desktop and just couldn’t get gpu passthrough working, until I decided to switch to proxmox and saw your helpful post (and more importantly, concise).

Will soon upgrade my lab gears with a z390 board and probably get two gpus installed and see how it goes.

Two notes from me: 1.machine q35 is essential, since default i440fx won’t work; 2. either `cpu: hidden=1` or `cpu: kvm=off` works for me

Reply
1. nicholas says:
  
  August 27, 2020 at 9:01 pm
  
  I’m glad this was helpful! Happy labbing
  
  Reply
Shippy says:

September 16, 2020 at 8:21 am

Great article and follow ups.

Just also watched a similar description for Windows by TechnoTim, March 26, ’20. He wasn’t clear if it was multiplayer gaming …

Questions:

1. Does this allow multiplayer gaming with the gpu machine as gaming server?

How many players can it support with a PC server with, say, AMD RX580 gpu running Minecraft with 4GB RAM?

Should Minecraft run on host machine or within the guest OS in Proxmox?

2. Or can this gpu passthu only support one client machine ?

3. Re:#1, if we use parsecgaming.com or Steam Remote Play Together like screen sharing, will the answer change?

4. Your article says to do this to avoid Code 43 error ( i.e., hide VM install from Nvidia driver):

cpu: host,hidden=1
Do we need to do this for AMD and Intel cards as well?

Does this mean anti-cheat is over ridden?

Reply
1. nicholas says:
  
  September 16, 2020 at 12:48 pm
  
  The way it works is 1 GPU per VM (you can’t share a GPU to multiple VMs.) So if you wanted one host machine to power multiple games at once, you’d need multiple GPUs and a different VM for each GPU. The Code 43 error only applies to NVIDIA cards (because NVIDIA deliberately wrote this into their drivers) I’m not aware of needing to do it for other GPUs.
  I have no idea about anything minecraft or anti-cheat related.
  Hope this helps!
  
  Reply