An unfortunate bug exists for the AMD Threadripper family of GPUs which causes VGA Passthrough not to work properly. Fortunately some very clever people have implemented a workaround to allow proper VGA passthrough until a proper Linux Kernel patch can be accepted and implemented. See here for the whole story.
Right now my Thrdearipper 1950x successfully has GPU passthrough thanks to HyenaCheeseHeads “java hack” applet. I went this route because I really didn’t want to try and recompile my ProxMox kernel to get passthrough to work. Per the description “It is a small program that runs as any user with read/write access to sysfs (this small guide assumes “root”). The program monitors any PCIe device that is connected to VFIO-PCI when the program starts, if the device disconnects due to the issues described in this post then the program tries to re-connect the device by rewriting the bridge configuration.” Instructions taken from the above Reddit post.
- Go to https://pastebin.com/iYg3Dngs and hit “Download” (the MD5 sum is supposed to be 91914b021b890d778f4055bcc5f41002)
- Rename the downloaded file to “ZenBridgeBaconRecovery.java” and put it in a new folder somewhere
- Go to the folder in a terminal and type “javac ZenBridgeBaconRecovery.java”, this should take a short while and then complete with no errors. You may need to install the Java 8 JDK to get the javac command (use your distribution’s software manager)
- In the same folder type “sudo java ZenBridgeBaconRecovery”
- Make sure that the PCIe device that you intend to passthru is listed as monitored with a bridge
- Now start your VM
In my case (Debian Stretch, ProxMox) I needed to install openjdk-8-jdk-headless
sudo apt install openjdk-8-jdk-headless
Next I have a little script on startup to spawn this as root in a detached tmux session, so I don’t have to remember to run it (If you try to start your VM before running this, it will hose passthrough on your system until you reboot it.) Be sure to change the script to point to wherever you compiled ZenBridgeBaconRecovery
cd /home/nicholas #change me to suit your needs
sudo java ZenBridgeBaconRecovery
And here is the command I use to run on startup:
tmux new -d '/home/nicholas/passthrough.sh'
Again, be sure to modify the above to point to the path of wherever you saved the above script.
So far this works pretty well for me. I hate having to run a java process as sudo, but it’s better than recompiling my kernel.
I started this blog four years ago to document my highly technical adventures – mainly so I could reproduce them later. One of my first articles dealt with GPU passthrough / virtualization. It was a complicated ordeal with Xen. Now that I’ve switched to KVM (ProxMox) I thought I’d give it another go. It’s still complicated but not nearly as much this time.
To get my Nvidia GTX 1070 GPU properly passed through to a Windows VM hosted by ProxMox 5 I simply followed this excellent guide written by sshaikh. I will summarize what I took from his guide to get my setup to work.
- Ensure VT-d is supported and enabled in the BIOS
- Enable IOMMU on the host
- append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
- Save your changes by running
- Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
- Save your changes by running
- Add the following lines to /etc/modules
- Determine the PCI address of your GPU
and look for your card. Mine was 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00
- Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
lspci -n -s 01:00
01:00.0 0300: 10de:1b81 (rev a1)
01:00.1 0403: 10de:10f0 (rev a1)
- Assign your GPU to vfio driver using the IDs obtained above. Example:
echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf
- Reboot the host
- Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet. Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
- Install Windows, including VirtIO drivers. Be sure to enable Remote desktop.
- Pass through the GPU.
- Modify /etc/pve/qemu-server/<vmid>.conf and add
hostpci0: <device address>,x-vga=on,pcie=1. Example
I received the dreaded code 43 error after installing CUDA drivers. The workaround was to add hidden=1 to the CPU option of the VM:
Blue screening when launching certain games
Heroes of the Storm and Starcraft II would consistently blue screen on me with the following error:
The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error
System Thread Exception Not Handled
The fix for this is the same – ignore_msrs=1
Give as many CPUs as the host (in my case 8) and then enable NUMA for the CPU. This appeared to make my GTX 1070 perform better in the VM – near native performance.