An unfortunate bug exists for the AMD Threadripper family of GPUs which causes VGA Passthrough not to work properly. Fortunately some very clever people have implemented a workaround to allow proper VGA passthrough until a proper Linux Kernel patch can be accepted and implemented. See here for the whole story.
Right now my Thrdearipper 1950x successfully has GPU passthrough thanks to HyenaCheeseHeads “java hack” applet. I went this route because I really didn’t want to try and recompile my ProxMox kernel to get passthrough to work. Per the description “It is a small program that runs as any user with read/write access to sysfs (this small guide assumes “root”). The program monitors any PCIe device that is connected to VFIO-PCI when the program starts, if the device disconnects due to the issues described in this post then the program tries to re-connect the device by rewriting the bridge configuration.” Instructions taken from the above Reddit post.
- Go to https://pastebin.com/iYg3Dngs and hit “Download” (the MD5 sum is supposed to be 91914b021b890d778f4055bcc5f41002)
- Rename the downloaded file to “ZenBridgeBaconRecovery.java” and put it in a new folder somewhere
- Go to the folder in a terminal and type “javac ZenBridgeBaconRecovery.java”, this should take a short while and then complete with no errors. You may need to install the Java 8 JDK to get the javac command (use your distribution’s software manager)
- In the same folder type “sudo java ZenBridgeBaconRecovery”
- Make sure that the PCIe device that you intend to passthru is listed as monitored with a bridge
- Now start your VM
In my case (Debian Stretch, ProxMox) I needed to install openjdk-8-jdk-headless
sudo apt install openjdk-8-jdk-headless javac ZenBridgeBaconRecovery.java
Next I have a little script on startup to spawn this as root in a detached tmux session, so I don’t have to remember to run it (If you try to start your VM before running this, it will hose passthrough on your system until you reboot it.) Be sure to change the script to point to wherever you compiled ZenBridgeBaconRecovery
#!/bin/bash cd /home/nicholas #change me to suit your needs sudo java ZenBridgeBaconRecovery
And here is the command I use to run on startup:
tmux new -d '/home/nicholas/passthrough.sh'
Again, be sure to modify the above to point to the path of wherever you saved the above script.
So far this works pretty well for me. I hate having to run a java process as sudo, but it’s better than recompiling my kernel.
Update 6/27/2018: I’ve created a systemd service script for the ZenBaconRecovery file to run at boot. Here is my file, placed in
/etc/systemd/system/zenbridge.service: (change your working directory to match the zenbridgebaconrecovery java file location. Don’t forget to do systemctl daemon-reload.)
[Unit] Description=Zen Bridge Bacon Recovery After=network.target [Service] Type=simple User=root WorkingDirectory=/home/nicholas ExecStart=/usr/bin/java ZenBridgeBaconRecovery Restart=on-failure # or always, on-abort, etc [Install] WantedBy=multi-user.target ~
Update 8/18/2018 Finally solved for everyone!
Per an update on the reddit thread motherboard manufactures have finally put out BIOS updates that resolve the PCI passthrough problems. I updated my X399 Tachi to the latest version of its UEFI BIOS (3.20) and indeed PCI passthrough worked without any more wonky workarounds!
Hi Nicholas,
What’s the latest on this setup, do you still use it regularly? We troubleshot for a good month back in Jan/Feb. MSI X399 + 1900x + GTX 770 or GTX 1030.
The Java hack had me working for a week on my passthrough VM. Eventually, it fell apart, starting with graphics/performance issues. We experimented with different drivers… The stock NVidia drivers make the ZenBridgeBaconRecovery.java console spam dozens of errors every second. Nouveau had me working for most of the week, but it ultimately became unbootable. We’ve tried a multitude of Guest OSes, different video cards, different PCI slot positions, different IOMMU grouping flags in the GRUB config, different drivers, and finally, we went down the route of patching and compiling our own kernel.
It was nice to have 16 cores to run make with… but, even with the kernel patch, I had similar issues. I’m running 4.15.12 (with tr.patch) now in ProxMox, and the guest OS works with a GTX 770 passed through without the Java hack, but dies and becomes unbootable similar to the last attempt.
I currently use my threadripper daily with the java passthrough method and haven’t had any of the issues you’ve described (so far)
Asrock X399 Taichi
Threadripper 1950x
GTX 1070
Radeon HD 7450
In reading your post it sounds like you only have one graphics card, is that correct? I haven’t tried that setup. My system has two video cards – one for Proxmox, and the other for the VM. I’ve configured Proxmox to blacklist both nouveau and nvidia drivers as well as add the card’s PCI ID to vfio.conf so the vfio driver claims that card on bootup (see details here.)
I currently have the java hack running as a systemd init script on bootup. It works pretty well.
Hi!
I’m thinking on upgrade my system to virtualized workstation with 3 kvm’s (1gpu each).
I was thinking on get a threadripper, but i’m not pretty sure if this could be accomplished (proxmox debian). Can you give me some advise for this configuration?
Will be 2 workstations (kvms), and the third will be mce for tv.
I’m waiting from january to find good information to avoid spent 2.5k USD and later realize that it will not work, and this problems with passthrough is what is breaking me to make this config.
Some advise? thanks!
My Threadripper machine has two graphics cards currently and it works wonderfully. For GPU passthrough to work properly you will need to make sure you’re running the latest BIOS for your motherboard (mine shipped with a buggy BIOS that broke PCI Passthrough completely, but the latest update fixed it.) The way I have it set up I have one VM with a GPU passed through and the other GPU is native to proxmox. I ran the ‘tasksel’ command to install a graphical interface for proxmox and use it just like any other Debian Linux machine.
In your case it might be easier to pass through two video cards to VMs and have the third video card native to Proxmox. If you really want three VMs, each with their own GPU, then you’ll need a fourth GPU for the host system if you ever want to interact with it graphically. Threadrippers are a good choice for this due to the number of PCI lanes they have. You’ll want to make sure you get a motherboard with enough pci express slots.