Category Archives: Networking

OpenWRT Wireguard client

My notes on how to configure an OpenWRT device to be a wireguard client (site to site VPN)

More or less follow the instructions from https://openwrt.org/docs/guide-user/services/vpn/wireguard/client

I commented out the IPv6 stuff as well as the pre-shared key. I also had already defined firewall rules so I skipped that section.

One note: make sure your WG_ADDR has the proper subnet mask (I made the mistake of making it a /32 when it needed to be a /24)


# Configuration parameters
WG_IF="wg0"
WG_SERV="remote_wireguard_server_address"
WG_PORT="remote_wireguard_port"
WG_ADDR="wireguard_subnet/wireguard_subnet_mask (/24 for example)"
#WG_ADDR6="fdf1:e8a1:8d3f:9::2/64"

# Generate keys
#umask go=
#wg genkey | tee wgserver.key | wg pubkey > wgserver.pub
#wg genkey | tee wgclient.key | wg pubkey > wgclient.pub
#wg genpsk > wgclient.psk
 
# Client private key
WG_KEY="$(cat wgclient.key)"
 
# Pre-shared key
#WG_PSK="$(cat wgclient.psk)"
 
# Server public key
WG_PUB="public_key_of_wireguard_server"

# Configure network
uci -q delete network.${WG_IF}
uci set network.${WG_IF}="interface"
uci set network.${WG_IF}.proto="wireguard"
uci set network.${WG_IF}.private_key="${WG_KEY}"
uci add_list network.${WG_IF}.addresses="${WG_ADDR}"
#uci add_list network.${WG_IF}.addresses="${WG_ADDR6}"
 
# Add VPN peers
uci -q delete network.wgserver
uci set network.wgserver="wireguard_${WG_IF}"
uci set network.wgserver.public_key="${WG_PUB}"
#uci set network.wgserver.preshared_key="${WG_PSK}"
uci set network.wgserver.endpoint_host="${WG_SERV}"
uci set network.wgserver.endpoint_port="${WG_PORT}"
uci set network.wgserver.route_allowed_ips="1"
uci set network.wgserver.persistent_keepalive="25"
uci add_list network.wgserver.allowed_ips="10.137.50.0/24"
#uci add_list network.wgserver.allowed_ips="::/0"
uci commit network
/etc/init.d/network restart

Wireguard on a USG Pro 4

I ran into some issues when trying to configure an OpenVPN tunnel between my Ubiquiti USG Pro 4 and a Debian VPS. I was very disappointed to discover that the version of OpenVPN on the USG only supports TLS 1.0! My issue was the Debian side rejecting that as insecure.

Thankfully, it was fairly painless to get Wireguard configured on the USG Pro 4. I was hesitant to do so at first because I knew every time my USG was updated I would lose the wireguard pacakge. Fortunately that can be resolved by configuring a post-install script. Thanks to ilar.in and calypte.cc and this github gist for the steps on how to do so.

curl -O https://raw.githubusercontent.com/britannic/install-edgeos-packages/master/install-pkgs
sudo install -o root -g root -m 0755 install-pkgs /config/scripts/post-config.d/install-pkgs
  • Add wireguard DEB package to persistent storage that the script will look for:
sudo mkdir -p /config/data/install-packages
cd /config/data/install-packages
curl -fLSs https://github.com/WireGuard/wireguard-vyatta-ubnt/releases/download/1.0.20200729-1/ugw4-v1-v1.0.20200729-v1.0.20200513.deb
  • Generate a public/private keypair for USG use
cd /config/auth
umask 077
wg genkey > wg_private.key
wg pubkey < wg_private.key > wg_public.key
  • Generate config.gateway.json config to use wireguard
    "interfaces": {
 ...
        "wireguard": {
            "wg0": {
                "address": "<IP_OF_USG_ON_WG_CLIENT_SUBNET>",
                "listen-port": "<WG_LISTEN_PORT>",
                "peer": {
                    "<ENDPOINT_CLIENT_PUBLIC_KEY>": {
                        "allowed-ips": "0.0.0.0/0",
                        "endpoint":  "<ENDPOINT_CLIENT_ADDRESS>:<ENDPOINT_CLIENT_PORT>
"
                    }
                },
                "private-key": "/config/auth/priv.key",
                "route-allowed-ips": false
            },
            "wg1": {
                "address": "<IP_OF_USG_ON_WG_SERVER_SUBNET",
                "firewall": {
                    "in": {
                        "name": "LAN_IN"
                    },
                    "local": {
                        "name": "LAN_LOCAL"
                    },
                    "out": {
                        "name": "LAN_OUT"
                    }
                },
                "listen-port": "<USG_WG_SERVER_LISTEN_PORT>",
                "mtu": "1352",
                "peer": {
                    "<PUBLIC_KEY_OF_WG_CONNECTING_CLIENT": {
                        "allowed-ips": "<SUBNETS_ON_REMOTE_HOST>"
                    }
                },
                "private-key": "/config/auth/wg-server.priv",
                "route-allowed-ips": true
            }
        }
    },

I have two different wireguard interfaces configured – wg0 to be a client to another server, and wg1 to be a server accepting other clients (site-to-site VPN.)

If you want to have multiple peers defined on a single wireguard interface, encapsulate the peers with brackets like so:

"peer": [{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.2/32"
							],
							"persistent-keepalive": 60
						}
					},
					{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.3/32"
							],
							"persistent-keepalive": 60
						}
					}
				],

Test configuration first

Before committing your config.gateway.json code, test it line by line by SSHing into the USG-Pro 4 and entering config mode. Then type out your JSON lines one at a time, with each key being a new argument separated by a space. The first section above would look like this:

configure
edit interfaces
set wireguard wg0 address WIREGUARD_ADDRESS
set wireguard wg0 listen-port WG_LISTEN_PORT
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY allowed-ips 0.0.0.0/0
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY endpoint ENDPOINT_ADDRESS:ENDPOINT_PORT
set wireguard wg0 private-key /config/auth/priv.key
set wireguard wg0 route-allowed-ips false
commit

If the commit works without error, you can then drop out of the configure section and look at your wireguard config:

exit
sudo wg show

If all looks well, then copy your config.gateway.json to your controller and trigger a reprovision.

Verify after provisioning: 

sudo netstat -npl | grep <WIREGUARD_PORT> | grep udp

Troubleshooting

USG not connecting to changed endpoint address

If you change the address of the wireguard endpoint, USG pro will not connect to that new address. You have to delete and re-create the interface (thanks to https://github.com/Lochnair/vyatta-wireguard/issues/72#issuecomment-423840448 for the information)

Fix this by deleting the wireguard interface

admin@Firewall:~$ configure
[edit]
admin@Firewall# delete interfaces wireguard
[edit]
admin@Firewall# commit


Then reprovision by making a small change, force provision, then change back, and force another provision (annoying) or alternatively reboot firewall.

Wireguard shows established but ping doesn’t work

Example error:

From 10.99.13.1 icmp_seq=5 Destination Host Unreachable
ping: sendmsg: Required key not available

To figure out what’s going on, enable logging to kernel buffer (dmesg) Thanks to procustodibus.com for the info.

echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

With debug on, ping again and check kernel messages (dmesg)

[Tue Dec 21 22:16:12 2021] wireguard: wg0: No peer has allowed IPs matching 10.99.13.2

This showed I didn’t have my access control properly configured. Modify /etc/wireguard/wg0.conf on your client config and make sure your AllowedIPs are properly letting traffic through.

AllowedIPs = 10.99.13.0/24

USG not allowing connections

Clients unable to connect to USG despite having a good config. Double check your firewall rules. I had neglected to create a WAN LOCAL rule allowing UDP packets on my wireguard port. Once that was configured, handshakes completed successfully.

Firewalls can’t ping each other

I had an issue where the firewalls would pass traffic through, but they couldn’t ping each other. The solution was to add the VPN subnet you created to allowed-ips on both sides of the connection.

Site to site Wireguard VPN between OPNSense & Debian Linux server

I have a Debian linode box acting as a wireguard server. I wanted to join my opnsense firewall to it to allow devices behind it to access the box through the wireguard tunnel. It was not as straightforward as I had hoped, but thankfully I got it all working.

OPNSense side

Documentation link

Install wireguard via GUI

Install the os-wireguard package. Manually drop to the CLI and install the wireguard package as well:
sudo pkg install wireguard

Configure Local instance

  • Name and listen port can be random. Tunnel address is the subnet you wish to expose to the other end (the subnet you wish to have access to the tunnel.)
  • Leave everything else blank and hit save
  • Edit your new connection and copy the Public key, this will need to be sent to the Debian server

Configure Endpoint

  • Name: hostname of Debian server
  • Public Key: Public key of server (can be obtained by running wg show on the server)
  • Shared Secret: blank (unless you’ve configured it on the server)
  • Allowed IPs: IPs or subnets on the Debian server you wish to expose to the client side (the OPNSense box)
  • Endpoint address: DNS name of Debian server
  • Endpoint port: Port Debian wireguard instance is listening on

Enable the VPN

General tab / Enable wireguard checkbox and hit apply.

Debian side

Take down the tunnel

sudo wg-quick down wg0

Edit wireguard config to add peer

sudo vim /etc/wireguard/wg0.conf

[Peer]
PublicKey = <PUBLIC_KEY_YOU_COPIED_IN_LOCAL_INSTANCE_STEP>
AllowedIPs = <IPs or Subnets behind the OPNSense side you wish to be exposed to the Debian side> 

Restart wireguard

sudo wg-quick up wg0

Check connections

Example wg show output below with dummy IPs:

sudo wg show
interface: wg0
  public key: f+/J4JO0aL6kwOaudAvZVa1H2mDzR8Nh3Vfeqq+anF8=
  private key: (hidden)
  listening port: 12345

peer: TuUW7diXcWlaV97z3cQ1/92Zal2Pm9Qz/W2OMN+v20g=
  endpoint: 10.0.0.1:54137
  allowed ips: 10.0.0.2/32
  latest handshake: 17 seconds ago
  transfer: 5.14 KiB received, 3.81 KiB sent

peer: CZuC/+wxvzj9+TiGeyZtcT/lMGZnXsfSs/h5Jtw2VSE=
  endpoint: 8.8.8.8:12345
  allowed ips: 192.168.1.1/32
  latest handshake: 7 minutes, 8 seconds ago
  transfer: 5.89 MiB received, 952.20 MiB sent

The endpoint: line gets populated when a successful VPN connection is made. If it’s missing, the tunnel was not established.

Troubleshooting

OPNSense box

Nothing happens after saving information and enabling tunnel

Make sure latest wireguard package is installed

sudo pkg install wireguard

Get more log output by opening a shell on your OPNSense box and running

sudo /usr/local/etc/rc.d/wireguard start

In my case I was getting this interesting message

[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Invalid argument). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                                                                                                                   │
│   Running wireguard-go is not required because this                                                     │
│   kernel has first class support for WireGuard. For                                                          │
│   information on installing the kernel module,                                                                 │
│   please visit                                                                                                                             │
│         https://www.wireguard.com/install/                                                                           │
└──────────────────────────────────────────────────────┘

I fixed this problem by manually installing wireguard with the pkg install command above.

Debian box

Wireguard config not saving

make sure to stop the tunnel first, otherwise your changes get overwritten by the daemon.

sudo wg-quick down wg0
<make changes>
sudo wg-quick up wg0

piKVM pushover startup script

I’ve had an issue where I wasn’t sure if my dynamic DNS provider registered properly. I then realized that I have a piKVM attached to one of my servers that boots on powerup, even if the server does not. I could utilize this piKVM to help me out.

Thanks to inspiration from Chris Dzombak I was able to whip up a little script that runs on startup. This script waits 5 minutes to allow for my firewall and modem to boot up, then sends a pushover notification to let me know the piKVM is online and what its external IP address is.

To get it working on the piKVM I had to enter into RW mode, write and save the script, add execute permissions to the script, then configure a systemd service to run the script at startup.

Here is the script, saved under /root/boot-pushover.sh

#!/usr/bin/env bash
set -eu

#Wait 5 minutes to allow router bootup
sleep 300

TOKEN="PUSHOVER_APPLICATION_TOKEN"
USER="PUSHOVER_USER_TOKEN"
EXTERNAL_IP="$(curl ifconfig.me)"
MESSAGE="$(hostname) is online. External IP: $EXTERNAL_IP"

#Send pushover command to alert it's up and send its external IP
curl -s \
  --form-string "token=$TOKEN" \
  --form-string "user=$USER" \
  --form-string "message=$MESSAGE" \
  https://api.pushover.net/1/messages.json

Set executable: chmod +x /root/boot-pushover.sh

Here is the systemd service, saved under /etc/systemd/system/boot-pushover-notification.service

[Service]
Type=oneshot
ExecStart=/root/boot-pushover.sh
RemainAfterExit=yes
User=root
Group=root
RestartSec=15
Restart=on-failure

[Unit]
Wants=network.target
After=network.target nss-lookup.target

[Install]
WantedBy=multi-user.target

Reload daemons & enable startup:

systemctl daemon-reload
systemctl enable boot-pushover-notification.service

Test by exiting rw mode and rebooting the piKVM:

ro
reboot

It works really well!

Fix no network after Proxmox 7 upgrade

I upgraded my proxmox server to version 7 and was dismayed to find it had no network connections after a reboot. After much digging I was finally able to find this post which mentioned:

After installing ifupdown2 everything works fine.

Sure enough, ifupdown2 was not installed anymore, and I had configured my networks with it. I had to manually assign an IP address to my node long enough to issue the command
apt install ifupdown2

Once I rebooted, everything came up like it should. Lesson learned: if you use ifupdown2, you must make sure it’s there before you reboot your server!

Reset root password on OPNSense with ZFS root

A lot of the guides for resetting the root password on an OPNSense box assume a UFS root partition. The password recovery steps do not work if you installed OPNSense with a ZFS root partition. If you try to follow the steps you get a lovely error about “unrecognized filesystem”

The process for ZFS (thanks to this article) is instead to run the following commands:

zfs set readonly=off zroot
zfs mount -a

Once that is done, you can proceed with the rest of the steps. Password recovery steps in full:

  1. Press the number 2 immediately on boot to go into single user mode
  2. Press enter when prompted for shell
  3. Make ZFS read/write:
    1. zfs set readonly=off zroot
    2. zfs mount -a
  4. Reset password
    1. opnsense-shell password
  5. Reboot

Connect Ubiquiti l2tp vpn with NetworkManager in Arch

I’ve recently moved and needed to connect to my (still existing) home network from my desktop. I’ve never had to VPN from my desktop before, so here my notes for getting it working.

Configuration

  1. Install necessary lt2p, pptp, and libreswan packages (I’m using yay as my package manager)
    yay -Sy community/networkmanager-l2tp community/networkmanager-pptp aur/networkmanager-libreswan aur/libreswan
  2. Configure VPN in GNOME settings (close settings window first if it was already open)
    1. Add VPN / Layer 2 Tunneling Protocol (L2TP)
    2. Gateway: IP/DNS of VPN
    3. User Authentication: Type: password
    4. IPsec Settings: Type: Pre-shared Key (PSK)
    5. PPP settings: Only check MSCHAPv2, check everything else. MPPE Security: 128-bit (most secure)

Troubleshooting

If something isn’t working the popup is not very descriptive. Network manager logs are stored in journald, so the best way to troubleshoot is to follow the logs: (-f for follow, -u for unit name)

sudo journalctl -f -u NetworkManager

In my case following the networkmanager logs I could see I didn’t have libreswan fully installed, and installing the libreswan package fixed it.

send test syslog messages with nc

I needed to send some test packets over UDP to make sure connectivity was working. I found this site which outlined how to do it really well

nc -u <IP/hostname> <port>

Then on the next line you can send test messages, then hit CTRL+D when done. In my case I wanted to test sending syslog data, so I did nc -u <hostname> 514, then wrote test messages. the -u specifies UDP and 514 is the syslog port. I was then able to confirm on the other end the message was received. Handy.

KVM with vga passthrough in arch linux

I’ve once again switched from Proxmox to Arch Linux for my desktop machine. Both use KVM so it’s really just a matter of using the different VM manager syntax (virt-manager vs qm.) I used my notes from my previous stint with Arch, my article on GPU Passthrough in Proxmox as well as a thorough reading of the Arch wiki’s PCI Passthrough article.

Enable IOMMU

Configure GRUB to load the necessary iommu modules at boot. Append amd_iommu=on iommu=pt to the end of GRUB_CMDLINE_LINUX_DEFAULT (change accordingly if you have Intel instead of AMD)

sudo vim /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 amd_iommu=on iommu=pt"

Run update-grub

sudo update-grub

Reserve GPU for VFIO

Reserve the GPU you wish to pass through to a VM for use with the vfio kernel driver (so the host OS doesn’t interfere with it)

  1. Determine the PCI address of your GPU
    1. Run lspci -v and look for your card. Mine was 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00
    2. Run lspci -n -s <PCI address from above> to obtain vendor IDs.
      Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  2. Assign your GPU to vfio driver using the IDs obtained above.
    Example using above IDs:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" >> /etc/modprobe.d/vfio.conf

Reboot the host to put the kernel / drivers into effect.

Configure virt-manager

Install virt-manager, dnsmasq & libvirtd:

pacman -Sy libvirtd virt-manager dnsmasq
sudo systemctl enable libvirtd
sudo systemctl start libvirtd

Configure Networking

Assuming you’re using network manager for your connections, create a bridge (thanks to ciberciti.biz & the arch wiki for information on how to do so.) Replace interface names with ones corresponding to your machine:

sudo nmcli connection add type bridge ifname br0 stp no
sudo nmcli connection add type bridge-slave ifname enp4s0 master br0 
sudo nmcli connection show
#Make note of the active connection name
sudo nmcli connection down "Wired connection 2" #from above
sudo nmcli connection up bridge-br0

Create a second bridge bound to lo0 for host-only communication. Change IP as desired:

sudo nmcli connection add type bridge ifname br99 stp no ip4 192.168.2.1/24
sudo nmcli connection add type bridge-slave ifname lo master br99
sudo nmcli connection up bridge-br99

Configure VM

Initial configuration

When creating the passthrough VM, make sure chipset is Q35.

Set the CPU model to host-passthrough (type it in, there is no dropdown for it.)

When adding disks / other devices, set the device model to virtio

Add your GPU by going to Add Hardware and finding it under PCI Host Device.

Windows 10 specific tweaks

If your passthrough VM is going to be windows based, some tweaks are required to get the GPU to work properly within the VM.

Ignore MSRs (blue screen fix)

Later versions of Windows 10 instantly bluescreen with kmode_exception_not_handled unless you pass an option to ignore MSRs. Add the kvm ignore_msrs=1 option in /etc/modprobe.d/kvm.conf to do so. Optionally add the report_ignored_msrs=0 option to squelch massive amounts of kernel messages every time an MSR was ignored.

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf
#Optional - ignore kernel messages from ignored MSRs
echo "options kvm report_ignored_msrs=0" >> /etc/modprobe.d/kvm.conf

Reboot to make those changes take effect.

NVIDIA Code 43 workaround

Use the virsh edit command to make some tweaks to the VM configuration. We need to hide the fact that this is a VM otherwise the GPU drivers will not load and will throw Error 43. We need to add a vendor_id in the hyperv section, and create a kvm section enabling hidden state, which hides certain CPU flags that the drivers use to detect if they’re in a VM or not.

sudo virsh edit <VM_NAME>

<features>
	<hyperv>
		...
		<vendor_id state='on' value='1234567890ab'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>

Optimize CPU

Determine architecture

If you operate on a multi-core system such as my AMD Ryzen Threadripper the you will want to optimize your CPU core configuration in the VM per the CPU Pinning section in the Arch Wiki

Determine your CPU topology by running lscpu -e and lstopo The important things to look for are the CPU number and core number. On my box, it looks like this:

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ
0 0 0 0 0:0:0:0 yes 3400.0000 2200.0000
1 0 0 1 1:1:1:0 yes 3400.0000 2200.0000
2 0 0 2 2:2:2:0 yes 3400.0000 2200.0000
3 0 0 3 3:3:3:0 yes 3400.0000 2200.0000
4 0 0 4 4:4:4:1 yes 3400.0000 2200.0000
5 0 0 5 5:5:5:1 yes 3400.0000 2200.0000
6 0 0 6 6:6:6:1 yes 3400.0000 2200.0000
7 0 0 7 7:7:7:1 yes 3400.0000 2200.0000
8 0 0 8 8:8:8:2 yes 3400.0000 2200.0000
9 0 0 9 9:9:9:2 yes 3400.0000 2200.0000
10 0 0 10 10:10:10:2 yes 3400.0000 2200.0000
11 0 0 11 11:11:11:2 yes 3400.0000 2200.0000
12 0 0 12 12:12:12:3 yes 3400.0000 2200.0000
13 0 0 13 13:13:13:3 yes 3400.0000 2200.0000
14 0 0 14 14:14:14:3 yes 3400.0000 2200.0000
15 0 0 15 15:15:15:3 yes 3400.0000 2200.0000
16 0 0 0 0:0:0:0 yes 3400.0000 2200.0000
17 0 0 1 1:1:1:0 yes 3400.0000 2200.0000
18 0 0 2 2:2:2:0 yes 3400.0000 2200.0000
19 0 0 3 3:3:3:0 yes 3400.0000 2200.0000
20 0 0 4 4:4:4:1 yes 3400.0000 2200.0000
21 0 0 5 5:5:5:1 yes 3400.0000 2200.0000
22 0 0 6 6:6:6:1 yes 3400.0000 2200.0000
23 0 0 7 7:7:7:1 yes 3400.0000 2200.0000
24 0 0 8 8:8:8:2 yes 3400.0000 2200.0000
25 0 0 9 9:9:9:2 yes 3400.0000 2200.0000
26 0 0 10 10:10:10:2 yes 3400.0000 2200.0000
27 0 0 11 11:11:11:2 yes 3400.0000 2200.0000
28 0 0 12 12:12:12:3 yes 3400.0000 2200.0000
29 0 0 13 13:13:13:3 yes 3400.0000 2200.0000
30 0 0 14 14:14:14:3 yes 3400.0000 2200.0000
31 0 0 15 15:15:15:3 yes 3400.0000 2200.0000

From the above output I see my CPU core 0 is shared by CPUs 0 & 16, meaning CPU 0 and CPU 16 (as seen by the Linux kernel) are hyperthreaded to the same physical CPU core.

Especially for gaming, you want to keep all threads on the same CPU cores (for multithreading) and the same CPU die (on my threadripper, CPUs 0-7 reside on one physical die, and CPUs 8-15 reside on the other, within the same socket.)

In my case I want to dedicate one CPU die to my VM with its accompanying hyperthreads (CPUs 0-7 & hyperthreads 16-23) You can accomplish this using the virsh edit command and creating a cputune section (make sure you have a matching vcpu count for the number of cores you’re configuring.) Also edit CPU mode with the proper topology of 1 socket, 1 die, 8 cores with 2 threads. Lastly, configure memory to only be from the proper NUMA node the CPU cores your VM is using (Read here for more info.)

sudo virsh edit <VM_NAME>

<domain type='kvm'>
  ...
  <vcpu placement='static' cpuset='0-7,16-23'>16</vcpu> 
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='17'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='18'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='19'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='20'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='21'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='22'/>
    <vcpupin vcpu='14' cpuset='7'/>
    <vcpupin vcpu='15' cpuset='23'/>
    <emulatorpin cpuset='0-7','26-23'/>
  </cputune>
  ...
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <feature policy='require' name='topoext'/>
    <numa>
      <cell id='0' cpus='0-15' memory='16777216' unit='KiB'/>
    </numa>
  </cpu>
  ...
</domain>

Configure NUMA

Non-uniform memory access is essential for 1st and 2nd gen Ryzen chips. It turns out that by default my motherboard hid the real NUMA configuration from the operating system. Remedy this by changing the BIOS setting to set Memory Interleaving = Channel (for my ASRock X399 motherboard it’s in CBS / DF options.) See here: https://www.reddit.com/r/Amd/comments/6vrcq0/psa_threadripper_umanuma_setting_in_bios/

After changing BIOS setting, lstopo now shows proper configuration:

CPU frequency

Change CPU frequency setting to use performance mode:

sudo pacman -S cpupower
sudo cpupower frequency-set -g performance

Enable Hugepages

Append default_hugepagesz=1G hugepagesz=1G hugepages=16 to the kernel line in /etc/default/grub and re-run sudo grub-mkconfig -o /boot/grub/grub.cfg

Configure FIFO CPU scheduling

The Arch Wiki mentions to run qemu-system-x86_64 with taskset and chrt but doesn’t mention how to do so if you’re using virt-manager. Fortunately this reddit thread outlined how to accomplish it: libvirt hooks. Create the following script and place it in /etc/libvirt/hooks/qemu  , change the VM variable to match the name of your VM, mark that new file as executable (chmod +x /etc/libvirt/hooks/qemu ) and restart libvirtd

#!/bin/bash
#Hook to change VM to FIFO scheduling to decrease latency
#Place this file in /etc/libvirt/hooks/qemu and mark it executable

#Change the VM variable to match the name of your VM
VM="win10"

if [ "$1" == "$VM" ] && [ "$2" == "started" ]; then
  if pid=$(pidof qemu-system-x86_64); then
     chrt -f -p 1 $pid
    echo $(date) changing CPU scheduling to FIFO for VM $1 pid $pid >> /var/log/libvirthook.log
  else
    echo $(date) Unable to acquire PID of $1 >> /var/log/libvirthook.log
  fi
fi
#Additional debug
#echo $(date) libvirt hook arg1=$1 arg2=$2 arg3=$3 arg4=$4 pid=$pid >> /var/log/libvirthook.log 

Isolate CPUs

Update 7/28/20: I no longer do this in favor of the qemu hook script above, which prioritizes to p1 the qemu process for the cores it needs. I’m leaving this section here for historical/additional tweaking purposes.

Update 6/28/20: Additional tuning since I was having some stuttering and framerate issues. Also read here about the emulatorpin option

Dedicate CPUs to the VM (host will not use them) – append isolcups, nohz_full & rcu_nocbs kernel parameters into /etc/default/grub

...
GRUB_CMDLINE_LINUX_DEFAULT=... isolcpus=0-7,16-23 nohz_full=0-7,16-23 rcu_nocbs=0-7,16-23
...

Update grub:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot, then check if it worked:

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/arch-root rw loglevel=3 amd_iommu=on iommu=pt isolcpus=0-7,16-23 nohz_full=0-7,16-23 rcu_nocbs=0-7,16-23
taskset -cp 1
pid 1's current affinity list: 8-15,24-31

You can still tell programs to use the CPUs the VM has manually with the taskset command:

chrt -r 1 taskset -c <cores to use> <name of program/process>

Low Latency Audio

Upbate 7/8/2020: I found this article and this reddit thread (and this one) on how to use pulseaudio for your guest VM to get low latency guest VM audio piped to the host machine.

Update qemu config

edit /etc/libvirt/qemu.conf: uncomment the line #user = "root" and replace “root” with your username

Update pulseaudio config

Edit /etc/pulse/daemon.conf and uncomment the following lines (remove semicolon)

;default-sample-rate = 44100
;alternate-sample-rate = 48000

Note: Change VM audio settings to match 44100 sample rate

Edit /etc/pulse/default.pa and append auth-anonymous=1 to load-module module-native-protocol-unix

load-module module-native-protocol-unix auth-anonymous=1

The restart pulseaudio:

pulseaudio -k

Update VM XML

remove all audio devices from the virtual hardware details bar (left side in VM info view).

Edit XML via virsh edit <VM_NAME>

Make sure top line reads

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Add the following after </devices> (bottom of file)

<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=0x1b'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-micro,audiodev=hda'/>
    <qemu:arg value='-audiodev'/>
    <qemu:arg value='pa,id=hda,server=unix:/run/user/1000/pulse/native'/>
</qemu:commandline>

Replace /user/1000 with the UID of your user (output of id command)

Final Win10 XML tweaks for 1950x threadripper

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 ...
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='17'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='18'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='19'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='20'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='21'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='22'/>
    <vcpupin vcpu='14' cpuset='7'/>
    <vcpupin vcpu='15' cpuset='23'/>
    <emulatorpin cpuset='8-15,24-31'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  ...
  <features>
     ...
    <hyperv>
     ...
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    ...
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <feature policy='require' name='topoext'/>
    <numa>
      <cell id='0' cpus='0-15' memory='16777216' unit='KiB'/>
    </numa>
  </cpu>
   ...
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=0x1b'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-micro,audiodev=hda'/>
    <qemu:arg value='-audiodev'/>
    <qemu:arg value='pa,id=hda,server=unix:/run/user/1000/pulse/native'/>
  </qemu:commandline>
</domain>

Profit

I’m very pleased with my current setup. It works well!

UBUNTU 20.04 cloned VM same DHCP IP fix

I cloned an Ubuntu 20.04 VM and was frustrated to see both boxes kept getting the same DHCP IP address despite having different network MAC addresses. I finally found on this helpful post which states Ubuntu 20.04 uses systemd-networkd for DHCP leases which behaves differently than dhclient. As wickedchicken states,

systemd-networkd uses a different method to generate the DUID than dhclientdhclient by default uses the link-layer address while systemd-networkd uses the contents of /etc/machine-id. Since the VMs were cloned, they have the same machine-id and the DHCP server returns the same IP for both.

To fix, replace the contents of one or both of /etc/machine-id. This can be anything, but deleting the file and running systemd-machine-id-setup will create a random machine-id in the same way done on machine setup.

So my fix was to run the following on the cloned machine:

sudo rm /etc/machine-id
sudo systemd-machine-id-setup
sudo reboot

That did the trick!


For the systems that registered their hostnames under the wrong IPs, I had to take the following action for my Ubuntu 20.04 desktop as well as my Ubiquiti USG-Pro 4

Ubiquiti: Clear DHCP lease

clear dhcp lease ip <ip_address>

Ubuntu desktop: Flush DNS

sudo systemd-resolve --flush-caches