Tag Archives: Networking

CLI, Virtualization

Unbind vfio driver from device in Proxmox

Leave a comment

I found myself with a Proxmox server that wouldn’t do anything with its network card. It took me a while to realize that at one point I had bound it to a VM. Even after removing it from the VM, the host wouldn’t do anything with it.

Discover which driver a device is using:

lspci -knn

In my case I found the culprit: the driver for the network card was still claimed by vfio-pci

08:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:0050]
Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

I finally found in this post how to tell the kernel to unbind from vfio-pci and bind to the network driver mlx4_core. Given the PCI bus location and device ID from the command, I was able to reclaim my network adapter to my host successfully:

echo -n "0000:08:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo -n "15b3 1003" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo -n "0000:08:00.0" > /sys/bus/pci/drivers/mlx4_core/bind

Success!

Networking, Virtualization

Fix no internet in KVM/QEMU VMs after installing docker

2 Comments

I ran into a frustrating issue where my KVM VMs would lose network connectivity if I installed docker on my Arch Linux system. After some digging I finally discovered the cause (thanks to anteru.net)

It turns out, docker adds a bunch of iptables rules by default which prevent communication. These will interfere with an already existing bridge, and suddenly your VMs will report no network.

There are two ways to fix this. I went with the route of telling docker to NOT mess with iptables on startup. Less secure, but my system is not directly connected to the internet. I created /etc/docker/daemon.json and added this to it:

{
    "iptables" : false
}

Then restarted my machine. This did the trick!

Networking, OS

ProxMox VMs reboot when switch is rebooted

Leave a comment

I came across an interesting situation where if I rebooted my Ubiquiti UniFi Switch 24 for a firmware upgrade, all my VM hosts would reboot themselves. It turned out to be due to my having enabled HA in ProxMox. The hosts would temporarily lose connectivity to each other and begin to fence themselves off from the cluster. This caused HA to kill the VMs on those hosts. Then once connectivity was restored everything would eventually come back up.

The proper way to fix this would be to have multiple paths for each host to talk to each other, so if one switch goes down the cluster is still able to communicate. In my case, where I only have one switch, the “poor man’s fix” was to simply disable HA altogether during the switch reboot, as outlined here. Then, once the switch is back up, re-enable HA.

On each node, stop the pve-ha-lrm service. Once it’s stopped on all hosts, stop the pve-ha-crm service. Then reboot your switch.

After the switch is back up, start pve-ha-lrm on each node first, then pve-ha-crm (if it doesn’t auto start itself) to re-enable HA.

CLI

Modify network buffers in Linux using sysctl

Leave a comment

After installing Netdata I kept getting alerts that packets were being dropped. After some research I read you can alleviate this by increasing your system’s networking buffer. You can accomplish this with sysctl.

If you run sysctl manually it will not survive a reboot. If you want persistence you simply add the sysctl command line to /etc/sysctl.conf. Thanks to Brennen Smith on serverfault for the info on my particular problem with UDP drops:

#as root:
echo "net.core.rmem_default=8388608" >> /etc/sysctl.conf
echo "net.core.rmem_max=16777216" >> /etc/sysctl.conf
sysctl -p

The above solved my udp problem but you can modify it to change any sysctl things you want to quickly configure and persist.