I recently had an issue with one of my Proxmox hosts where it would max out all swap and slow down to a crawl despite having plenty of physical memory free. After digging and tweaking, I found this post which directed to set the kernel swappiness setting to 0. More reading suggested I should set it to 1, which is what I did.
Append to /etc/sysctl.conf:
#Fix excessive swap usage vm.swappiness = 1
Apply settings with:
This did the trick for me.