All posts by nicholas

arch install notes 2020-06

My install notes to get Arch Linux set up just the way I like it, June 2020 edition. Reference:

Change to dvorak layout:
loadkeys dvorak

Sync NTP time:
timedatectl set-ntp true

Configure disk:
#create separate efi partition, LVM root & swap
pvcreate <dev>
vgcreate arch <dev>
lvcreate -L+2G arch -n swap
lvcreate -l100%FREE -n root arch

Initialize swap:
mkswap /dev/arch/swap
swapon /dev/arch/swap

Format & Mount root:
mkfs.ext4 /dev/arch/root
mount /dev/arch/root /mnt

Create EFI partition
mkdosfs -F32 <partition 1>
mkdir /mnt/efi
mount <partition 1> /mnt/efi

Make mirrorlist use only xmission
sed -i 's/^Server/#Server/g;s/#Server\(.*xmission.*\)/Server\1/g' /etc/pacman.d/mirrorlist

Install base system plus extra packages:
pacstrap /mnt base linux linux-firmware lvm2 efibootmgr samba vim htop networkmanager inetutils man-db man-pages texinfo openssh grub

Generate fstab
genfstab -U /mnt >> /mnt/etc/fstab

Enter new environment chroot
arch-chroot /mnt

Set timezone
ln -sf /usr/share/zoneinfo/America/Boise /etc/localtime

Configure en_US locales
sed -i 's/^#en_US\(.*\)/en_US\1/g' /etc/locale.gen

Make dvorak layout permanent
echo "KEYMAP=dvorak" > /etc/vconsole.conf

Set hostname
echo "_HOSTNAME_" > /etc/hostname
echo " _HOSTNAME_._DOMAIN_ _HOSTNAME_" >> /etc/hosts

Enable lvm2 hook for initial ramdisk (boot)
sed -i 's/HOOKS=(.*\<block\>/& lvm2/' /etc/mkinitcpio.conf

Generate initial ramdisk
mkinitcpio -P

Set password for root user:

Install Grub (EFI)
grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg

Enable networking & SSH on bootup:
systemctl enable NetworkManager sshd

Configure NTP
yum -y install ntp
#modify /etc/ntp.conf for timeservers as desired
systemctl enable ntpd

Exit chroot & reboot

Zimbra expired ldap certificate fix

I started getting SSL errors with my Zimbra mail server despite having a valid SSL certificate everywhere I knew where to check. When I tried to use zmcontrol status I got this error:

Unable to start TLS: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed when connecting to ldap master.

Eventually I found this blog post explaining the problem – it’s with the LDAP component in Zimbra. You have to switch it from ldap to ldaps. Why did this change? I do not know.


sudo -u zimbra bash
zmlocalconfig -e ldap_master_url=ldaps://$ZIMBRA_HOSTNAME:636
zmlocalconfig -e ldap_url=ldaps://$ZIMBRA_HOSTNAME:636
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_port=636
zmcontrol stop
zmcontrol start

This did the trick. The errors went away.

Threadripper / Epyc processor core optimization

I had a pet project (folding@home) where I wanted to maximize computing power. I became frustrated with default CPU scheduling of my folding@home threads. Ideal performance would keep similar threads on the same CPU, but the threads were jumping all over the place, which was impacting performance.

Step one was to figure out which threads belonged to which physical cores. I found on this site that you can use cat to find out what your “sibling threads” are:

cat /sys/devices/system/cpu/cpu{0..15}/topology/thread_siblings_list

The above command is for my Threadripper & Epyc systems, which each have 16 cores hyperthreaded to 32 cores. Adjust the {0..15} number to match your number of cores (core 0 being the fist core.) This was my output:

cat /sys/devices/system/cpu/cpu{0..15}/topology/thread_siblings_list


Now that I know the sibling threads are offset by 16, I can use this information to optimize my folding@home VMs. I modified my CPU pinning script to take this into consideration. The script ensures that each VM is pinned to only use sibling threads (ensuring they all stay on the same physical CPU.)

This script should be used with caution. It pins processes to specific CPUs, which limits the kernel scheduler’s ability to move things around if needed. If configured badly this can cause the machine to lock up or VMs to be terminated.

I saw some impressive results spinning up four separate 8 core VMs and pinning them to sibling cores using this script. It almost doubled the rate at which I completed folding@home work units.

And now, the script:

#Properly assign CPU cores to their respective die for EPYC/Threadripper systems
#Based on how hyperthreads are done in these systems
#cat /sys/devices/system/cpu/cpu{0..15}/topology/thread_siblings_list

#The script takes two arguments - the ID of the Proxmox VM to modify, and the core to begin the VM on
#If running this against multiple VMs, make sure to increment this second number by half of the cores of the previous VM
#For example, if I have one 8 core VM and I run this script specifying 0 for the offset, if I spin up a second VM, the second argument would be 4
#this would ensure the second VM starts on core 4 (the 5th core) and assigns sibling cores to match

set -eo pipefail

#take First argument as which VMID to pin CPU cores to, the second argument is which core to start pinning to

#Determine offset for sibling threads
SIBLING_THREAD_OFFSET=$(cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list| sed 's/,/ /g' | awk '{print $2}')

#Function to determine number of CPU cores a VM has
cpu_tasks() {
	expect <<EOF | sed -n 's/^.* CPU .*thread_id=\(.*\)$/\1/p' | tr -d '\r' || true
spawn qm monitor $VMID
expect ">"
send "info cpus\r"
expect ">"

#Only act if VMID & OFFSET are set
if [[ -z $VMID  || -z $OFFSET ]]
	echo "Usage: <VMID> <OFFSET>"
	exit 1
	#Get PIDs of each CPU core for VM, count number of VM cores, and get even/odd PIDs for assignment
	VCPU_EVEN_THREADS=($(for EVEN_THREAD in "${VCPUS[@]}"; do echo $EVEN_THREAD; done | awk '!(NR%2)'))
	VCPU_ODD_THREADS=($(for ODD_THREAD in "${VCPUS[@]}"; do echo $ODD_THREAD; done | awk '(NR%2)'))

	if [[ $VCPU_COUNT -eq 0 ]]; then
		echo "* No VCPUS for VM$VMID"
		exit 1

	echo "* Detected ${#VCPUS[@]} assigned to VM$VMID..."
	echo "* Resetting cpu shield..."

	#Start at offset CPU number, assign odd numbered PIDs to their own CPU thread, then increment CPU core number
	#0-3 if offset is 0, 4-7 if offset is 4, etc
	for PID in "${VCPU_ODD_THREADS[@]}"
		echo "* Assigning ODD thread $ODD_CPU_INDEX to $PID..."
		taskset -pc "$ODD_CPU_INDEX" "$PID"

	#Start at offset + CPU count, assign even number PIDs to their own CPU thread, then increment CPU core number
	#16-19 if offset is 0,	20-23 if offset is 4, etc
	for PID in "${VCPU_EVEN_THREADS[@]}"
		echo "* Assigning EVEN thread $EVEN_CPU_INDEX to $PID..."
		taskset -pc "$EVEN_CPU_INDEX" "$PID"

UBUNTU 20.04 cloned VM same DHCP IP fix

I cloned an Ubuntu 20.04 VM and was frustrated to see both boxes kept getting the same DHCP IP address despite having different network MAC addresses. I finally found on this helpful post which states Ubuntu 20.04 uses systemd-networkd for DHCP leases which behaves differently than dhclient. As wickedchicken states,

systemd-networkd uses a different method to generate the DUID than dhclientdhclient by default uses the link-layer address while systemd-networkd uses the contents of /etc/machine-id. Since the VMs were cloned, they have the same machine-id and the DHCP server returns the same IP for both.

To fix, replace the contents of one or both of /etc/machine-id. This can be anything, but deleting the file and running systemd-machine-id-setup will create a random machine-id in the same way done on machine setup.

So my fix was to run the following on the cloned machine:

sudo rm /etc/machine-id
sudo systemd-machine-id-setup
sudo reboot

That did the trick!

For the systems that registered their hostnames under the wrong IPs, I had to take the following action for my Ubuntu 20.04 desktop as well as my Ubiquiti USG-Pro 4

Ubiquiti: Clear DHCP lease

clear dhcp lease ip <ip_address>

Ubuntu desktop: Flush DNS

sudo systemd-resolve --flush-caches

Folding@home opencl error fix

I decided to contribute my GPU on my Ubuntu-based system to the Folding@Home effort for COVID-19. I kept getting this error message for my NVIDIA GeForce GTX 1050 TI when I tried:

ERROR:WU00:FS00:Failed to start core: OpenCL device matching slot 0 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually

I had the nvidia opencl packages installed but apparently missed something. I finally found on the folding at home forum what I was missing – ocl-icd-opencl-dev

sudo apt install ocl-icd-opencl-dev

After running the above command and restarting the FAHClient service, the GPU started folding. For science!

EDIT 5/6/2020: After a re-install I had the issue where the GPU wouldn’t show up at all. It addition to ocl-icd-opencl-dev, it looks like you also need nvidia-cuda-dev.

sudo apt install ocl-icd-opencl-dev nvidia-cuda-dev

Sort by middle of a string

I had a list of items I wanted to sort in a non-standard way:

It’s a generalized list for publication but you get the idea. I wanted to sort by site name. Thanks to this post I found it’s relatively easy. You can tell the sort command to use a character as a tab delimiter (-t) and then specify which key “column” to sort by (-k)

In my case I sorted by site by specifying the dot character '.' as the delimiter, and the second “column” as the key '-k2'

The end result was this:

cat apps-by-site-unsorted.txt | sort -t. -k2


create podman services with podman-compose

Podman is a fork of Docker that Redhat is using. I really liked docker-compose functionality; fortunately there is a podman-compose project which is more or less the same thing.

I now have a setup where each podman container is controlled by a systemd service, set to run on startup, with version controlled podman-compose files.

First, I installed podman-compose:

sudo curl -o /usr/local/bin/podman-compose
chmod +x /usr/local/bin/podman-compose

I then created podman-compose files (syntax identical to docker-compose) for each container. Here is one example (jackett.yml)

version: "2"
    image: linuxserver/jackett
    container_name: jackett
      - PUID=1000
      - PGID=1000
      - TZ=America/Boise
      - /mnt/storage/Docker/Jackett/config:/config
      - /mnt/storage/Docker/Jackett/downloads:/downloads
      - 9117:9117
    restart: unless-stopped

I then created a corresponding systemd unit file for each container:



# Compose up
ExecStart=/usr/local/bin/podman-compose -f /home/nicholas/podman/jackett.yml up

# Compose down, remove containers and volumes
ExecStop=/usr/local/bin/podman-compose -f /home/nicholas/podman/jackett.yml down -v


I then do a systemctl daemon-reload, and enable the service for startup:

sudo systemctl daemon-reload
sudo systemctl enable jackett


Why not create a single podman-compose file for all my services, instead of creating individual services for each container? I wanted to be able to clearly see log output for each container with journalctl -f -u <service name.> If you lump all your services in a single compose file, the output from each container gets all jumbled into that single service log. Separating out each container into its own service was more clean.

git checkout only specific directory from repo

I have a git repo where I just wanted a specific folder, not the entire repo, cloned to one of my virtual machines. Git doesn’t handle this straightforwardly, but thanks to this article I found there is a roundabout way of doing it., by combining a git sparse checkout and a git shallow checkout.

Below are the commands to run (I ran these directly in my home directory.) Replace FOLDER with the folder from within the repository you wish to clone.

git init <repo> 
cd <repo>
git remote add origin <url to remote repo> 
git config core.sparsecheckout true 
echo "FOLDER/*" >> .git/info/sparse-checkout 
git pull --depth=1 origin master 

Success! Now this particular machine only has the folder within the repo I want, not the entire git repository.

Podman no internet in container fix

I’ve started experimenting with CentOS 8 & Podman (a fork of Docker.) I ran into an issue where one of my containers needed internet access, but could not connect. After some digging I found this site which explains why:

I had to configure the firewall on the podman host to allow for IP masquerade:

sudo firewall-cmd --zone=public --add-masquerade --permanent

After running the above command, my container had internet access!