Category Archives: CLI

Free up RAM after Proxmox live migration

I ran into an issue where after migrating a bunch of VMs off of one of  my hosts, the remaining VMs on it refused to turn on. Every time I tried the command would hang for a while and eventually error out with this message

TASK ERROR: start failed: command '/usr/bin/kvm -id <truncated>... ' failed: got timeout

I suspected this might be due to RAM use and sure enough the usage was too high for a system that didn’t have any VMs running on it.  I found here that I could run a command to flush the cache:

echo 3 > /proc/sys/vm/drop_caches

That caused the RAM usage to go down but the symptom of the VM not starting remained. I then saw the KSM sharing still had some memory in it. I decided to restart the KSM sharing service:

sudo systemctl restart ksmtuned

After running that the VM started!

Supermicro fans constantly spinning to 100% fix

My fancy new Supermicro-powered AMD Epyc 7 series server is the bee’s knees. When I first set up I had an annoying problem though – the fans would spin to 100% and back down to normal speeds constantly. Logs kept spamming the same thing over and over:

SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: Assertion
EVENT SEVERITY:"information"
SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: De-assertion
EVENT SEVERITY:"information"

It was doing this for all 3 fans I had plugged in there. I finally came across this post which explained what the problem was. The fans I had installed defaulted to a low RPM mode, too low for the motherboard’s liking. The BMC would detect the low RPM and assume something was wrong and bring all fans to 100% in order to rescue the system. After doing this it would see the RPM go back to normal range and turn off the “emergency fan mode” only to turn back on when the RPMs of my fans went below the threshold.

The fix, thankfully, is pretty simple provided you have ipmitools installed. One simply has to use the tool to change the fan thresholds. For my Debian-based Proxmox install I did the following to quiet this beast:

apt install ipmitool
ipmitool sensor thresh FAN1 lower 300 300 400
ipmitool sensor thresh FAN2 lower 300 300 400
ipmitool sensor thresh FAN5 lower 300 300 400
#you'll want to modify the fans to correspond with your own server.

I ran the above commands to turn my 3 fans back to a sane speed. I got caught up for a while because I only ran this command on 2 of my 3 fans at first. The deafening noise continued. This is because if any fan in the system goes below, all fans spin up. Once I changed that third fan’s threshold all was gravy. My ears were ringing for a while, but they’re fine now.

Setup remote git repository with SSH & GIT

I wanted to set up a simple git repository to synchronize my bash scripts between a couple hosts, no fancy github or gitlab software required. These are my notes on how I got it working. Thanks to this site for the information.

On the remote host (server)

mkdir GIT_PROJECT_DIR.git
cd GIT_PROJECT_DIR.git
git init --bare

On the local hosts (client)

Create a git repository and add files to it:

cd GIT_FOLDER
git init
git add *
git commit -m "Initial commit"
git remote add origin USER@REMOTE_HOST:GIT_PROJECT_DIR.git
git push origin master
git branch --set-upstream-to=origin/master

Modify network buffers in Linux using sysctl

After installing Netdata I kept getting alerts that packets were being dropped. After some research I read you can alleviate this by increasing your system’s networking buffer. You can accomplish this with sysctl.

If you run sysctl manually it will not survive a reboot. If you want persistence you simply add the sysctl command line to /etc/sysctl.conf. Thanks to Brennen Smith on serverfault for the info on my particular problem with UDP drops:

#as root:
echo "net.core.rmem_default=8388608" >> /etc/sysctl.conf
echo "net.core.rmem_max=16777216" >> /etc/sysctl.conf
sysctl -p

The above solved my udp problem but you can modify it to change any sysctl things you want to quickly configure and persist.

Mount encfs folder on startup with systemd

A quick note on how to encrypt a folder with encfs and then mount it on boot via a systemd startup script. In my case the folder is located on a network drive and I wanted it to happen whether I was logged in or not.

Create encfs folder:

encfs <path to encrypted folder> <path to mount decrypted folder>

Follow the prompts to create the folder and set a password.

Next create a file which will contain your decryption password

echo "YOUR_PASSWORD" > /home/user/super_secret_password_location
chmod 700 /home/user/super_secret_password_location

Create a simple script to be called by systemd on startup using cat to pass your password over to encfs

#!/bin/bash
cat super_secret_password_location | encfs -S path_to_encrypted_folder path_to_mount_decrypted_folder

Finally create a systemd unit to run your script on startup:

vim /etc/systemd/system/mount-encrypted.service
[Unit] 
Description=Mount encrypted folder 
After=network.target 

[Service] 
User=<YOUR USER> 
Type=oneshot 
ExecStartPre=/bin/sleep 20 
ExecStart=PATH_TO_SCRIPT
TimeoutStopSec=30 
KillMode=process 

[Install] 
WantedBy=multi-user.target

Then enable the unit:

sudo systemctl daemon-reload
sudo systemctl enable mountt-encrypted.service

CPU Pinning in Proxmox

Proxmox uses qemu which doesn’t implement CPU pinning by itself. If you want to limit a guest VM’s operations to specific CPU cores on the host you need to use taskset. It was a bit confusing to figure out but fortunately I found this gist by ayufan which handles it beautifully.

Save the following into taskset.sh and edit VMID to the ID of the VM you wish to pin CPUs to. Make sure you have the “expect” package installed.

#!/bin/bash

set -eo pipefail

VMID=200

cpu_tasks() {
	expect <<EOF | sed -n 's/^.* CPU .*thread_id=\(.*\)$/\1/p' | tr -d '\r' || true
spawn qm monitor $VMID
expect ">"
send "info cpus\r"
expect ">"
EOF
}

VCPUS=($(cpu_tasks))
VCPU_COUNT="${#VCPUS[@]}"

if [[ $VCPU_COUNT -eq 0 ]]; then
	echo "* No VCPUS for VM$VMID"
	exit 1
fi

echo "* Detected ${#VCPUS[@]} assigned to VM$VMID..."
echo "* Resetting cpu shield..."

for CPU_INDEX in "${!VCPUS[@]}"
do
	CPU_TASK="${VCPUS[$CPU_INDEX]}"
	echo "* Assigning $CPU_INDEX to $CPU_TASK..."
	taskset -pc "$CPU_INDEX" "$CPU_TASK"
done

Update 9/29/18: Fixed missing done at the end. Also if you want to offset which cores this script uses, you can do so by modifying  the $CPU_INDEX variable to do a bit of math, like so:

        taskset -pc "$[CPU_INDEX+16]"

The above adds 16 to each process ID, so instead of staring on thread 0 it starts on thread 16.

Fix USB bluetooth in KDE Plasma on CentOS 7

I spent too many hours trying to figure this stupid thing out.. but FINALLY! I have my bluetooth headset working in CentOS 7 with the KDE 4 Plasma environment. Read on if you dare…

First, you must configure dbus to allow your user to use the bluetooth dongle. Add the following above the closing /busconfig tag.  Be sure to replace USERNAME with your user account:

sudo nano /etc/dbus-1/system.d/bluetooth.conf
  <policy user="USERNAME">
    <allow send_destination="org.bluez"/>
    <allow send_interface="org.bluez.Agent1"/>
    <allow send_interface="org.bluez.GattCharacteristic1"/>
    <allow send_interface="org.bluez.GattDescriptor1"/>
    <allow send_interface="org.freedesktop.DBus.ObjectManager"/>
    <allow send_interface="org.freedesktop.DBus.Properties"/>
  </policy>

Remove and re-plug the adapter in.

Next, follow Arch Linux’s excellent guide on how to pair a bluetooth device using bluetoothctl


bluetoothctl
[bluetooth]# power on
[bluetooth]# agent on
[bluetooth]# default-agent
[bluetooth]# scan on

Now make sure that your headset is in pairing mode. It should be discovered shortly. For example,

[NEW] Device 00:1D:43:6D:03:26 Lasmex LBT10

shows a device that calls itself “Lasmex LBT10” and has MAC address “00:1D:43:6D:03:26”. We will now use that MAC address to initiate the pairing:

[bluetooth]# pair 00:1D:43:6D:03:26

After pairing, you also need to explicitly connect the device (every time?):

[bluetooth]# connect 00:1D:43:6D:03:26

If you’re getting a connection error org.bluez.Error.Failed retry by killing existing PulseAudio daemon first:

$ pulseaudio -k
[bluetooth]# connect 00:1D:43:6D:03:26

Finally, configure pulseaudio to automatically switch all audio to your headset by adding the following line to the bottom of /etc/pulse/default.pa:

nano /etc/pulse/default.pa

# automatically switch to newly-connected devices
load-module module-switch-on-connect

Update 7/27: I rebooted my machine and lost my bluetooth, to my dismay. I discovered that my user needs to be a member of the audio group. Since I’m in an active directory environment I think the local audio group got removed at reboot. So, to restore it, as root I had to run this:

usermod -aG audio <user>

After doing that, to prevent logging out and back in again, you can do the following:

su - <USERNAME>

Once that’s done all the bluetoothctl commands worked again.

Backup and restore docker container configurations

I came across a need to start afresh with my docker setup. I didn’t want to re-create all the port and volume mappings for my various containers. Fortunately I found a way around this by using docker-autocompose to create .yml files with all my settings and docker-compose to restore them to my new docker host.

Backup

Docker-autocompose source: https://github.com/Red5d/docker-autocompose

git clone https://github.com/Red5d/docker-autocompose.git
cd docker-autocompose
docker build -t red5d/docker-autocompose .

With docker-autocompose created you can then use it to create .yml files for each of your running containers by utilizing a simple BASH for loop:

for image in $(docker ps --format '{{.Names}}'); do docker run -v /var/run/docker.sock:/var/run/docker.sock red5d/docker-autocompose $image > $image.yml; done

Simple.

Restore

To restore, install and use docker-compose:

sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Next we use another simple for loop to go through each .yml file and import them into Docker. The sed piece escapes any $ characters in the .yml files so they will import properly.

for file in *.yml; do sed 's/\$/\$\$/g' -i $file;
docker-compose -f $file up --force-recreate -d; done

You can safely ignore the warnings about orphans.

That’s it!

Troubleshooting

ERROR: Invalid interpolation format for “environment” option in service “Transmission”: “PS1=$(whoami)@$(hostname):$(pwd)$ “

This is due to .yml files which contain unescaped $ characters.

Escape any $ with another $ using sed

sed 's/\$/\$\$/g' -i <filename>.yml

ERROR: The Compose file ‘./MariaDB.yml’ is invalid because:
MariaDB.user contains an invalid type, it should be a string

My MariaDB docker .yml file had a user: environment variable that was a number, which docker compose interpreted as a number instead of a string. I had to modify that particular .yml file and add quotes around the value that I had for the User environment variable.

Accept multiple SSH RSA keys with ssh-keyscan

I came across a new machine that needed to connect to many SSH hosts via ansible. I had a problem where ansible was prompting me for each post if I wanted to accept the RSA key. As I had dozens of hosts I didn’t want to type yes for every single one; furthermore the yes command didn’t appear to work. I needed a way to automatically accept all SSH RSA keys from a list of server names. I know you can disable RSA key checking but I didn’t want to do that.

I eventually found this site which suggested a small for loop, which did the trick beautifully. I modified it to suit my needs.

This little two-liner takes a file (in my case, my ansible hosts file) and then runs ssh-keyscan against it and adds the results to the .ssh/known_hosts file. The end result is an automated way to accept many SSH keys.

SERVER_LIST=$(cat /etc/ansible/hosts)
for host in $SERVER_LIST; do ssh-keyscan -H $host >> ~/.ssh/known_hosts; done

Using a Bus Pirate to fix Seagate drives

I wrote these notes almost three years ago but never published them. Since I’ve now referenced them again I’ll publish them albeit in a crude state.

7200.11 BSY bug

I had a need to fix the firmware of a Seagate  7200.11 BSY bug, which involved connecting to the RS232 serial ports on the drive and issuing a few commands to clear SMART data. Details here:

http://www.arvydas.co.uk/2012/07/fixing-a-seagate-7200-11-hard-drive-with-arduino/

http://hackaday.com/2012/07/30/recovering-from-a-seagate-hdd-firmware-bug/

https://plus.google.com/u/0/+BillFarrow/posts/ir1xnfu46TE

http://webcache.googleusercontent.com/search?q=cache:F1J2P5E3mrIJ:haquesprojects.com/embedded-device-hacking/using-a-bus-pirate-as-a-usb-ttl-serial-converter/+&cd=4&hl=en&ct=clnk&gl=us

http://fillwithcoolblogname.blogspot.com/2011/02/fixing-seagate-720011-bsy-0-lba-fw-bug.html

Using a Bus Pirate:

Find out what device the bus pirate is given:

dmesg | tail

usb 1-1.6.3: FTDI USB Serial Device converter now attached to ttyUSB0

Next, add your user to the dialout group (thanks to here for the hint)

usermod -a -G dialout $USER

You may need to log out and log back in after issuing the above command for it to take effect.

Fire up a terminal editor (I used screen after learning about my options from here.)

screen /dev/ttyUSB0 115200 8N1

Press Enter and you should be greeted with the Bus Pirate’s HiZ> prompt. Next, enter the following:

1. m – to change the mode
2. 3 – for UART mode
3. 7 – for 38400 bps
4. 1 – for 8 bits of data, no parity control
5. 1 – for 1 stop bit
6. 1 – for Idle 1 receive polarity
7. 2 – for Normal output type

At the “UART>” prompt. Enter “(0)” to show available macros:

UART>(0)
0.Macro menu
1.Transparent bridge
2.Live monitor
3.Bridge with flow control

Now enter “(3)”  (don’t forget the parenthesis – this burned me) to enter bridge mode with flow control and hit “y” at the “Are you sure?” prompt. The terminal will receive input from your device.

UART>(3)
UART bridge
Reset to exit
Are you sure?

Now plug in pins to hard drive. Use this site as a guide for which pins to use. The drive should be upside down to expose the controller board.
BP Gnd (top left) to Gnd on drive (Second pin from the left)
BP MISO (UART RX – bottom right) to TX on drive (far right pin)
BP MOSI (UART TX) to RX on drive (Seconf from the right pin)

I only ended up needing MISO & MOSI, ground wasn’t required.

Un-screw hard drive, add shim to prevent electrical contact

Power on drive

CTRL+Z

/2

(wait 30 seconds)

Z

(un-shim, re-screw hard drive)

U

/1

N1

Power down drive, wait few seconds, power back up

CTRL + Z

m0,2,2,0,0,0,0,22 (enter)

Clear SMART data

A couple years later I came across some old NAS drives that I wanted to use. I ran a full battery of burn-in tests using badblocks and the drives passed with flying colors. The only problem is they had SMART data saying Reallocated_Sector_Ct was past the threshold. Barely. I decided to roll the dice with these drives anyway given their proven performance currently and over the years.

The problem is FreeNAS will e-mail spam you about that SMART attribute. I couldn’t find a good way to suppress those alerts yet have them alert if that number gets worse, so I decided to cheat and clear all SMART data from those drives, thus getting FreeNAS happy with me yet alerting me if the reallocated sector count increases in the future.

I read a few sources to accomplish this with my bus pirate.

https://blog.zencoffee.org/2011/07/bus-pirate-as-ftdi-cable/

https://forum.hddguru.com/viewtopic.php?f=1&t=33886&start=20&mobile=mobile

https://forum.hddguru.com/viewtopic.php?f=1&t=33886&start=20

Use the same instructions as above for hooking up the bus pirate to the drive’s RS232 ports (to the right of the SATA port.)

Once you’ve serial connected to the drive, it’s three simple commands to clear the SMART data:

CTRL + Z
/1
N1