Tag Archives: linux

Transfer linode VM over ssh

I love Linode for their straightforward pricing. I can use them for temporary infrastructure and not have to worry about getting overcharged. When it comes time to transfer infrastructure back, the process is fairly straightforward. In my case I wanted to keep a disk image of my Linode VM for future use.

The linode documentation is very good. I used their copy an image over ssh article combined with their rescue and rebuild article sprinkled with a bit of gzip compression and use of pv to grab my linode image locally, complete with a progress bar.

First, boot your linode into recovery mode via dashboard / Linodes / <name of your linode>, then click on Rescue tab, map your drives as needed.

Launch console (top right) to get into the recovery shell. In my case I wanted to SSH into my linode to grab the image, so I set a password and started the ssh service:

/etc/init.d/ssh start

Then on your end, pipe ssh , gzip, pv and dd together to grab the compressed disk image with progress monitoring:

ssh root@ "dd if=/dev/sda | gzip -1 -" | pv | dd of=linode-image.gz


Using ProxMox as a NAS

Lately I’ve been very unhappy with latest FreeBSD causing reboots randomly during disk resilvering. I simply cannot tolerate random reboots of my fileserver. This fact combined with the migration of OpenZFS to the ZFS on Linux code base means it’s time for me to move from a FreeBSD based ZFS NAS to a Linux-based one.

Sadly there aren’t many options in this space yet. I wanted something where basic tasks were taken care of, like what FreeNAS does, but also supports ZFS. The solution I settled on was ProxMox, which is a hypervisor, but it also has ZFS support.

The biggest drawback of ProxMox vs FreeNAS is the GUI. There are some disk-related GUI options in ProxMox, but mostly it’s VM focused. Thus, I had to configure my required services via CLI.

Following are the settings I used when I configured my NAS to run ProxMox.

Repo setup

If you don’t want to pay for a proxmox license, change the PVE enterprise repository to the free version by modifying /etc/apt/sources.list.d/pve-enterprise.list to the following:

deb http://download.proxmox.com/debian/pve buster pve-no-subscription

Then run at apt update & apt upgrade.

Email alerts

Postfix configuration

Edit /etc/postfix/main.cf and tweak your mail server config as needed (relayhost). Restart postfix after editing:

systemctl restart postfix

Forward mail for root to your own email

Edit /etc/aliases and add an alias for root to forward to your desired e-mail address. Add this line:


Afterward run:


ZFS configuration

Pool Import

Import the pool using the zpool import -f command (-f to force import despite having been active in a different system)

zpool import -f  

By default they’re imported into the main root directory (/). If you want to have them go to /mnt, use the zfs set mountpoint command:

zfs set mountpoint=/mnt/ 


Install and configure zfs-zed

apt install zfs-zed

Modify /etc/zfs/zed.d/zed.rc and uncomment ZED_EMAIL_ADDR, ZED_EMAIL_PROG, and ZED_EMAIL_OPTS. Edit them to suit your needs (default values work fine, they just need to be uncommented.) Optionally uncomment ZED_NOTIFY_VERBOSE and change to 1 if you want more verbose notices like what FreeNAS does (scrub notifications, for example.)

After modifying /etc/zfs/zed.d/zed.rc, restart zed:

systemctl restart zfs-zed


By default ProxMox scrubs each of your datasets on the second Sunday of every month. This cron job is located in /etc/cron.d/zfsutils-linux. Modify to your liking.

Snapshot & Replication

There are many different snapshot & replication scripts out there. I landed on Sanoid. Thanks to SvennD for helping me grasp how to get it working.

Install sanoid :

#Install necessary packages
apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer git
# Clone repo, build deb, install
git clone https://github.com/jimsalterjrs/sanoid.git cd sanoid
ln -s packages/debian . 
dpkg-buildpackage -uc -us 
apt install ../sanoid_*_all.deb 


Edit /etc/sanoid/sanoid.conf with a backup and retention schedule for each of your datasets. Example taken from sanoid documentation:

	use_template = production
	use_template = production
	recursive = yes
	process_children_only = yes
	hourly = 4

# templates below this line #

        frequently = 0
        hourly = 36
        daily = 30
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

Once sanoid.conf is to your liking, create a cron job to launch sanoid every hour (sanoid determines whether any action is needed when executed.)

crontab -e
#Add this line, save and exit
0 * * * * /usr/sbin/sanoid --cron


syncoid (part of sanoid) easily replicates snapshots. The syntax is pretty straightforward:

syncoid <source> <destination> -r 
#-r means recursive and is optional

For remote locations specify a username@ before the ip/hostname, then a colon and the dataset name, for example:

syncoid root@ localDataset -r

You can even have a remote source go to a different remote destination, which is pretty neat.

Other syncoid options of interest:

--debug  #for seeing everything happening, useful for logging
--exclude #Regular expression to exclude certain datasets
--src-bwlimit #Set an upload limit so you don't saturate your bandwidth
--quiet #don't output anything unless it's an error

Automate synchronization by placing the same syncoid command into a cronjob:

0 */4 * * * /usr/sbin/syncoid --exclude=bigdataset1 --source-bwlimit=1M --recursive pool/data root@
#if you don't want status emails when the cron job runs, add --quiet


Install the nfs-kernel-server package and specify your NFS exports in /etc/exports.

apt install nfs-kernel-server portmap

Example /etc/exports :


Restart nfs-server after modifying your exports:

systemctl restart nfs-server


Install samba, configure /etc/samba/smb.conf, and add users.

apt install samba
systemctl enable smbd

/etc/samba/smb.conf syntax is fairly straightforward. See the samba documentation for more information. Example share configuration:

comment = Example share
path = /mnt/example
valid users = user1 user2
writable = yes

Add users to the system itself with the adduser command:

adduser user1

Add those same users to samba with the smbpasswd -a command. Example:

smbpasswd -a user1

Restart samba after making changes:

systemctl restart smbd

SMART monitoring

Taken from https://pve.proxmox.com/wiki/Disk_Health_Monitoring:

By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and /dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem. 

Edit the file /etc/smartd.conf to suit your needs. You can specify/exclude devices, smart attributes, etc there. See here for more information. Restart the smartd service after modifying.

UPS monitoring

apc-upsd was easiest for me to configure, so I went with it. Thanks to this blog for giving me the information to get started.

First, install apcupsd:

apt install apcupsd apcupsd-doc

As soon as it was installed my console kept getting spammed about IRQ issues. To stop these errors I stopped the apcupsd daemon:

 systemctl stop apcupsd

Now modify /etc/apcupsd/apcupssd.conf to suit your needs. The section I added for my CyberPower OR2200LCDRT2U was simply:


Then modify /etc/default/apcupsd to specify it’s configured:


After configuring, you can restart the apcupsd service

systemctl start apcupsd

To check the status of your UPS, you can run the apcaccess status command:

/sbin/apcaccess status

Log monitoring

Install Logwatch to monitor system events. Here is a good primer on all of Logwatch’s options.

apt install logwatch

Modify /usr/share/logwatch/default.conf/logwatch.conf to suit your needs. By default it runs daily (defined in /etc/cron.daily/00logwatch). I added the following lines for my config to filter out unwanted information:

Service = "-zz-disk_space"
Service = "-postfix"
Service = "vsmartd"
Service = "-zz-lm_sensors"

Manually run logwatch to get a preview of what you’ll see:

logwatch --range today --mailto YOUR_EMAIL_ADDRESS

UPDATE 8/29/2020
I discovered additional tweaking to logwatch to get it exactly how I like it (thanks to this post and this one at serverfault.)

Defaults for monitored services are located in /usr/share/logwatch/default.conf/services/

You can copy this default file to /etc/logwatch/conf/services/<filename.conf> and then modify the service as needed. In my case I wanted to ignore logins for a particular user from a particular machine. This can be done by copying & editing sshd.conf and adding the following:

# Ignore these hosts
*Remove =
*Remove = X.Y.123.123
# Ignore these usernames
*Remove = testuser
# Ignore other noise. Note that we need to escape the ()
*Remove = "pam_succeed_if\(sshd:auth\): error retrieving information about user netscan.*


ZFS-ZED not sending email

If ZED isn’t sending emails it’s likely due to an error in the config. For some reason default values still need to be uncommented for zed to work, even if left unaltered. Thanks to this post for the info.

Samba share access denied

If you get access denied when trying to write to a SMB share, double check the file permissions on the server level. Execute chmod / chown as appropriate. Example:

chown user1 -R /mnt/example/user1

zfs drive removal ‘part of active pool’ fix

Occasionally I will manually offline a disk in my ZFS pool for one reason or another. Annoyingly I will sometimes get this error when I try to online that same disk back into the pool:

cannot online /dev/sda: cannot relabel '/dev/sda': unable to read disk capacity

The fix, thankfully, is fairly simple. Simply run the following command (make double sure you’re doing it on the correct device!)

sudo wipefs -a <DEVICE>

After I ran that command ZFS automatically picked the disk back and resilvered it into the pool.

Thanks to this superuser.com discussion for the advice!

rsync to remote directory containing spaces

I ran into an issue where if I ran rsync to a remote server with a path that contained spaces it would simply terminate at the first space instead of copying the whole path. I learned you need to do some quote tricks to get this to work properly (thanks to this site for helping me.)

The trick is to enclose the entire remote path, server included, in SINGLE quotes. In addition to the single quotes you still need to escape every space with a backslash ( \ )

rsync -aP <source_folder> 'REMOTE_SERVER:/full\ path\ with\ escaped\ spaces/folder/'

Manually set monitor refresh rate with xrandr

My recent distro hopping has found some distros don’t let you change your monitor’s refresh rate via the GUI. After a while I found this post which explains the syntax for xrandr to force the refresh rate of 100hz (adjust to your liking)

xrandr --output DP-0 --mode 3440x1440 --rate 100

Create an auto start item with this line of code in it to get this to happen automatically on startup.

KeePassRPC incompatible with the current KeePass version

I keep forgetting about this snag so I’ll document it. In Debian / Ubuntu distros, once you’ve added the PPA to have the latest version of KeePass installed, if you try to install the KeePassRPC plugin (Kee) it will tell you the version is incompatible, even though it is.

The following plugin is incompatible with the current KeePass version: /usr/lib/keepass2/Plugins/KeePassRPC.plgx

Have a look at the plugin's website for an appropriate version.

The fix, as found here, is to install the mono-complete package

sudo apt install mono-complete

Restart KeePass after installation. That’s it!

Headless Private Internet Access with OpenVPN & Port Forwarding

I wanted to initiate a connection to my VPN provider Private Internet Access (PIA) from one of my headless CentOS 7 servers. Unfortunately PIA software requires a GUI to work. Fortunately, PIA provides ovpn files that fully work with the openVPN software, which does work headlessly.

I followed this guide from htpcguides.com to get me going. I tweaked it a bit to work with CentOS 7. I then used this guide to help me set up port forwarding to work with transmission-daemon.

First, install openvpn & iproute

sudo yum -y install openvpn

Next grab PIA’s ovpn files for their servers:

wget https://www.privateinternetaccess.com/openvpn/openvpn.zip
sudo unzip openvpn.zip -d /etc/openvpn/pia

Now create a credentials file to specify your PIA username and password. The format is simply the first line being your PIA username, and the second line being your PIA password, nothing else.

#Create file
sudo vim /etc/openvpn/pia/.secret

#Enter PIA credentials into the file, then save and exit

#Change permissions to be readable only by root
sudo chmod 700 /etc/openvpn/pia/.secret

Test if it’s working by manually running OpenVPN

sudo openvpn --config /etc/openvpn/PIA_SITE_FILENAME.ovpn --auth-user-pass /etc/openvpn/pia/.secret

Create a systemd service file to get OpenVPN to run on startup

sudo vim /usr/lib/systemd/system/pia.service

After=syslog.target network.target

ExecStart=/usr/sbin/openvpn --daemon --config "/etc/openvpn/PIA_SITE_FILENAME.ovpn" --auth-user-pass /etc/openvpn/.secret


Reload systemd daemons and enable at boot:

sudo systemctl daemon-reload
sudo systemctl enable pia.service

Start the service (or reboot) and then use curl to verify your external IP has changed

sudo systemctl start pia.service
curl https://ipinfo.io/ip

Port Forwarding

PIA is awesome because it supports port forwarding to facilitate things like torrents. I took the script from this guide and tweaked it a bit to suit my needs (and to correct a syntax error)

Note: PIA only supports port forwarding from specific regions. Here is the current list.

Download script to pia directory

sudo wget -O /etc/openvpn/pia/portforward.sh https://raw.githubusercontent.com/HTPCGuides/pia-port-forwarding-scripts/master/transmission-pia-port-forwarding.sh
sudo chmod +x /etc/openvpn/portforward.sh

Edit the script to provide PIA username and pass, and Transmission user/pass (if you have that configured.) The variables to change are USERNAME,PASSWORD, TRANSUSER, and TRANSPASS

Test the script by running it manually (make sure both your pia service and transmission services are already running.) I’ve noticed that the script will say success even on failure which is why I suggest using bash -x so you can see actual error messages from PIA regarding port forwarding:

sudo bash -x /etc/openvpn/pia/portforward.sh

After that’s run you can check your transmission daemon to ensure it is listening on the PIA port that’s been forwarded.

Configure to run on startup and every two hours by invoking cron:

sudo crontab -e

@reboot sleep 60 && /etc/openvpn/pia/portforward.sh | while IFS= read -r line; do echo "$(date) $line"; done >> /var/log/pia_portforward.log 2>&1 #PIA Port Forward
0 */2 * * * /etc/openvpn/pia/portforward.sh | while IFS= read -r line; do echo "$(date) $line"; done >> /var/log/pia_portforward.log 2>&1 #PIA Port Forward

If you are paranoid about transmission running without the VPN working you can tweak your transmission service to rely on the pia service before starting. Modify /usr/lib/systemd/system/transmission-daemon.service and tweak the After line and add a Required line so it now reads this:

After=network.target pia.service

Don’t forget to run a sudo systemctl daemon-reload afterward.


If you get the error message “/etc/openvpn/pia/portforward.sh: line 35: [: : integer expression expected” it means you have to tweak the script a bit (it’s broken?)

Change line 35 to be the following:

if [ "$SPLITVPN" == 1 ]; then

Merge multiple MP4 files into one with ffmpeg concat

I had a bunch of MP4 files I wanted to merge into a single file. FFMpeg came to the rescue, but it had a bit of a quirk I needed to figure out. Thanks to this site for the help.

First, create a list of files you wish to merge. In this example I want every file with a specific file pattern beginning with 2019-05-02. I take that output and run sed against it to add “file ‘” to the begging and a closing ” ‘ ” at the end. The end result is a nice list of files for ffmpeg to ingest.

ls 2019-05-02* | sed "s/^/file '/g; s/$/'/g" > filelist.txt

With our shiny new list we can now tell ffmpeg to use it to concatenate to a single file:

ffmpeg -f concat -safe 0 -i filelist.txt -c copy OUTPUT_FILENAME.mp4

Split flac files with shnsplit

I had a few single FLAC files with cue files I wanted to put into Plex but to my dismay it doesn’t read the CUE files at all. Thus I needed to split the one FLAC file into multiple pieces with shnsplit. Thanks to Stack Exchange for the help.

On my Debian system:

 sudo apt install cuetools shntool flac

With the necessary tools installed you simply have to run the shnsplit command:

 shnsplit -f FILENAME.cue -t "%a - %n %t - %p" FILENAME.flac

the -t parameters formats the filename as desired per the manpage

-t fmt
Name output files in user‐specified format based on CUE sheet fields. The following formatting strings are recognized: 

Track title
Track number