All posts by nicholas

Xenserver SSH backup script

Citrix Xenserver is an amazing hypervisor with pretty much every function released to you for free. One thing they do not handle, however, is automated backups.

I have hacked together a backup script for myself that seems to work fairly well. It is my own mix of this and this script along with some logic for e-mail reporting that I came up with myself. It does not require any modification of the xenserver host at all (no need to mount anything!) with the exception of adding a public key to the xenserver’s authorized_keys file.

This script can be run on anything with BASH and the appropriate UNIX tools (even other xenservers if you want) and uses SSH to initiate and transfer the backups to a location of your choosing.

Place this script on the machine you want to be initiating / saving the backups on. It requires that you generate an SSH public/private key pair, which can be done with this command:

ssh-keygen

Add the resulting .pub file’s contents to your xenserver’s /root/.ssh/authorized_keys file (create it if it doesn’t exist.) Take note of the location of the private key file that was generated with that command and put that path in the script.

You can download the script here or view it below. This script has been working pretty well for me. Note it will not work with any VMs that have spaces in their names. I was too lazy to debug this so I just renamed the problem VMs to remove the spaces. Enjoy!

#!/bin/bash

# Modified August 30, 2015 by Nicholas Jeppson
# Taken from http://discussions.citrix.com/topic/345960-xenserver-automated-snapshots-script/ and modified to allow for ssh backups
# Additional insight taken from https://github.com/cepa/xen-backup

# [Usage & Config]
# This script involves two computers: a xenserver machine and a backup machine.
# Put this script on the backup server and run with any account that has privileges to the desired export directory.
#
# This script assumes you have already created a private and public key pair on the backup server
# as well as adding respective the public key to the xenserver authorized_keys file.
#
# [How it works]
# Step1: Snapshots each VM on the xen server
# Step2: Backs up the snapshots to specified location
# Step3: Deletes temporary snapshot created in step 1
# Step4: Deletes old VM backups as defined later in this file
#
# [Note]
# This script will only work with VMs that don't have spaces in their names
# Please make sure you have enough disk space for BACKUP_PATH, or backup will fail
#
# Tested on xenserver 6.5
#
# Modify the variables in the config section below to suit your particular environment's needs.
#

############### Config section ###############

#Location where you want backups to go
BACKUP_PATH="/mnt/backup/OS/"

#SSH configuration
SSH_CIPHER="arcfour128"

#Number of backups to keep
NUM_BACKUPS=2

#Xenserver ssh configuration
#This dictates the address and location of keyfiles as they reside on the xenserver
XEN_ADDRESS="192.168.1.1"
XEN_KEY_LOCATION="/home/backup/.ssh/backup"
XEN_USER="root"

#E-mail configuration
EMAIL_ADDRESS="youremail@provider.here"
EMAIL_SUBJECT="`hostname -s | awk '{print "["toupper($1)"]"}'` VM Backup Report: `date +"%A %b %d %Y"`"

########## End of Config section ###############

ret_code=0

#Replace any spaces found with backslashes because dd doesn't like them
BACKUP_PATH_ESCAPED="`echo $BACKUP_PATH | sed 's/ /\\\ /g'`"

# SSH command
remote_exec() {
chmod 0600 $XEN_KEY_LOCATION
ssh -i $XEN_KEY_LOCATION -o "StrictHostKeyChecking no" -c $SSH_CIPHER $XEN_USER@$XEN_ADDRESS $1
}

backup() {
echo "======================================================"
echo "VM Backup started: `date`"
begin="$(date +%s)"
echo "Backup location: ${BACKUP_PATH}"
echo

#add a slash to the end of the backup path if it doesn't exist
if [[ "$BACKUP_PATH" != */ ]]; then
BACKUP_PATH="$BACKUP_PATH/"
fi

#Build array of VM names
VMNAMES=$(remote_exec "xe vm-list is-control-domain=false | grep name-label | cut -d ':' -f 2 | tr -d ' '")

for VMNAME in $VMNAMES
do
echo "======================================================"
echo "$VMNAME backup started `date`"
echo
before="$(date +%s)"

# create snapshot
TIMESTAMP=`date '+%Y%m%d-%H%M%S'`
SNAPNAME="$VMNAME-$TIMESTAMP"
SNAPUUID=$(remote_exec "xe vm-snapshot vm=\"$VMNAME\" new-name-label=\"$SNAPNAME\"")

# export snapshot
# remote_exec "xe snapshot-export-to-template snapshot-uuid=$SNAPUUID filename= | gzip" | gunzip | dd of="$BACKUP_PATH/$SNAPNAME.xva"
remote_exec "xe snapshot-export-to-template snapshot-uuid=$SNAPUUID filename=" | dd of="$BACKUP_PATH/$SNAPNAME.xva"

#if export was unsuccessful, return error
if [ $? -ne 0 ]; then
echo "Failed to export snapshot name = $snapshot_name$backup_ext"
ret_code=1

else
#calculate backup time, print results
after="$(date +%s)"
elapsed=`bc -l <<< "$after-$before"`
elapsedMin=`bc -l <<< "$elapsed/60"`
echo "Snapshot of $VMNAME saved to $SNAPNAME.xva"
echo "Backup completed in `echo $(printf %.2f $elapsedMin)` minutes"

# destroy snapshot
remote_exec "xe snapshot-uninstall force=true snapshot-uuid=$SNAPUUID"

#remove old backups (uses num_backups variable from top)
BACKUP_COUNT=$(find $BACKUP_PATH -name "$VMNAME*.xva" | wc -l)

if [[ "$BACKUP_COUNT" -gt "$NUM_BACKUPS" ]]; then

OLDEST_BACKUP=$(find $BACKUP_PATH -name "$VMNAME*" -print0 | xargs -0 ls -tr | head -n 1)
echo
echo "Removing oldest backup: $OLDEST_BACKUP"
rm $OLDEST_BACKUP
if [ $? -ne 0 ]; then
echo "Failed to remove $OLDEST_BACKUP"
fi
fi
echo "======================================================"
fi
done

end=$"$(date +%s)"
total_time=`bc -l <<< "$end-$begin"`
total_time_min=`bc -l <<< "$total_time/60"`
echo "Backup completed: `date`"
echo "VM Backup completed in `echo $(printf %.2f $total_time_min)` minutes"
echo
}

#Run the backup function and save all output to a variable, including stderr
BACKUP_OUTPUT=$(backup 2>&1)

#Clean up the output of the backup function
#Remove records count from dd, do some basic math to make dd's numbers more human readable
BACKUP_OUTPUT_HUMANIZED=$(echo "$BACKUP_OUTPUT" | sed -r '/.*records /d' | tr -d '()' \
| awk '{sub(/.*bytes /, $1/1024/1024/1024" GB "); sub(/in .* secs/, "in "$5/60" mins "); sub(/mins .*/, "mins (" $7/1024/1024" MB/sec)"); print}')

#Send a report e-mail with the backup results
echo "$BACKUP_OUTPUT_HUMANIZED" | mail -s "$EMAIL_SUBJECT" "$EMAIL_ADDRESS"

exit $ret_code

Avoid prompts when installing FreeBSD ports

The FreeBSD ports tree is wonderful for installing software but sometimes it can be a real pain. Recently I was trying to install Emby in FreeBSD because why not? The instructions were easy enough except for when I ran

make install clean

I was constantly barraged with choices for things. I want to assume the default on all of these and not be barraged with questions.

Thanks to stack exchange I learned it’s relatively easy to bypass all these questions. Simply add:

BATCH=yes

to the end of your make install clean statement to assume the defaults to all the questions for the package. The Emby guide is pretty comprehensive, but I would add this command at the bottom:

make install clean BATCH=yes

Handy.

Measure SSH transfer speeds

SSH is a beautiful thing. In addition to remotely administering machines you can use it to transfer files. To do this one simply pipes the cat command on both ends. For example, to copy hello.txt on the source host to hi.txt on the destination host, the command would be:

ssh remote_host cat hello.txt | cat > hi.txt

The command takes the contents of hello.txt and pipes it over to the remote host. The cat command on the remote host takes what was piped to it as  input and the > sign instructs cat to take its input and output it to hi.txt.

A great way to measure transfer speeds using ssh between two hosts is to take /dev/zero on the source host and output it to /dev/null on the destination host. This bypasses any disk speed bottlenecks and only measures network throughput. Combine this with the pv command to get a nice graphical view of how fast the transfer is going.

ssh remote_host cat /dev/zero | pv | cat > /dev/null

The default options between my machines result in about a 65 megabytes a second transfer speed.

1

It turns out that the encryption cipher used makes a big difference on transfer speeds. Use the -c command to specify which cipher to use and see how much of a difference this makes. -o compression=no can also help with transfer speeds.

The fastest cipher I’ve found is arcfour. It’s touted as less secure, but for my local network I can accept the risk (thanks to slashdot for the discussion.)

ssh -c arcfour -o Compression=no remote_host cat /dev/zero | pv | cat > /dev/null

2

Using acrfour more than doubles the speed for me! Amazing.

Allow non-root users to mount disks

I came across a need today to allow a regular (non-root) user to mount disks in Ubuntu 14.04 Trusty Tahr. I usually use sudo but in this case I needed to be able to run photorec as a regular user.

The way to accomplish this is to add the regular user to the disk group. To accomplish this, run this command:

sudo usermod -a -G disk <username>

If you are logged in as that user, you will have to log out and log back in to receive the permissions. Once this is done you should be able to mount disks without using sudo or being root.

Disable IPv6 on an interface in Linux

After tethering my phone to my laptop and googling “what is my ip” I was surprised to find an IPv6 address. Apparently my mobile carrier has implemented IPv6. Bravo to them.

Unfortunately, when I initiated my VPN, which is supposed to tunnel all traffic through it, my IP address didn’t change. This is because my VPN is IPv4 only. My system prioritizes IPv6 traffic, so if I happen to go to any IPv6 enabled site such as google, my VPN tunnel is bypassed entirely.

I don’t like the security implications of this. The long term solution is to implement IPv6 with my VPN; however while traveling I won’t be able to do that. The short term solution is to simply disable IPv6 for the interface that has it, in my case usb0 as that is what is tethered to my phone.

This simple command will do the trick:

sudo sh -c 'echo 1 > /proc/sys/net/ipv6/conf/usb0/disable_ipv6'

Change USB0 to whatever interface you would like (or all of them) and you’re done! Thanks to this site for the information.

 

 

 

Install Cinnamon on a Chromebook with Crouton

I really love using Crouton on my Chromebook Pixel LS 2015. I was sad to see that there is no cinnamon desktop environment target with the latest versions of crouton. Below is what I did to get Cinnamon on my chromebook. Much of what I did was taken from https://gist.github.com/sohjsolwin/5939948

  1. Create a base chroot
  2. Enter your chroot
sudo apt-get update
sudo apt-get install software-properties-common python-software-properties
sudo add-apt-repository ppa:tsvetko.tsvetkov/cinnamon
sudo apt-get update
sudo apt-get install cinnamon

Once Cinnamon was installed I needed to know how to start it manually. Thanks to the Arch Linux forums for explaining it. You have to create a .xinitrc file in your home directory within your chroot.

echo "exec cinnamon-session" > ~/.xinitrc

Trying to manually start cinnamon by typing startx didn’t work – I got a blank screen and had to hard reset to get anything to come back. Thanks to github I learned you need to use xinit instead of startx.

Lastly, we need to create a suitable startcinnamon script.

wget https://gist.github.com/sohjsolwin/5934362/raw/f68fc0942798902a0bd48f40c17dc0cd5cf585ea/startcinnamon

Modify the file to remove the startx command with xinit. Also remove everything after xinit. My file is as follows:

APPLICATION="${0##*/}"

USAGE="$APPLICATION [options]

Wraps enter-chroot to start a Mint session.
By default, it will log into the primary user on the first chroot found.

Options are directly passed to enter-chroot; run enter-chroot to list them."

exec sh -e "`dirname "$0"`/enter-chroot" "$@" xinit

Make this file executable (chmod +x startcinnamon) and move it to the /usr/local/bin directory of your chromebook (not your chroot.) Now all you need to do is enter

sudo startcinnamon

and your cinnamon desktop should come up!


 

Update 2016-01-04

These two scripts seem to work a little bit better. Place this one within your chroot under /usr/local/bin/startcinnamon:

#!/bin/sh -e
# Copyright (c) 2015 The crouton Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

# Launches GNOME; automatically falls back to gnome-panel

exec crouton-noroot gnome-session-wrapper cinnamon

Place this one in /usr/local/bin outside your chroot (on your chromebook itself.)

#!/bin/sh -e
# Copyright (c) 2015 The crouton Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

set -e

APPLICATION="${0##*/}"

USAGE="$APPLICATION [options]

Wraps enter-chroot to start a GNOME session.
By default, it will log into the primary user on the first chroot found.

Options are directly passed to enter-chroot; run enter-chroot to list them."

exec sh -e "`dirname "\`readlink -f "$0"\`"`/enter-chroot" -t cinnamon "$@" "" \
    exec startcinnamon

Install Ubuntu chroot on your Chromebook

I recently got a Chromebook Pixel 2015 LS. It is a very nice device. Chromium OS is great but a power user like myself wants a little more functionality out of this beautiful machine.

Fortunately it’s not too difficult to get an Ubuntu chroot running side by side with chromium. The Google developers have made a script to automate the process.

Below is my experience installing an Ubuntu Trusty chroot on my chromebook 2015 LS.

Prepwork

  • Enter developer mode:
    Press ESC, Refresh, power simultaneously (when the chromebook is on)

    • Every time you power on the chromebook from now on you’ll get a scary screen. Press CTRL-D to bypass it (or wait 30 seconds)
    • If you hit space on this screen instead of CTRL+D it will powerwash (nuke) your data
      A scary screen will pop up saying the OS is missing or damaged. Press CTRL D, then press Enter when the OS verification screen comes up.
  • Wait several minutes for developer mode to be installed. Note it will wipe your device to do this.

Install Crouton

Now that we’re in developer mode we will use a script called crouton to install an Ubuntu chroot (thanks to lifehacker for the guidance.)

  1. Download Crouton:  https://github.com/dnschneid/crouton
  2. Press CTRL ALT T to open terminal
  3. Type ‘shell’ (without quotes) and hit enter
  4. sudo sh ~/Downloads/crouton -r trusty -t touch,extension,unity-desktop,keyboard,cli-extra -e -n unity
    1. Note the arguments are suited to my needs. You will want to read up on the documentation to decide which options you want, i.e. desktop environment
  5. Install this crouton extension to integrate clipboard (in conjuction with the ‘extension’ parameter above)

General points of interest / lessons learned

  • Don’t enter the chroot and type startx. It will hard freeze your chromebook.
  • You don’t need to blow your chroot away if you want a different desktop environment, simply install desired environment on your existing chroot
  • To switch between chroots pres Ctrl + Alt + Shift + F2 or F3 (back or forward arrows on top row, not to be confused with the arrows on the bottom right of the keyboard)

High DPI

High DPI screens are a pain to deal with. Here are my tweaks:

  • Go to System settings / Displays / Scale for menu and title bars. I like 1.75
  • Alternatively you can change your resolution. If you mess up and X won’t start properly, delete ~/.config/monitors.xml (thanks to askubuntu)
  • Use the setres script to enable other resolutions in the display manager
    • setres 1440 960
  • Firefox fix tiny text:
    • go to about:config and modify layout.css.devPixelsPerPx, set to 2

Other tweaks:

  • Make trackpad match Chrome:
    • System settings / mouse and trackpad / Check “Natural Scrolling”
  • Remove lens suggestions:
    • Install unity-tweak-tool, notify-osd, overlay-scrollbar, unity-webapps-service
    • Run unity-tweak-tool and uncheck “search online sources” from the search tab
  • Move docky bar to the left:
    • sudo apt-get install gconf-editor
    • Press Alt+F2, enter: gconf-editor and in this configuration editor, navigate to “apps -> docky-2 -> Docky -> Interface -> DockPreferences -> Dock1″
    • On the right side there are some properties with their corresponding values, including the position of the dock which you can change from “Bottom” to “Top/Left/Right” to move Docky to the upper part of the desktop.
  • Install Mac OSX theme
  • Install elementary OS chroot

Garbled mouse cursor when switching between chroots

Sometimes the mouse cursor would get all weird when switching between my chroots. The fix is to install the latest Intel drivers within your chroot.

sudo apt-get install software-properties-common python-software-properties
sudo add-apt-repository https://download.01.org/gfx/ubuntu/14.04/main
wget --no-check-certificate https://download.01.org/gfx/RPM-GPG-KEY-ilg -O - | sudo apt-key add -
wget --no-check-certificate https://download.01.org/gfx/RPM-GPG-KEY-ilg-2 -O - | sudo apt-key add -
sudo apt-get update
sudo apt-get upgrade

That’s it.. for now 🙂


 

Update 07/27/2015

I discovered that creating chroots was taking a very long time due to the mirror being chosen. I discovered the -m parameter of crouton which allows you to specify a mirror of your choosing. My updated setting is thus:

sudo sh ~/Downloads/crouton -r trusty -t touch,extension,kde-desktop,keyboard,cli-extra -e -n unitykde -m http://mirrors.xmission.com/ubuntu

If you happened to do a CTRL + C to cancel an existing chroot install that was going slowly, you can simply append the -m parameter above along with -u -u to resume with the updated mirror:

sudo sh ~/Downloads/crouton -r trusty -t touch,extension,kde-desktop,keyboard,cli-extra -e -n unitykde -m http://mirrors.xmission.com/ubuntu -u -u

Install OpenWRT on ASUS RT-16N

My parents’ ASUS RT-16N has been running dd-wrt for years now. I recently enhanced it with optware but something went horribly wrong after a few days. A drive out to their house revealed that the whole unit had spontaneously reset itself to factory defaults.

OpenWRT has come a long way since I last investigated it. I decided to give it another try as it’s till actively being developed whereas dd-wrt is not.

The wiki article on this device is a little bit out of date. I had to update it a little bit to get it to work.

To install OpenWRT on this device, SSH into it and run the following commands:

cd /tmp
wget http://downloads.openwrt.org/barrier_breaker/14.07/brcm47xx/generic/openwrt-brcm47xx-generic-squashfs.trx mtd -r write openwrt-brcm47xx-generic-squashfs.trx linux

That part went smoothly. The last part to configure was wireless N. After some searching I came across this post on the OpenWRT forums which worked nicely for me.  SSH into the router and do the following to enable full wireless N functionality:

opkg update
opkg install kmod-brcmsmac
opkg install kmod-brcmutil
rmmod b43
rmmod b43legacy
rmmod wl
rmmod brcmsmac
rmmod brcmutil
modprobe brcmsmac

# make sure to delete the old config files ... you have to ...
rm -f /etc/config/wireless 
wifi detect > /etc/config/wireless
vi /etc/config/wireless

Now comment out # "option disabled 1"

I had to take navid’s steps a little bit further by tweaking /etc/config/wireless a bit to add some features. My working wireless configuration is below:

config wifi-device 'radio0'
 option type 'mac80211'
 option channel '11'
 option hwmode '11ng'
 option path 'bcma0:0'
 list ht_capab 'GF'
 list ht_capab 'SHORT-GI-20'
 list ht_capab 'SHORT-GI-40'
 option txpower '19'
 option country '00'

config wifi-iface
 option device 'radio0'
 option network 'lan'
 option mode 'ap'
 option ssid 'SSID'
 option encryption 'psk2'
 option key 'SSIDKEY'

Success! Fully functional OpenWRT on my parents’ Asus RT-16N.

Owncloud server did not acknowledge the last chunk error

I experienced an issue with Owncloud today where small files wouldn’t synchronize properly. The error message was

The server did not acknowledge the last chunk. (No e-tag were present)

I could not find a way around this issue. Some googling revealed this page on github. It appears I’m not the only one with this issue.

Deep in the thread, asinteg-daehn provided a workaround for the issue. It’s not ideal, but it works. Rename the file to something else, wait for it to sync, then rename it back.

Update:
Currently only found a WORKAROUND: Renaming of all affected files.

  • open activity dialog of OC Client
  • go to each affected file by double clicking on it’s error message
  • rename it by e.g. a prefix “_” -> “_myfile.txt”
  • resync succeeds
  • now rename it back
  • resync should succeed, too

This is very annoying, but a simple workaround.

It worked for me. Hopefully it will work for you too.

Install OTRW2 on DD-WRT

Optware done the right way 2 is a set of scripts designed to enhance the functionality of your DD-WRT router. I’ve recently installed it on my parents’ router so I can more or less have a full Linux box running in their house (it makes my life easier.)

The tutorial for install it is pretty comprehensive. These are my notes on the experience.

  • USB devices needs to be ext2 formatted (fat won’t do.) This is because the script makes a bunch of symlinks to that device.
  • Mount ext2 formatted drive as /opt (Services / USB / Disk Mount Point)
  • Reboot router if you made any changes to mountpoints.

SSH into the router and run the following:

wget -O /tmp/prep_optware http://dd-ware.googlecode.com/svn/otrw2/prep_optware
sh /tmp/prep_optware

Installation takes some time. Wait one minute after installation complete message and reboot router.

Once rebooted, use the service command to see which services are available. Green means the service is enabled.

service mypage on

Enables the mypage service, but you still have to reboot. Reboot router after any changes to services to enable / disable them.

Small overview of services as taken from dd-wrt forums:

rotate_log = move the log file to /opt/var/log/
pixelserv = addblocking on you network.
unfsd = nfs server
zabbix = zabbix client (useless since its included in kong build)
pound = reverse proxy which you can use since you host multiple sites
sshhack = block ips hammering ssh with incorrect credentials.
stophack = BLock ips which are trying to hack server (only combined with pound or vlighttpd)
stophammer = block ips which are hammering ports
nzbget & transmission for downloading.
fixtables rearranges the firewall entries

A full explanation on how otrw2 enhances your router and what each package does is located here.

All in all, pretty straightforward once you get the right filesystem on your media and have it mounted on the right mountpoint. OTRW2 gives your router a whole lot more usefulness and the ability to install a wide range of packages on it.