Category Archives: CLI

CLI

The power of find, grep, and xargs

Leave a comment

Recently I needed to find folders with two different things in the path – mysql and DB. I toyed around with a bunch of options but finally settled on using xargs. I don’t use it much. I should use it more.

The command below takes output from find, greps it twice (thus looking for things that have both terms in them) and then creates a symbolic link of the results.

 find . -type d | grep mysql | grep DB | xargs -n 1 ln -s

This accomplished what I needed quite well. In a huge stash of folders there were a subset that contained both the words mysql and DB in their paths that  I was interested in. Find, grep, and xargs to the rescue.

CLI

Grep handy tips

Leave a comment

I found some grep flags that are really handy and wanted to write them down.

Only list exact matches:  -w

When I run the “netstat -an | grep LISTEN” command it would include all text that included LISTEN – including LISTENING, which I didn’t want to see. Appending -w to the netstat command makes sure grep only displays exact matches.

netstat -an | grep -w LISTEN

Include extra lines above and below the match: -A, -B

When administering xenserver systems it’s often useless to use grep defaults because Xen likes to include relevant information on different lines. To fix this, use the -A and/or the -B flags to specify the number of lines after (A) or before (B) to include in the results. A real world example using grep to return the 3 lines above and below the line matching the word Splunk:

xe vdi-list | grep -A3 -B3 Splunk

 

CLI

Embed commands in if statements in bash

Leave a comment

I’ve recently had to do some bash-fu and thought I’d document it in case I come across the need again. It involved an if statement inside a for loop. The if statement looked at the result of an external command and acted if conditions were met.

The scenario: An application created folders beginning with a series of digits.  Later it was decided to add a prefix to new folders. A problem occurred where there were folders with the same numeric sequence – corresponding to the same user – but the program was saving things in both prefixed and non-prefixed folders at random. We needed a way to copy information from the numeric only folders into the prefix folders, then backup and delete the numeric-only folders. We also wanted to be warned about any file overwrites in the process.

After a bunch of research and experimentation I came up with the following one-line bash script:

for d in [0-9]*; do BN=$(basename "$d"); if [[ $(find . -maxdepth 1 -type d -name "*$d" | grep -o $d | wc -l) = 2 ]]; then  cp -i -p -r "$d" ../archive/"$d"; cp -i -p -r "$d"/* "PREFIX_$BN"; rm -rf "$d"; fi; done

It does the following:

  • Scan the current directory for files (or folders) beginning with numbers
  • Save the basename of discovered file to a variable (basename was required to remove the ./ that showed up in the results) and use that variable for the copy command
  • Scans the current directory to see if there is another folder with the same string of numbers in its name (same name but only with a prefix attached)
  • If there is a folder with the same string of numbers in its name, copy the non-prefixed folder to an archive location, then copy its contents to the folder with the prefix, prompting before overwriting anything.
  • Once the copy is complete, delete the original non-prefixed folder

The big learning moment for me was embedding a bash command into an if statement. The if statement runs the find command, pipes to wc -l to count the number of results, and then compares that result to something else. Pretty handy.

Thanks to these sites for helping me in my journey:

If statement inside for loop: https://unix.stackexchange.com/questions/52800/how-to-do-an-if-statement-from-the-result-of-a-executed-command

Find results only in current directory:  https://unix.stackexchange.com/questions/162411/find-maxdepth-0-not-returning-me-any-output

Count results from find command: https://stackoverflow.com/questions/6181324/counting-regex-pattern-matches-in-one-line-using-sed-or-grep

Warn before overwriting files: https://askubuntu.com/questions/236478/how-do-i-make-bash-warn-me-when-overwriting-an-existing-file

CLI

Add prefix to filenames in bash

Leave a comment

A quick handy little way to add a prefix to files in bash (taken from here)

for f in * ; do mv "$f" "PRE_$f" ; done

In my case I wanted to rename all sub-100 filenames to have an extra zero so sorting played nicely with filenames beginning with 100+. To accomplish this I found about the rename command (thanks to this site.)  The command I used to enforce natural sorting was the following:

rename 's/\d+/sprintf("%03d", $&)/e' *

The command looked for anything beginning with a number, then used sprintf to make the number 3 digits. The asterisk instructed the rename command to work on every file. Success.

 

CLI, OS

Mount folder from another system over SSH

Leave a comment

I recently had a need to mount a folder over SSH to allow my file manager to browse through the files on a remote system. Two great resources led me to the solution to this problem: sshfs

I first came across this little tutorial on how to install sshfs on my shiny new Linux Mint 18 box:

sudo apt-get install sshfs
sudo mkdir /mnt/droplet #<--replace "droplet" with whatever you prefer
sudo sshfs -o allow_other,defer_permissions root@xxx.xxx.xxx.xxx:/ /mnt/droplet

Pretty slick. If you want to use a keyfile instead of being prompted for a password, you can use the IdentityFile option:

sudo sshfs -o allow_other,defer_permissions,IdentityFile=~/.ssh/id_rsa root@xxx.xxx.xxx.xxx:/ /mnt/droplet

You can have this handled in /etc/fstab for automounting. Thanks to this Arch Linux guide for the info. (The command below requires systemd.)

user@host:/remote/folder /mount/point  fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/user/.ssh/id_rsa,allow_other,reconnect 0 0

I tweaked my /etc/fstab file a bit because it complained that allow_other required a configuration change. Since I’m the only user of this box it didn’t matter to me. Here is my configuration:

nicholas@remote:/ /home/desktop/remote fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/desktop/.ssh/keyfile,reconnect,allow_other 0 0

I’m mounting the root folder of my remote machine into a folder named remote on my desktop machine. I generated ssh keyfiles so that no password was required. Now the mount shows up under “Devices” in my file manager and a simple click mounts the folder gets me there. Sweet.

Edit 2/25/18: Added allow_other option per this article

 

CLI, Virtualization

Transfer VMs between Xenserver pools

8 Comments

When Xenserver 7 came out I found myself unable to easily upgrade to it thanks to my custom RAID 1 build.  If I wanted Xenserver 7 I would have to blow the whole instance away and start from scratch. This posed a problem because I have a pool of 2 xenserver hosts. You cannot add a server with a higher xenserver version to a lower versioned pool; the pool master must always have the highest version of Xenserver installed. My decision to have an mdadm RAID 1 setup on my pool master ultimately turned into forced VM downtime for an upgrade despite having a pool of other xenserver hosts.

After transferring VMs to my secondary host and promoting it to pool master, I wiped my primary xenserver and installed 7. When it was up and running I essentially had two separate pools running. To transfer my VMs back to my primary server I had to resort to the command line.

Offline VM transfer

The xe vm-export and vm-import commands work with stdin/out and piping. This is how I accomplished transferring my VMs directly between two pools. Simply pipe xe vm-export commands with an ssh xe vm-import command like so:

xe vm-export uuid=<VM_UUID> filename= | ssh <other_server> xe vm-import filename=/dev/stdin

Note the lack of a filename – this instructs xenserver to pipe to standard output instead. Also note that transferring the VM scrambles the MAC addresses of its interfaces. If you want to keep the MAC address you’ll have to manually re-assign it after the copy is complete.

Minimal downtime

For the method above you will have to turn the VM off in order to transfer it.  I had some VMs that I didn’t want to stay down for the entire transfer. A way around this is to take a snapshot of the VM and then copy the snapshot to the other pool. Note that this method does not retain any changes made inside the VM that occurred after you took the snapshot. You will have to manually transfer any file changes that took place during the VM transfer (or be fine with losing them.)

In order to export a snapshot you must first convert it to a VM from a template (thanks to this site for outlining how.) The full procedure is as follows:

  1. Take a snapshot of the VM you want to move 
    xe vm-snapshot uuid=<VM_UUID> new-name-label=<snapshotname>
  2. Convert the snashot template to a VM (the command xe snapshot-list is a handy way to obtain UUIDs of your snapshots) 
    xe template-param-set is-a-template=false ha-always-run=false uuid=<UUID of snapshot>
  3. Transfer the template to the new pool 
    xe vm-export uuid=<UUID of snapshot> filename= | ssh <other_server> xe vm-import filename=/dev/stdin
  4. Rename VM and/or modify interface MAC addresses as needed on the new host. Stop the VM on the old host and start it on the new one.

I used both methods above to successfully move my VMs from my older 6.5 pool to the newer 7 pool. Success.

CLI, Virtualization

PCI Passthrough in Xenserver 7 “Dundee”

20 Comments

I’ve recently upgraded to the latest version of Citrix Xenserver 7 (codenamed “Dundee”.) 7 is based on CentOS 7 and has a massive amount of changes under the hood. One such change was how they handle PCI Passthrough.

It took some time to figure PCI Passthrough out. 7 uses grub instead of extlinux for the bootloader. It appears to be grub2 but they don’t use the standard update-grub tool, rather you simply edit the config file and do nothing else.

After much searching I found this post which led me in the right direction. In Xenserver 7, for pci passthrough support you must do the following:

  • Prepare the VM for PCI passthrough (this part hasn’t changed) 
    xe vm-param-set other-config:pci=0/0000:B:D.f uuid=<vm uuid>
  • Modify /boot/grub/grub.cfg and append the following to the end of the module2 line (if you boot from EFI the file to modify is /boot/efi/EFI/xenserver/grub.cfg) 
     xen-pciback.hide=(B:D.f)
  • Reboot

You will now be able to pass through hardware to your virtual machines in Xenserver 7. Hooray.

CLI

Use Irfanview in Linux with Wine

Leave a comment

IrfanView is a very versatile little program that you can use to manipulate and view image files. I am very familiar with it and often find myself wishing I could use it in Linux. Well, it turns out I can!

There are a few things you need to do. First, install wine

#For debian-based distros
sudo apt-get install wine

After wine is installed, use it to install irfanview

wine iview442_setup.exe

If you get the following error, it means you don’t have mfc42.dll installed:

err:module:import_dll Library MFC42.DLL (which is needed by L"Z:\\home\\nicholas\\Downloads\\iview442_setup.exe") not foun

The solution is to use winetricks to install mcf42.dll (thanks to winehq for the soultion to this)

winetricks -q mfc42

Optionally you can make wine your default image program (or just have it in the list of available image handling programs.) The way to do this is to write a quick script and mark it executable (thanks to linuxquestions for the answer on how to do this)

#!/bin/sh
IRFANVIEW="C:\\Program Files\Irfanview (x86)\i_view32.exe"
ROOT_DRIVE="Z:\\"
for arg
do
	wine "$IRFANVIEW" "${ROOT_DRIVE}$(echo "$arg" | sed 's/\//\\/g')"
done

You may need to tweak it a little bit if you don’t have 64bit wine installed.  Save that file somewhere you will remember and then mark it executable with chmod +x.

Lastly, when you right click a file and do open with, simply point it to the above script. The result: wine goodness.