I needed a quick way to determine my IP address for a script. If you run the ip addr show command it outputs a lot of information I don’t need. I settled on using grep, awk, and cut to get the information I want
ip addr show <interface name> |egrep 'inet '| awk '{print $2}'| cut -d '/' -f1
The result is a clean IP address. Beautiful. Thanks to this site for insight into how to use cut.
I wrote a simple little network mount script for Linux desktops. I wanted to replicate my Windows box as best as I could where a bunch of network drives are mapped upon user login. This script relies on having gvfs-mount and the cifs utilities installed (installed by default in Ubuntu.)
#!/bin/bash
#Simple script to mount network drives
#Specify network paths here, one per line
#use forward slash instead of backslash
FOLDER=(
server1/folder1
server1/folder2
server2/folder2/folder3
server3/
)
#Create a symlink to gvfs mounts in home directory
ln -s $XDG_RUNTIME_DIR/gvfs ~/Drive_Mounts
for mountpoint in "${FOLDER[@]}"
do
gvfs-mount smb://$mountpoint
done
Mark this script as executable and place it in /usr/local/bin. Then make it a default startup application for all users:
vim /etc/xdg/autostart/drive-mount.desktop
[Desktop Entry] Name=Mount Network Drives Type=Application Exec=/usr/local/bin/drive-mount.sh Terminal=false
Voila, now you’ve got your samba mount script starting up for every user.
The title isn’t very descriptive. I recently came across a need to script adding users & groups to the “RequireMembershipOf” directive of PowerBroker Open. PowerBroker is a handy tool that really facilitates joining a Linux machine to a Windows domain. It has a lot of configurable options but the one I was interested in was RequireMembershipOf – which as you might expect requires that the person signing into the Linux machine be a member of that list.
The problem with RequireMembershipOf is, as far as I can tell, it has no append function. It has an add function which frustratingly erases everything that was there before and includes only what you added onto the list. I needed a way to append a member to the already existing RequireMembershipOf list. My solution involves the usage of bash, sed, and a lot of regex. It boils down to two lines of code:
#take output of show require membership of, remove words multistring & local policy, replace spaces with carat (pbis space representation) and put results into variable (which automatically puts results onto a single line) add=$(/opt/pbis/bin/config --show RequireMembershipOf | sed 's/\(multistring\)\|\(local policy\)//g' | sed 's/ /^/g') #run RequireMembershipOf command with previous output and any added users sudo /opt/pbis/bin/config RequireMembershipOf "$add" "<USER_OR_GROUP_TO_ADD>"
That did the trick.
I recently acquired a shiny new set of SSDs to host my VMs. The problem is I needed to create a new ZFS array to accommodate them. I needed to figure out a way to migrate my VMs to the new array and then instruct Xenserver to use the new array instead of the old one.
Fortunately with a bit of research I learned this is fairly painless. Thanks to this discussion on citrix forums that got me pointed in the right direction. To change the server / IP address of an existing NFS storage repository in Xenserver you must do the following:
In my case since my VMs were on an existing ZFS volume with snapshots I wanted to preserve, I used ZFS send and receive to transfer data from my old array to my SSD array. Bonus: I was able to do this while the VMs were still running to ensure minimal downtime. My ZFS copy procedure was as follows:
In the above examples my source dataset was storage/VMs and the destination dataset was ssd/VMs.
Once the data was all transferred to the new location it was time to tell Xenserver about it. I had enough VMs that it was worth my time to write a little script to do it. It’s quick and dirty but it did the job. Behold:
#!/bin/bash
#Author: Nicholas Jeppson
#A simple script to change a xenserver NFS storage repository address to a new location
#Modify NFS_SERVER, NFS_PATH and/or NFS_VERSION to match your environment.
#Run this script on each xenserver host in your pool. Empty output means the transfer was successful.
#This script takes one argument - the name of the SR to be transferred.
SR_NAME="$1"
NFS_SERVER=10.0.0.1
NFS_PATH=/mnt/ssd/VMs/$SR_NAME
NFS_VERSION=4
#Use sed and awk to grab necessary UUIDs
HOST_UUID=$(xe host-list|egrep -B3 `hostname`$ | grep uuid | awk '{print $5}')
PBD_UUID=$(xe pbd-list|grep -A4 -B4 $SR_NAME | grep -B2 $HOST_UUID |grep -w '^uuid ( RO)' | awk '{print $5}')
SR_UUID=$(xe pbd-list|grep -A4 -B4 $SR_NAME | grep -A2 $HOST_UUID | grep 'sr-uuid' | awk '{print $4}')
#Unplug & destroy old NFS location, create new NFS location
xe pbd-unplug uuid=$PBD_UUID
xe pbd-destroy uuid=$PBD_UUID
NEW_PBD_UUID=$(xe pbd-create host-uuid=$HOST_UUID sr-uuid=$SR_UUID device-config-server=$NFS_SERVER device-config-serverpath=$NFS_PATH device-config-nfsversion=$NFS_VERSION)
xe pbd-plug uuid=$NEW_PBD_UUID
Download the script here (right click / save as)
You can run this script in a simple for loop with something like this:
for SR in <list of SR names separated by a space>; do bash <name of script saved from above> $SR; done
If you named the above script nfs-migrate.sh, and you had three SRs to change (blog1, blog2, blog3) then it would be:
for SR in blog1 blog2 blog3; do bash nfs-migrate.sh $SR; done
After I migrated the data and ran that script, my VMs booted up using the new SSD array. Success.
My new project recently has been to configure sonarr to work with transmission. The challenge was getting these two pieces of software to properly interface with each other. Sonarr would successfully instruct transmission to download the requested show but once the download completed it would not import the show to its library. The reason behind this was my torrent tracker – most torrents are downloaded as multi part rar files. Sonarr has no mechanism for processing rar files so I had to get creative.
The solution was to write a simple script and have transmission execute it after finishing the download. The script uses the find command to look for rar files in the directory transmission created for that particular torrent. If any rar files are found it will extract them into that same directory. This was important because sonarr will only look in the torrent download directory for the completed video file.
After some tweaking I got it to work pretty well for me. Here is the code I used (thanks to this site for the direction.)
#!/bin/bash
#A simple script to extract a rar file inside a directory downloaded by Transmission.
#It uses environment variables passed by the transmission client to find and extract any rar files from a downloaded torrent into the folder they were found in.
find /$TR_TORRENT_DIR/$TR_TORRENT_NAME -name "*.rar" -execdir unrar e -o- "{}" \;
Save the above script into a file your transmission client can read and make it executable. Lastly configure transmission to run this script on torrent completion by modifying your settings.json file (mine was located at /var/lib/transmission/.config/transmission-daemon/settings.json) Modify the following variables (be sure to stop your transmission client first before making any changes.)
"script-torrent-done-enabled": true, "script-torrent-done-filename": "/path/to/where/you/saved/the/script",
That’s it! Sonarr will now properly import shows that were downloaded via multipart rar torrent.
I had a website where I was curious what the top 10 URLs that were returning 404s were along with how many hits those URLs got. This was after a huge site redesign so I was curious what old links were still trying to be accessed.
Getting a report on this can be accomplished with nothing more than the Linux command line and the log file you’re interested in. It involves combining grep, sed, awk, sort, uniq, and head commands. I enjoyed how well these tools work together so I thought I’d share. Thanks to this site for giving me the inspiration to do this.
This is the command I used to get the information I wanted:
grep '404' _log_file_ | sed 's/, /,/g' | awk {'print $7'} | sort | uniq -c | sort -n -r | head -10
Here is a rundown of each command and why it was used:
This was my output – exactly what I was looking for. Perfect.
2186 http://<sitename>/source/quicken/index.ini 2171 http://<sitename>/img/_sig.png 1947 http://<sitename>/img/email/email1.aspx 1133 http://<sitename>/source/quicken/index.ini 830 http://<sitename>/img/_sig1.png 709 https://<sitename>/img/email/email1.aspx 370 http://<sitename>/apple-touch-icon.png 204 http://<sitename>/apple-touch-icon-precomposed.png 193 http://<sitename>/About-/Plan.aspx 191 http://<sitename>/Contact-Us.aspx
I wrote up a little script to run when you migrate a wordpress installation from one host to another (hostname change.) Once this script is run you can access the site via the hostname of the server it’s running on and then change the URL to whatever you like. This comes in handy for when you want to migrate one internal host to another, then specify an external hostname once things are looking how you like them.
Change SQL_COMMAND to reflect the name of the wordpress table in the destination server. Thanks to this site for the guidance in writing the script.
#!/bin/bash #A simple script to update the wordpress database to reflect a change in hostname #Run this after changing the hostname / IP of a wordpress server #Prompt for mysql root password read -s -p "Enter mysql root password: " SQL_PASSWORD SQL_COMMAND="mysql -u root -p$SQL_PASSWORD wordpress -e" #Determine what the old URL was and save to variable OLD_URL=$(mysql -u root -p$SQL_PASSWORD wordpress -e 'select option_value from wp_options where option_id = 1;' | grep http) #Get current hostname HOST=$(hostname) #SQL statements to update database to new hostname $SQL_COMMAND "UPDATE wp_options SET option_value = replace(option_value, '$OLD_URL', 'http://$HOST') WHERE option_name = 'home' OR option_name = 'siteurl';" $SQL_COMMAND "UPDATE wp_posts SET guid = replace(guid, '$OLD_URL','http://$HOST');" $SQL_COMMAND "UPDATE wp_posts SET post_content = replace(post_content, '$OLD_URL', 'http://$HOST');" $SQL_COMMAND "UPDATE wp_postmeta SET meta_value = replace(meta_value,'$OLD_URL','http://$HOST');"
I wrote a simple script to check to see if a specific mountpoint on a Linux system is still live. It does this by trying to read a specific file on the share, and if it cannot, write the event to a log, unmount, and then re-mount the folder. The need arose for instances where a file server has been rebooted and the linux system loses the connection to the share. This way it will automatically re-mount.
Modify the variables section as needed and then have a cron job run the script as root at whatever interval you want. Enjoy.
#!/bin/bash
#Script to monitor mount directories to ensure they are properly mounted
#Place a file with the word "mounted" in it inside all mounted directories
#The script will try to read the file and attempt to unmount and remount the folder if it fails to read the file
#Updated 8/30/2016 by Nicholas Jeppson
#---------Variable section------------#
#Place mount folder locations here, separated by space
#Paths containing spaces need to have quotes around them
LOCATIONS=(/home/njeppson /home/njeppson/Desktop)
#Name of file to try to read
TEST_FILENAME="mountcheck"
#---------End Variable Section--------#
#-----Do not edit below this line-----#
#Read file, if contents don't contain "mounted" then attempt to unmount and re-mount the folder, output attempt to /var/log/mountcheck
for FOLDER in "${LOCATIONS[@]}"; do
if [[ $(cat $FOLDER/$TEST_FILENAME) != "mounted" ]]; then
echo "$(date "+%b %d %T") $(hostname) $FOLDER Not mounted, remounting." >> /var/log/mountcheck
umount $FOLDER
mount $FOLDER
fi
done
I’ve recently had to do some bash-fu and thought I’d document it in case I come across the need again. It involved an if statement inside a for loop. The if statement looked at the result of an external command and acted if conditions were met.
The scenario: An application created folders beginning with a series of digits. Later it was decided to add a prefix to new folders. A problem occurred where there were folders with the same numeric sequence – corresponding to the same user – but the program was saving things in both prefixed and non-prefixed folders at random. We needed a way to copy information from the numeric only folders into the prefix folders, then backup and delete the numeric-only folders. We also wanted to be warned about any file overwrites in the process.
After a bunch of research and experimentation I came up with the following one-line bash script:
for d in [0-9]*; do BN=$(basename "$d"); if [[ $(find . -maxdepth 1 -type d -name "*$d" | grep -o $d | wc -l) = 2 ]]; then cp -i -p -r "$d" ../archive/"$d"; cp -i -p -r "$d"/* "PREFIX_$BN"; rm -rf "$d"; fi; done
It does the following:
The big learning moment for me was embedding a bash command into an if statement. The if statement runs the find command, pipes to wc -l to count the number of results, and then compares that result to something else. Pretty handy.
Thanks to these sites for helping me in my journey:
If statement inside for loop: https://unix.stackexchange.com/questions/52800/how-to-do-an-if-statement-from-the-result-of-a-executed-command
Find results only in current directory: https://unix.stackexchange.com/questions/162411/find-maxdepth-0-not-returning-me-any-output
Count results from find command: https://stackoverflow.com/questions/6181324/counting-regex-pattern-matches-in-one-line-using-sed-or-grep
Warn before overwriting files: https://askubuntu.com/questions/236478/how-do-i-make-bash-warn-me-when-overwriting-an-existing-file
I have a Citrix xenserver 6.5 host which hosts a FreeNAS VM that exports an NFS share. I then have that same xenserver host use that NFS export as a SR for other VMs on that same server. It’s unusual, but it saves me from buying a separate server for VM storage.
The problem is if you reboot the hypervisor it will fail to connect to the NFS export (because the VM hosting it hasn’t booted yet.) Additionally it appears Xenserver does not play well at all with hung NFS mounts. If you try to shutdown or reboot your FreeNAS VM while Xenserver is still using its NFS export, things start to freeze. You will be unable to do anything to any of your VMs thanks to the hung NFS share. It’s a problem!
My hack around this mess is to have FreeNAS, not Xenserver, control starting and stopping these VMs.
First, create public/private key pair for ssh into xenserver
ssh-keygen
This will generate two files, a private key file and a public (.pub) file. Copy the contents of the .pub file into the xenserver’s authorized_keys file:
echo "PUT_RSA_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
Copy the private key file (same name but without .pub extension) somewhere on your FreeNAS VM.
Next, create NFS startup and shutdown scripts. Thanks to linuxcommando for some guidance with this. Replace the -i argument with the path to your SSH private key file generated earlier. You will also need to know the PBD UUID of the NFS store. Discover this by issuing
xe pbd-list
Copy the UUID for use in the scripts.
vi nfs-startup.sh
#!/bin/bash #NFS FreeNAS VM startup script SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>" #Attach NFS drive first, then start up NFS-reliant VMs $SSH_COMMAND xe pbd-plug uuid=<UUID_COPIED_FROM_ABOVE> sleep 10 #Issue startup commands for each of your NFS-based VMs, repeat for each VM you have $SSH_COMMAND xe vm-start vm="VM_NAME" ...
vi nfs-shutdown.sh
#!/bin/bash #NFS FreeNAS VM shutdown script #Shut down NFS-reliant VMs, detach NFS SR #Re-establish networking to work around the fact that Network goes down before this script is executed within FreeNAS /sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % up SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>" #Issue shutdown commands for each of your VMs $SSH_COMMAND xe vm-shutdown vm="VM_NAME" sleep 60 $SSH_COMMAND xe pbd-unplug <UUID_OF_NFS_SR> #Take the networking interfaces back down for shutdown /sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % down
Don’t forget to mark them executable:
chmod +x nfs-startup.sh chmod +x nfs-shutdown.sh
Now add the scripts as a startup task in FreeNAS and shutdown task respectively by going to System / Init/Shutdown Scripts. For startup, Select Type: Script, Type: postinit and point it to your nfs-startup.sh script. For shutdown, select Type: Script and Type: Shutdown.
Success! Now whenever your FreeNAS VM is shut down or rebooted, things will be handled properly which will prevent your hypervisor from freezing.