Tag Archives: find

The power of find, grep, and xargs

Recently I needed to find folders with two different things in the path – mysql and DB. I toyed around with a bunch of options but finally settled on using xargs. I don’t use it much. I should use it more.

The command below takes output from find, greps it twice (thus looking for things that have both terms in them) and then creates a symbolic link of the results.

 find . -type d | grep mysql | grep DB | xargs -n 1 ln -s

This accomplished what I needed quite well. In a huge stash of folders there were a subset that contained both the words mysql and DB in their paths that  I was interested in. Find, grep, and xargs to the rescue.

Embed commands in if statements in bash

I’ve recently had to do some bash-fu and thought I’d document it in case I come across the need again. It involved an if statement inside a for loop. The if statement looked at the result of an external command and acted if conditions were met.

The scenario: An application created folders beginning with a series of digits.  Later it was decided to add a prefix to new folders. A problem occurred where there were folders with the same numeric sequence – corresponding to the same user – but the program was saving things in both prefixed and non-prefixed folders at random. We needed a way to copy information from the numeric only folders into the prefix folders, then backup and delete the numeric-only folders. We also wanted to be warned about any file overwrites in the process.

After a bunch of research and experimentation I came up with the following one-line bash script:

for d in [0-9]*; do BN=$(basename "$d"); if [[ $(find . -maxdepth 1 -type d -name "*$d" | grep -o $d | wc -l) = 2 ]]; then  cp -i -p -r "$d" ../archive/"$d"; cp -i -p -r "$d"/* "PREFIX_$BN"; rm -rf "$d"; fi; done

It does the following:

  • Scan the current directory for files (or folders) beginning with numbers
  • Save the basename of discovered file to a variable (basename was required to remove the ./ that showed up in the results) and use that variable for the copy command
  • Scans the current directory to see if there is another folder with the same string of numbers in its name (same name but only with a prefix attached)
  • If there is a folder with the same string of numbers in its name, copy the non-prefixed folder to an archive location, then copy its contents to the folder with the prefix, prompting before overwriting anything.
  • Once the copy is complete, delete the original non-prefixed folder

The big learning moment for me was embedding a bash command into an if statement. The if statement runs the find command, pipes to wc -l to count the number of results, and then compares that result to something else. Pretty handy.

Thanks to these sites for helping me in my journey:

If statement inside for loop: https://unix.stackexchange.com/questions/52800/how-to-do-an-if-statement-from-the-result-of-a-executed-command

Find results only in current directory:  https://unix.stackexchange.com/questions/162411/find-maxdepth-0-not-returning-me-any-output

Count results from find command: https://stackoverflow.com/questions/6181324/counting-regex-pattern-matches-in-one-line-using-sed-or-grep

Warn before overwriting files: https://askubuntu.com/questions/236478/how-do-i-make-bash-warn-me-when-overwriting-an-existing-file

Exclude directories in find command

For some reason I had a hell of a time finding a way of excluding a directory from the ‘find’ command. I finally found a solution that works thanks to this website.

The solution for me was to add the following to my find command:

! -path "<directory to exclude>"

So to find all files modified less than 24 hours ago on my grandmother-in-law’s computer, I used the following command:

find . -mtime -1 ! -path "./AppData/*"

Grandma’s computer runs Windows 7 but I’ve configured Cygwin and ssh so I can use the tools I’m fimilar with. The above command excludes appdata and searches her user profile for all files modified today.

Install multiple xenserver patches at once

I came across a need to install multiple patches manually (via SSH) on one of my xenservers. It’s quite tedious to do this manually so I found a way to here.)

Download all the patch .zip files to a directory your xenserver can access. Then, extract them all with this command:

find *.zip -exec unzip {} \;

Next, upload all the .xsupdates:

find *.xsupdate -exec xe patch-upload file-name={} \;

This spits out a bunch of UUIDs. Make note of these. You will also need to get your host-uuid by using the

xe host-list

command.

Lastly, a quick for loop applies the patches we want (replace the UUIDs with those of the patches uploaded earlier and the host-uuid with yours)

for file in c3520494-be00-4133-afb3-adf8ab5edb11 7fea2d85-7ce1-428c-a92f-57e37551d6f1 d9862b7f-9be6-4672-b9a8-4f52f776fd03 a424dfe5-8be8-4bd6-a49e-62620e369a43 e28bb0ae-e43f-46d9-9147-c7dc712508eb; do xe patch-apply uuid=$file host-uuid=46f8ef28-8ee1-44b5-967c-b8e48585094b; done

That did the trick for me. After applying the patches I came across this post which appears to have a much better script. Whatever works.

Use FFMPEG to batch convert video files

Below are the tips and tricks I’ve learned in my quest to get all my family home movies working properly in Plex. I ended up having to re-encode most of them to have a proper codec.

ffmpeg is what I ended up using. Thanks to here and here I found a quality setting that I liked: x264 video with crf of 18 and veryslow encoder preset, AAC audio at 192kbps, mp4 container.

ffmpeg -i <file> -acodec libfdk_aac -b:a 192k -ac 2 -vcodec libx264 -crf 18 -preset veryslow <filename>.mp4

I tried to use exiftools to add metadata to the file but Plex wouldn’t read it. Instead I discovered you can use ffmpeg to encode things like title and date taken directly into the file. The syntax for ffmpeg is

-metadata "key=value"

The metadata values I ended up using were:

  • date – Plex looks at this
  • Title – what’s displayed in plex
  • comment – not seen by plex but handy to know about anyway

Encoding

My first pass at encoding combined find with ffmpeg to re-encode all my avi files. I used this command:

find . -name *.avi -exec ffmpeg -i {} -acodec libfdk_aac -b:a 192k -ac 2 -vcodec libx264 -crf 18 -preset veryslow {}.mp4 \;

Interrupted

I bit off more than I could chew with the above command. I had to kill the process because it was chewing up too much CPU. There are two ways to figure out which file ffmpeg is on:

ps aux|grep ffmpeg

will list the file that ffmpeg is currently working on. Take note of this because you’ll want to remove it so ffmpeg will re-encode it.

You can also run this command to see the last 10 most recently modified files, taken from here:

find $1 -type f -exec stat --format '%Y :%y %n' "{}" \; | sort -nr | cut -d: -f2- | head

Resume

Use the above command to remove what ffmpeg was working on when killed. Append -n to tell ffmpeg to exit and not overwrite existing files

find . -name *.avi -exec ffmpeg -n -i {} -acodec libfdk_aac -b:a 192k -ac 2 -vcodec libx264 -crf 18 -preset veryslow {}.mp4 \;

Cleanup

The command above gave me a bunch of .avi.mp4 files. To clean them up to just be .mp4 files combine find with some bash-fu (thanks to this site for the info)

find . -name “*.avi.mp4” -exec sh -c ‘mv “$1” “${1%.avi.mp4}.mp4″‘ _ {} \;

Finally, remove original .avi files:

find . -name "*.avi" -exec rm {} \;

 

Update 2/3/2016: Added a verification step.

Verification

Well, it looks like the conversion I did was not without problems. I only detected them when I tried to change the audio codec I used. You can use ffmpeg to analyze your file to see if there are any errors with it. I learned this thanks to this thread.

The command is:

ffmpeg -v error -i <file to check> -f null -

This will have ffmpeg go through each frame and log and tell you any errors it finds. You can have it dump them to a logfile instead by redirecting the output like so:

ffmpeg -v error -i <file to scan> -f null - >error.log 2>&1

What if you want to check the integrity of multiple files? Use find:

find . -type f -exec sh -c 'ffmpeg -v error -i "{}" -f null - > "{}".log 2>&1' \;

The above command searches for any file in the current directory and subdirectories, uses ffmpeg to analyze the file, and will output what it finds to a log file of the same name as the file that was analyzed. If nothing was found, a 0 byte file was created. You can use find to remove all 0 byte files so all that’s left are logs of files with errors:

find . -type f -size 0 -exec rm {} \;

You can open each .log file to see what was wrong, or do what I did and see that the presence of the log file at all means that the file needs to be re-encoded from source.

Checking the integrity of the generated files themselves made me realize I had to re-encode part of my home movie library. After re-encoding, there were no more errors. Peace of mind. Hooray!

Randomize files in a folder

I wanted to make a simple slideshow for a cheap Kindle turned photo frame in my office. Windows Movie Maker (free, already installed program) does not have a randomize function when importing photos. I had a lot of photos I wanted imported and I wanted them randomized. Movie Maker doesn’t include subfolders for some dumb reason, so I also needed a way to grab pictures from various directories and put them in a single directory.

My solution (not movie-maker specific) was to use bash combined with find, ln, and mv to get the files exactly how I want them. The process goes as follows:

  1. Create a temporary folder
  2. Use the find command to find files you want
    1. Use -type f argument to find only files (don’t replicate directory structure)
    2. Use the -exec argument to call the ln command to create links to each file found
    3. Use the -s argument of ln to create symbolic links
    4. Use the -b argument of ln to ensure duplicate filenames are not overwritten
  3. Invoke a one line bash command to randomize the filenames of those symbolic links

It worked beautifully. The commands I ended up using were as follows:

mkdir temp
cd temp
find /Pictures/2013/ -type f -exec ln -s -b {} . \;
#repeat for each subfolder as needed, unless you want all folders in which case you can just specify the directory beneath it.
find /Pictures/2014/ -type f -exec ln -s -b {} . \;
find /Pictures/2015/ -type f -exec ln -s -b {} . \;

for i in *.JPG; do mv "$i" "$RANDOM.jpg"; done
#repeat for all permutations. The -b argument of ln creates files with tildes in the extension - don't forget about them.
for i in *.jpg; do mv "$i" "$RANDOM.jpg"; done
for i in *.JPG~; do mv "$i" "$RANDOM.jpg"; done
for i in *.jpg~; do mv "$i" "$RANDOM.jpg"; done

The end result was a directory full of pictures with random filenames, ready to be dropped into any crappy slideshow making software of your choosing 🙂

Upgrade Linux Mint 16 to 17.1

I realized recently that my desktop system is quite out of date. It has worked so well for so long that I didn’t realize for a while that it was end of support. I was running Linux Mint 16 – Petra.

Thanks to this site the upgrade was fairly painless – a few repository updates, upgrade, and reboot. Simple! The steps I took are below

Update all repositories

Use sed in conjunction with find to quickly and easily update all your repository files from saucy to trusty, and from petra to rebecca, making a backup of files modified.

sudo find /etc/apt/sources.list.d/ -type f -exec sed --in-place=.bak 's/saucy/trusty/' {} \;
sudo find /etc/apt/sources.list.d/ -type f -exec sed --in-place=.bak 's/petra/rebecca/' {} \;

Update your system

This took a while. It had to download 1.5GB of data and install it.

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get upgrade

Cleanup

After running the upgrade I had a notice that I had many packages that were installed but no longer required. To remove unnecessary packages after the upgrade:

sudo apt-get autoremove

Install new language settings:

sudo apt-get install mintlocale

Install gvfs-backends (for Thunar)

sudo apt-get install gvfs-backends

Reboot

Flawless! It worked on the first try. Awesome.

Unzip multiple files into a single directory

Occasionally I have a need to unzip multiple zip files into a single directory, renaming any files with duplicate names so all zip contents end up in the same directory. I accomplish this in a lazy fashion with find and unzip.

First, put all the zip files you need to extract in the same folder. I used find with the -ctime flag to find files created today (as those are the ones I want.) I use the -exec flag to execute a command on the resulting files; in this case, the unzip program with the -B flag, which doesn’t overwrite files with duplicate names, and the -d flag to specify which folder to extract to:

find *.zip -ctime 1 unzip -B {} -d example/ \;

This finds and unzips all my files, but there is a catch: files with the same filename have been renamed with tildes at the end of each file, for example pic1.jpg~. I do another quick find operation to simply tack .jpg to the end of each of these files

find example/ -name "*~*" -exec mv {} {}.jpg \;

The result is a directory full of the files you desire. My case is very simplistic as it assumes that all files in each zip file are of the same extension. You could use something like awk to parse the result of find command and re-add appropriate extensions, but I won’t detail that here (see laziness reference above.)

 

Xenserver – The uploaded patch file is invalid

It has been six months since I’ve applied any patches to my Citrix Xenserver hypervisor. Shame on me for not checking for updates. The thing has been humming along without any issues so it was easy to forget about.

In trying to install xenserver patches today I kept getting this error message no matter what I tried:

The uploaded patch file is invalid

After deleting everything I could (including files hanging out in /var/patch) I realized that I was simply Doing It Wrong™. D’oh!

When applying xenserver updates, the expected file extension is .xsupdate. I had been trying to xe patch-upload the downloaded zip file, whereas I was supposed to have extracted those zips before trying to upload them.  This quick little line unzipped all my patch ZIP files for me in one swoop:

find *.zip -exec unzip {} \;

Once everything was unzipped I was able to upload and apply the resulting .xsupdate files without issue.