Monthly Archives: October 2014

CLI

Find out free disk space from the command line

Leave a comment

The du command can be used in any Unix shell to determine how much space a folder is taking. It’s a quick way to determine which directory is using the most space.

If you run du on the root drive “/” then you will get an idea of how much space the drive is using. One unfortunate side effect of that command is if you have any mounted drives or other filesystems, it will search the disk usage of those folders as well. Aside from taking a long time that method provides data we don’t necessarily want.

Fortunately, there are a few switches you can use to fix this problem. The -x switch tells du to ignore other filesystems. Perfect.

There are a few other switches that prove useful. Below is a list of my favorites:

  • -x Ignore other filesystems
  • -h Use human readable numbers (kilobytes, megabytes, and gigabytes) instead of raw bytes
  • -s Provide a summary of the folder instead of listing the size of each file inside the folder
  • * (not a switch, but rather an argument to put after your directory.) This summarizes subfolders in that directory, as opposed to simply returning the size of that directory.

For troubleshooting low disk space errors, the following command will give you a good place to start:

du -hsx /*

You can then dig further by altering the above command to reflect a directory instead of /, or simply do * if you want a breakdown of the directory you are currently in.

CLI

Compare two latest ZFS snapshots for differences

Leave a comment

In my previous post about ZFS snapshots I discussed how to get the latest snapshot name. I came across a need to get the name of the second to last snapshot and then compare that with the latest. A little CLI kung-fu is required for this but nothing too scary.

The command of the day is: zfs diff.

zfs diff storage/mythTV@auto-20141007.1248-2h storage/mythTV@auto-20141007.1323-2h

If you get an error using zfs diff, you aren’t running as root. You will need to delegate the diff ZFS permission to the account you’re using:

zfs allow backup diff storage

where backup is the account you want to grant permissions for and storage is the dataset you want to grant permissions to.

The next step is to grab the two latest snapshots using the following commands.

Obtain latest snapshot:

zfs list -t snapshot -o name -s creation -r storage/Documents | tail -1

Obtain the second to latest snapshot:

zfs list -t snapshot -o name -s creation -r storage/Documents | tail -2 | sort -r | tail -1

Putting it together in one line:

zfs diff `zfs list -t snapshot -o name -s creation -r storage/Documents | tail -2 | sort -r | tail -1` `zfs list -t snapshot -o name -s creation -r storage/Documents | tail -1`

While doing some testing I came across an unfortunate bug with the ZFS diff function. Sometimes it won’t show files that have been deleted! It indicates that the folder where the deleted files were in was modified but doesn’t specify any further. This bug appears to affect all ZFS implementations per here and here. As of this writing there has been no traction on this bug. The frustrating part is the bug is over two years old.

The workaround for this regrettable bug is to use rsync  with the -n parameter to compare snapshots. -n indicates to only do a dry run and not actually try to copy anything.

To use Rsync for comparison, you have to do a little more CLI-fu to massage the output from the zfs list command so it’s acceptable to rsync as well as include the full mountpoint of both snapshots. When working with rsync, don’t forget the trailing slash.

rsync -vahn --delete /mnt/storage/Documents/.zfs/snapshot/`zfs list -t snapshot -o name -s creation -r storage/Documents | tail -2 | sort -r | tail -1 | sed 's/.*@//g'`/ /mnt/storage/Documents/.zfs/snapshot/`zfs list -t snapshot -o name -s creation -r storage/Documents | tail -1 | sed 's/.*@//g'`/

Command breakdown:

Rsync arguments:
-v means verbose (lists files added/deleted)
-a means archive (preserve permissions)
-h means human readable numbers
-n means do a dry run only (no writing)
–delete will delete anything in the destination that’s not in the source (but not really since we’re doing -n – it will just print what it would delete on the screen)

Sed arguments
/s search and replace
/.*@ simple regex meaning anything up to and including the @ sign
/  What comes after this slash is what we would like to replace what was matched in the previous command. In this case, we choose nothing, and move directly to the last argument
/g tells sed to keep looking for other matches (not really necessary if we know there is only one in the stream)

All these backticks are pretty ugly, so for readability sake, save those commands into variables instead. The following is how you would do it in bash:

FIRST_SNAPSHOT="`zfs list -t snapshot -o name -s creation -r storage/Documents | tail -2 | sort -r | tail -1 | sed 's/.*@//g'/`"
SECOND_SNAPSHOT="`zfs list -t snapshot -o name -s creation -r storage/Documents | tail -1 | sed 's/.*@//g'/`"
rsync -vahn --delete /mnt/storage/Documents/.zfs/snapshot/$FIRST_SNAPSHOT /mnt/storage/Documents/.zfs/snapshot/$SECOND_SNAPSHOT

I think I’ll stop for now.

CLI

Blow away ZFS snapshots and watch the progress

Leave a comment

For the last month I have had a testing system (FreeNAS) take ZFS snapshots of sample datasets every five minutes. As you can imagine, the snapshot count has risen quite dramatically. I am currently at over 12,000 snapshots.

In testing a backup script I’m working on I’ve discovered that replicating 12,000 snapshots takes a while. The initial data transfer completes in a reasonable time frame but copying each subsequent snapshot takes more time than the original data. Consequently, I decided to blow away all my snapshots. It took a while! I devised this fun little way to watch the progress.

Open two terminal windows. In terminal #1, enter the following:

bash
while [ true ]; do zfs list -H -t snapshot | wc -l; sleep 6; done

The above loads BASH and the runs a simple loop to count the total number of snapshots on the system. The sleep command is only there because it takes a few seconds to return the results when you have more than 10,000 snapshots.

Alternatively you could make the output a little prettier by entering the following:

while [ true ]; do REMAINING="`zfs list -H -t snapshot | wc -l`"; echo "Snapshots remaining: $REMAINING" ; sleep 6; done

In terminal #2, enter the following (taken from here):

bash
for snapshot in `zfs list -H -t snapshot | cut -f 1`
do
zfs destroy $snapshot
done

You can now hide terminal#2 and observe terminal #1. It will show you how many snapshots are left, refreshing the number every 6 seconds. Neat.