It turns out that when hard drives fail, they don’t all fail completely. In fact, most fail silently, getting worse and worse as time moves on, causing bitrot and other issues.
I had a suspicion that one of my drives was failing so I thought I would test it. The tool for the job: badblocks.
badblocks writes data to the drive and then reads it back to ensure it gets the expected result. I have learned a lot about hard drive failure lately and now subscribe to running badblocks on every new hard drive I receive to ensure it is a good drive. The command I use is:
badblocks -wsv <device>
This is a destructive write test – it will wipe the disk. You can also run a non-destructive test, but for new disks you can go ahead and wipe them. I also use badblocks to ensure old disks can still be trusted with data. It’s great for “burn in” testing to ensure a drive won’t fail.
Update 3/1/19: If you encounter the following error:
badblocks: Value too large for defined data type invalid end block (5860522584): must be 32-bit value
It means your drive is too big for badblocks to recognize using the default sector size. Fix this by specifying a 4k sector size:
badblocks -b 4096 -wsv <device>
Thanks to Ubuntu Forums for the info.
While I hear that “ZFS likes to use the whole disk” I wanted to experiment with creating a RaidZ pool with disks of different sizes. This requires partitioning the larger disks. The GUI in FreeNAS does not allow you to do this, so we must venture toward the command line. While these commands were run in FreeNAS they will work in any FreeBSD based system.
The commands below assume you are using the first four disks in the system for the RaidZ pool. I realize you can make these commands more efficient by using shell-fu but I will put them all here for completeness.
Partition the disks
Create GPT table for each disk
- gpart create -s gpt ada1
- gpart create -s gpt ada2
- gpart create -s gpt ada3
- gpart create -s gpt ada4
If gpart complains (probably due to the disk already having a GPT table) you can nuke the GPT setup and start over via the following commands, replacing ada2 with the stubborn drive:
- gpart destroy -F ada2
- gpart create -s gpt ada2
Create partition for each disk
- gpart add -s 232G -t freebsd-zfs -l test0 ada1
- gpart add -s 232G -t freebsd-zfs -l test0 ada2
- gpart add -s 232G -t freebsd-zfs -l test0 ada3
- gpart add -s 232G -t freebsd-zfs -l test0 ada4
- zpool create storage raidz ada1p1 ada2p1 ada3p1 ada4p1
Why does everyone call their pool “tank”? It must be in some documentation somewhere that everyone copies.
If you want to replace a failed disk in a pool after the faulty disk has been removed, issue the following command:
- zpool replace storage <old/failed device name> ada2p1
Sometimes even though you’ve nuked the gpt data of the disk zpool will complain about the disk already being a member of a pool, e.g. “/dev/ada2p1 is part of active pool ‘storage'” Another scenario is if you have properly replaced the failed drive but the pool still shows degraded with a hash referring to the old drive showing in zpool status. To fix these issues use the zpool detach command
zpool detach storage ada2p1
Check pool status
If it comes out healthy you are good to go.
With the pool manually created you can now import it into FreeNAS so it can be monitored / managed.
- Click on Storage / Volumes / Auto Import Volume
- Click No, skip to import
- Wait a minute for it to scan, then click OK
GREP is a very handy little tool that will look inside files for things. I came across a need to recursively search all files in my current directory for a certain string. Grep did this beautifully. The command I used is:
grep -r “text string to search” .
That did the trick! Thanks to cyberciti.biz for the information.