Tag Archives: RaidZ

CLI, OS

Create a RaidZ array with missing drive in FreeNAS

2 Comments

I came across a need to create a ZFS Raid-Z array with a missing drive. This is easy to do with mdadm but not as easy with ZFS. It is possible, though. The trick is to create an image file with dd, then map that image file as a loopback device. Once that’s done you can treat it as if it were a regular hard drive and add it to the array. Once added to the array you can take the loopback device offline and remove it from the array, then add an actual HDD later.

Create loopback device

Thanks to this site for the information.

First, use dd to create an image file. Change the seek parameter to whatever size disk you wish to emulate.

dd if=/dev/zero of=temp.img bs=1 count=1 seek=1024G

Next, initialize the loopback driver and create the loopback device (md0 in my case)

sudo losetup -a
sudo mdconfig -a -t vnode -f temp.img -u 0

List your loopback devices with the following command to verify your new loopback device:

sudo mdconfig -l

Create array using loopback device

You can now partition and add your loopback device as if it were a regular hard drive. Change volume name, array name, and device names as necessary for your environment.

sudo gpart add -t freebsd-zfs -l <volume name> md0
sudo zpool create <array name> raidz ada7p1 ada8p1 ada9p1 md0p1

Fail & Remove loopback device

Now that our new array is up and running properly we can fail out the loopback device. Make sure to modify the command to use your array name and loopback device/partition number.

sudo zpool offline <array_name> md0p1

Import new array into FreeNAS GUI

To get our new array in freenas we must export the array from the command line, then import it from the GUI.

sudo zpool export <array name>

Once the array has been exported, navigate to the FreeNAS GUI and go to Storage / Volumes / Import Volume.

You should now have your new array minus one drive ready to go in FreeNAS. You can now add a physical HDD when it becomes available (in my case, when it returns from RMA.)

CLI

Recovering a failed RaidZ pool

Leave a comment

Scenario: A drive is your RaidZ pool has gone bad. You have a replacement drive ready to go. You pull the drive you thought was the failed drive.. only to realize that you just pulled a good drive out, causing the array to go completely offline.

Has this happened to you? It has not happened to me yet, but I wanted to see how ZFS responded. I have to say I am pretty impressed.

I purposely pulled two working drives from my test zpool array. The status of the pool became Unavailable, as is to be expected. The zpool status command gave a helpful hint “Replace the drive and run zpool clear”

I replaced the last drive I had previously pulled and ran the command:

zpool clear storage

That was all I had to do! The array came back up (although in a degraded state) and all my files were there.

Output of zpool status at this point:

[root@freenas /data]# zpool status
  pool: storage
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0 in 0h26m with 0 errors on Sun Sep  7 09:51:19 2014
config:

        NAME                     STATE     READ WRITE CKSUM
        storage                  DEGRADED     0     0     0
          raidz1-0               DEGRADED     0     0     0
            ada2p1               ONLINE       0     0     0
            7167795297630497018  REMOVED      0     0     0  was /dev/ada3p1
            ada4p1               ONLINE       0     0     0
            ada1p1               ONLINE       0     0     0

errors: No known data errors

My next experiment was to bring the pool back to full health again. I tried to simply re-insert the last drive into my pool but it complained that it was already a part of the pool. The drive in question used to be labeled ada3p1. I tried “zpool detach storage ada3p1” but it complained: only applicable to mirror and replacing vdevs

After searching I found a mention here that said you can call out specific devices in your pool to clear. I ran the command
“zpool clear storage ada3p1” and it completed without any issues; however it still wouldn’t let me add the drive back into the pool saying it was already there.

What allowed me to bring the array back to full health was:

zpool online storage ada3p1

The amazing part – zfs realized that it only needed to sync a small amount of data to bring it back into sync with the pool!

 scan: resilvered 24K in 0h0m with 0 errors on Sun Sep  7 12:23:39 2014
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            ada2p1  ONLINE       0     0     0
            ada3p1  ONLINE       0     0     0
            ada4p1  ONLINE       0     0     0
            ada1p1  ONLINE       0     0     0

Compared to mdadm where it would rebuild the whole array even if it was the same disk you pulled, this is astounding.

I realized that this issue would only happen if you’re putting the same drive you just pulled back into the array, so I then tried pulling a drive and putting another in its place. After partitioning the drive, a simple

zpool replace storage 7167795297630497018 ada3p1

Did the trick (where the string of numbers is the placeholder for the drive you pulled – a zfs status will tell you what that number is.)  Done.

CLI

Creating a ZFS RaidZ volume with different sized disks

Leave a comment

While I hear that “ZFS likes to use the whole disk” I wanted to experiment with creating a RaidZ pool with disks of different sizes. This requires partitioning the larger disks. The GUI in FreeNAS does not allow you to do this, so we must venture toward the command line. While these commands were run in FreeNAS they will work in any FreeBSD based system.

The commands below assume you are using the first four disks in the system for the RaidZ pool. I realize you can make these commands more efficient by using shell-fu but I will put them all here for completeness.

Partition the disks

Create GPT table for each disk

  • gpart create -s gpt ada1
  • gpart create -s gpt ada2
  • gpart create -s gpt ada3
  • gpart create -s gpt ada4

If gpart complains (probably due to the disk already having a GPT table) you can nuke the GPT setup and start over via the following commands, replacing ada2 with the stubborn drive:

  •  gpart destroy -F ada2
  • gpart create -s gpt ada2

Create partition for each disk

  • gpart add -s 232G -t freebsd-zfs -l test0 ada1
  • gpart add -s 232G -t freebsd-zfs -l test0 ada2
  • gpart add -s 232G -t freebsd-zfs -l test0 ada3
  • gpart add -s 232G -t freebsd-zfs -l test0 ada4

Create the pool

  • zpool create storage raidz ada1p1 ada2p1 ada3p1 ada4p1

Why does everyone call their pool “tank”? It must be in some documentation somewhere that everyone copies.

If you want to replace a failed disk in a pool after the faulty disk has been removed, issue the following command:

  • zpool replace storage <old/failed device name> ada2p1

Sometimes even though you’ve nuked the gpt data of the disk zpool will complain about the disk already being a member of a pool, e.g. “/dev/ada2p1 is part of active pool ‘storage'” Another scenario is if you have properly replaced the failed drive but the pool still shows degraded with a hash referring to the old drive showing in zpool status. To fix these issues use the zpool detach command

zpool detach storage ada2p1

 Check pool status

  • zpool status

If it comes out healthy you are good to go.

Import the raidZ into FreeNAS

With the pool manually created you can now import it into FreeNAS so it can be monitored / managed.

  • Click on Storage / Volumes / Auto Import Volume
  • Click No, skip to import
  • Wait a minute for it to scan, then click OK

Done.