FreeNAS on Xenserver with PVHVM support

In my current home setup I have a single server performing many functions thanks to Citrix Xenserver 6.2 and PCI Passthrough. This single box is my firewall, webservers, and NAS. My primary motivation for this is power savings – I didn’t want to have more than one box up 24/7 but still wanted all those separate services, some of which are software appliances that aren’t very customizable.

My current NAS setup is a simple Debian Wheezy virtual machine with the on-board SATA controller from the motherboard passed through to it. The VM runs a six drive software RAID 6 using mdadm and LVM volume management on top of it. Lately, though, I have become concerned with data integrity and my use of commodity drives. It prompted me to investigate ZFS as a replacement for my current setup. ZFS has many features, but the one I’m most interested in is its ability to detect and correct any and all corrupted files / blocks. This will put my mind at ease when it comes to the thousands of files that I have which are accessed infrequently.

I decided to try out FreeNAS, a NAS appliance which utilizes ZFS. After searching on forums it quickly became clear that the people at FreeNAS are not too keen on virtualizing their software. There is very little help to be had there in getting it to work in virtual environments. In the case of Xenserver, FreeNAS does work out of the box but it is considerably slower than bare metal due to its lack of support of Xen HVM drivers.

Fortunately, a friendly FreeNAS user posted a link to his blog outlining how he compiled FreeNAS to work with Xen. Since Xenserver uses Xen (it’s in the name, after all) I was able to use his re-compiled ISO (I was too lazy to compile my own) to test in Xenserver.

There are some bugs to get around to get this to work, though. Wired dad’s xenified FreeNAS doesn’t appear to like to boot in Xenserver, at least out of the box. It begins to boot but then hangs indefinitely on the following error:

run_interrupt_drive_hooks: still waiting after 60 seconds for xenbusb_nop_confighook_cb

This is the result of a bug in the version of qemu Xenserver uses. The bug causes BSD kernels to really not like the DVD virtual device in the VM and refuse to boot. The solution is to remove the virtual DVD drive. How, then, do you install FreeNAS without a DVD drive?

It turns out that all the FreeNAS installer does is extract an image file to your target drive. That file is an .xz file inside the ISO. To get wired dad’s FreeNAS Xen image to work in Xenserver, one must extract that .xz file from the ISO, expand it to an .img file, and then apply that .img file to the Xenserver virtual machine’s hard disk. The following commands can be run on the Xenserver host machine to accomplish this.

  1. Create a virtual machine with a 2GB hard drive.
  2. Mount the FreeNAS-xen ISO in loopback mode to get at the necessary file
    mkdir temp
    mount -o loop FreeNAS-9.2.1.5-RELEASE-xen-x64.iso temp/
  3. Extract the IMG file from the freeNAS ISO
    xzcat ~/temp/FreeNAS-x64.img.xz | dd of=FreeNAS_x64.img bs=64k
    

    Note that the IMG file is 2GB in size, which is larger than can sit in the root drive of a default install Xenserver. Make sure you extract this file somewhere that has enough space.

  4. Import that IMG file into the virtual disk you created with your VM in step 1.
    cd ..
    xe vdi-import uuid=<UUID of the 2GB disk created in step 1> filename=FreeNAS_x64.img
    

    This results in an error:

    The server failed to handle your request, due to an internal error.  The given message may give details useful for debugging the problem.
    message: Caught exception: VDI_IO_ERROR: [ Device I/O errors ]
    

    This error can be safely ignored – it did indeed copy the necessary files.
    Note: To obtain the UUID of the 2GB disk you created in step 1, run the “xe vdi-list” command and look for the name of the disk.

  5. Remove the DVD drive from the virtual machine. From Xencenter:
    Shutdown the VM
    Mount xs-toos.iso
    Run this command in a command prompt:

    xe vm-cd-remove uuid=<UUID of VM> cd-name=xs-tools.iso
  6. Profit!

There is one aspect I haven’t gotten to work yet, and that is Xenserver Tools integration. The important bit – paravirtualized networking – has been achieved so once I get more time I will investigate xenserver tools further.

20 thoughts on “FreeNAS on Xenserver with PVHVM support”

  1. I followed your steps but I’m using FreeNAS-9.2.1.8-RELEASE-xen-x64.iso from that website.

    Everything went find until booting up the VM. It states: “No /boot/loader”

    Any advise?

      1. Hmm, interesting. One suggestion I have would be to try and make your VM hard drive more than 2G, maybe two and a half. Perhaps the way Xenserver 6.5 provisions disks is different in such a way that it cuts off part of the FreeNAS image.

        I haven’t tried getting FreeNAS working in 6.5 but I will see if I can get it going this weekend.

      2. I just spun up a test instance of Xenserver 6.5 and followed my steps, using FreeNAS-9.2.1.8-RELEASE-xen-x64.img.xz from Wired Dad’s website. The installation worked perfectly.

        I noticed that you said you used the ISO. Wired Dad updated that article to include a link to the image file directly at the bottom of the page. Try downloading the image file instead of extracting it from the ISO and see if that will work. Hope this helps.

  2. Thank you for this pot. Followed the above instructions and I am getting ‘tapdisk experienced an error’ every time I try to start the VM. Googling has not turned up very much so far.

    Any thoughts?

    1. I have not experienced that error before, but the limited research I did on it suggests it’s an issue with Xenserver trying to use the DVD drive on the VM when there is no medium / file available. One suggestion (taken from here) is that you try to boot the VM with a CD in the drive at least once (it will hang in FreeNAS’s case but you can still have it try.) You can then try to remove the DVD drive from that VM completely and see if it will boot. Hope this helps.

  3. Great post! I’ve been looking for this a long time. However how do I extract the FreeNAS-x64.img.xz file to an other location than to the root of my xenserver install? I have other storage devices connected to the xenserver with plenty of free space but I dont know how to access them from the xenserver command line.
    I’ve tried to extract it in the xenserver root because I’ve installed xenserver on a 16gb usb flash disk. However I get when I try to expand I get the following message:
    dd: writing ‘FreeNAS_x64.img’ : No space left on device
    So it seems not enough space is available.
    I hope you guys can update me with some basic knowledge about how I can solve this problem. Once again, great post!

    1. In my case what I did was mount an external storage device (another hard drive) and do the extraction there. After I wrote this post, Wired Dad was kind enough to provide the link to the img.xz file directly so you no longer have to extract it from the ISO file. It is located on this page, at the bottom of the post where it says “And lastly, the image file:”

      Hope this helps.

      1. Finally got it working 🙂 Thanks for the tip. The problems I had initially were with the .img file it self (not the .img.xz) but I managed to extract it and get FreeNas running now!
        Do you guys maybe know how I can passthrough hard drives installed locally on the XenServer to the FreeNAS VM and manage the ZFS RAID from there? I think this is the way I will configure my FreeNAS. I have 3 3T WD Red drives in there.
        How are things going with the support of Xenservertools?

        1. The only way I’m aware of for passing drives through to guests in Xenserver is to use PCI Passthrough. It requires having hardware capable of doing it (Motherboard & Processor.) Once you have the necessary hardware and the feature enabled in the BIOS, you can pass the hard drive controller those drives are attached to through to the VM guest. Follow this guide I wrote back in December for details on how to do that.

          As for xenserver tools, I’ve stopped looking into finding a way to get that to work. From the looks of it you’ll have to re-compile FreeNAS from source in order to add it, and I’ve just not had the time or motivation to do that.

    1. Unfortunately no. I haven’t had much motivation to do it since everything works great without guest utilities installed. You will have to re-compile FreeNAS from source to add guest utilities support.

    2. I did build a version that has the guest utilities built in, so the only thing I run as a task on freeNAS boot is:
      /usr/local/etc/rc.d/xenguest onestart

  4. How exactly did you configure the VM itself to get proper PVHVM support from xenserver? Could you post the commands, as I assume this cannot be done through xencenter.

    Thanks a lot!

    1. PVHVM support was compiled into the FreeNAS image itself – there were no commands run. Wired Dad has a writeup on how to do this here I am simply using the FreeNAS image he compiled.

        1. Yes, all I did was use the “Other Media” template, which tells Xenserver to create an HVM host. The PV in PVHVM refers to the guest OS having paravirtualized (PV) drivers for disk and network access. In FreeNAS’ case that means compiling it with PV drivers, which Wired Dad has graciously done. The setup should be simply using the Other template in Xenserver in conjunction with Wired Dad’s compiled version of FreeNAS.

  5. Hey there,

    This post caught my attention, as it’s dealing with stuff that’s similar to issues I’m working with myself. This post is great information – thanks.

    I’m a freenas noob, but quickly falling in like with it. Before I get too serious, and bed the guy down, I’ve gotta decision to make.

    1) Virtualise the freenas box, with sata passthrough. Your post will probably get me there. Thanks!
    2) Use the BSD jails for the rest of my virtualisation needs…. though I’m still new at this, I understand I can setup plex ‘plugins’ (which as a dead easy jail from what I can see). I should be able to run virtualboxen in jails as well….

    This decision is a little complicated by a current thought to use the freenas appliance as my:

    Openvpn server
    pf wall
    media services
    virtual hosting.

    I’ve written quiet a long muse on this at https://forums.whirlpool.net.au/forum-replies.cfm?t=2467512 . As it turns out, it doesn’t appear that many people are playing in this space.. Would appreciate any thoughts.

    Thanks heaps for the help so far.

    Ben

    1. Ben,

      My current setup is Option #1. I picked this setup mainly because I was not happy with how much hacking was involved to get some of the FreeNAS jails working. Yes it’s possible to do your setup in either environment, but for my particular setup (which is pretty similar to yours) I decided it was easier to virtualize everything. For example, it was much easier for me to spin up the latest version of owncloud in a distro with plenty of documentation (debian, ubuntu) than it was to wade through endless documentation to try and get the latest version working in FreeNAS (the jail plugin doesn’t appear to be updated nearly as frequently as I’d like.) That being said, both options are pretty rare setups and it’s hard to find good documentation for either one, as I’m sure you’ve become aware.

      I have been running a virtualized version of FreeNAS for over a year now and have not had any problems with it. I have separate VMs for owncloud, pfsense, splunk, plex, mythtv, sophos UTM, several hosted sites (including this one) and even a Windows VM for Windows apps I can’t get away from. It’s all sitting on a single box in my basement, because why not? it’s fun 🙂 I’ve been using xenserver for all this for about a year and a half, starting from 6.2 and now on 6.5, and it’s all been working wonderfully.

      I didn’t get very far with going the hardware FreeNAS route so I can’t speak to which one is actually easier, but in the little time I spent on trying to get it to work it seemed like virtualizing everything was easier. Your mileage may vary, of course. I’d be interested to know which route you pick.

      Hope this helps.

      -Nick

Leave a Reply

Your email address will not be published. Required fields are marked *