I have written previously about how to convert Citrix Xenserver 6.2 to a software RAID 1. When I upgraded to Xenserver 6.5 I found I had to re-install the xenserver instance because the upgrade didn’t recognize the software RAID. When trying to follow my own guide I found that I couldn’t create the array – it gave the following error message:
mdadm: unexpected failure opening /dev/md0
It turns out 6.5 handles RAID differently. You have to manually load the RAID kernel modules before you can create arrays. I was able to get this running successfully thanks to guidance from this site, specifically comments on it by Olli.
The majority of this can simply be copy/pasted into the command window, once drive paths have been updated for your specific setup.
# Prepare /dev/sdd sgdisk --zap-all /dev/sdd sgdisk --mbrtogpt --clear /dev/sdd sgdisk -R/dev/sdd /dev/sdc # Replicate partion table from /dev/sdc to /dev/sdd with unique identifier sleep 5 # Sleep 5 seconds here if you script this… sgdisk --typecode=1:fd00 /dev/sdd sgdisk --typecode=2:fd00 /dev/sdd sgdisk --typecode=3:fd00 /dev/sdd sleep 5 # Sleep 5 seconds here if you script this… modprobe md_mod # load raid, because it isn't load by default (XS6.5 only) yes|mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdd1 missing # Create md0 (root) yes|mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdd2 missing # Create md0 (swap) yes|mdadm --create /dev/md2 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdd3 missing # Create md0 (storage) sleep 5 # Sleep 5 seconds here if you script this… mkfs.ext3 /dev/md0 # Create root FS mount /dev/md0 /mnt # Mount root FS cp -xR --preserve=all / /mnt # Replicate root files mdadm --detail --scan > /mnt/etc/mdadm.conf #generate RAID configuration sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device mount --bind /dev /mnt/dev mount -t sysfs none /mnt/sys mount -t proc none /mnt/proc chroot /mnt /sbin/extlinux --install /boot dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdd chroot /mnt mkinitrd -v -f --theme=/usr/share/splash --without-multipath /boot/initrd-`uname -r`.img `uname -r` exit sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf # Update extlinux for new RAID device cd /mnt && extlinux --raid -i boot/ sgdisk /dev/sdd --attributes=1:set:2 #Unmount filesystems and reboot cd umount /mnt/dev umount /mnt/sys umount /mnt/proc umount /mnt sync reboot
Tell BIOS to use disk B
After reboot to disk B…
sgdisk -R/dev/sdc /dev/sdd # Replicate partition table from /dev/sdd to /dev/sdc with unique identifier sgdisk /dev/sdc --attributes=1:set:2 sleep 5 # Sleep 5 seconds here if you script this… mdadm -a /dev/md0 /dev/sdc1 mdadm -a /dev/md1 /dev/sdc2 mdadm -a /dev/md2 /dev/sdc3 # If this command gives error, you need to forget/destroy an active SR first #This next command is the only command you have to manually update before pasting in. Find the UUID of your xenserver host and paste it between the <> below xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=<UUID of xenserver host> name-label="RAID 1" shared=false type=lvm # Watch rebuild progress and wait until no arrays are rebuilding before proceeding with any reboot watch “mdadm --detail /dev/md* | grep rebuild”
Done!
I am trying to do this on a HP proliant with a B120i “RAID” controller. It works perfectly for md1 and md2, but I am not able to add /dev/sda1 to md0 because I am not able to select to boot from sdb1 in the HP bios. I have tried adding this to extlinux.conf:
label xeraid
# XenServer
kernel mboot.c32
append /boot/xen.gz dom0_mem=752M,max:752M watchdog dom0_max_vcpus=4 crashkernel=128M@32M cpuid_mask_xsave_eax=0 console=vga vga=mode-0x0311 — /boot/vmlinuz-3.10-xen root=/dev/md0 ro hpet=disable xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash — /boot/initrd-3.10-xen.img
and select it when booting.
Then I get some error on swap and mounting boot filesystem. The server reboots before I am able to note it down.
Do you have any suggestions on how to fix this?
It’s most likely either md0 not properly being initialized / set up, or the /etc/fstab file located within it not being updated properly. Make sure all references to the root filesystem are for /dev/md0, and that /dev/md0 has a complete copy of the OS except with /etc/fstab modified to refer to /dev/md0 as the root partition. Hope this helps.
My md0 fstab looks like this:
/dev/md0 / ext3 defaults 1 1
/var/swap/swap.001 swap swap defaults 0 0
none /dev/pts devpts defaults 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
none /sys/fs/cgroup cgroup memory 0 0
/opt/xensource/packages/iso/XenCenter.iso /var/xen/xc-install iso9660 loop,ro 0 0
Hmm, I don’t see anything wrong there. It’s hard to say without the full error message. Could you perhaps take a video of it that you can pause to read the error message?
We just wiped everything and started over and it worked perfectly.
To boot from disk B, we just swapped the drives.
I think my problems came from trying to use the 6.2 procedure first, and then some setting must have been left hanging.
Thanks for posting this procedure, it is really handy. 🙂
Thanks for this guide, very useful and seems to work a charm. I also swapped the drives over half way through rather than changing BIOS as mentioned by Hans.
With my limited understanding I just wanted to check something without having to test it: Should my server now boot from either disk in the event of failure and rebuild automatically when a replacement is added?
I’m glad this guide was helpful!
Yes, if you remove either one of the drives, your system will boot up just fine from the other drive (provided the BIOS is configured properly.) However, if you replace the drive it will not automatically rebuild. You will have to partition the new drive to match the surviving one, and then run the command “mdadm –add /dev/mdX (array device name) /dev/sdX (replacement device name).” For example, “mdadm –add /dev/md1 /dev/sda1”. This mdadm cheat sheet has a lot of useful information with regard to managing software RAIDs.
Very good !!!! This article was very useful for me. I install Citrix 6.5 in Lenovo ThinkServer TS440.
Hi there! at first, thanks for share your knowledge!!!
I have a Question: All related to md0 and swap works perfect, but md2 for storage not works for me since the mdadm –create step. I follow your instructions twice with the same result. Can you help me with this?
Many Thnks again!
Ariel
I’m not sure why it wouldn’t be working. Can you provide the error message you get when you try to use md2 for storage?
Hi. Thank you for helping. “Missing OS” is error when I booting from second disk. The process continues by booting on first disk. I don’t know what I missed… Can you indicate where I must to look? Thanks.
Hi. I found the issue. The root array must be level 1, not level 10 as I did.
Thanks a lot for this post, very helpful. I don’t think I would have done it by myself…
Great post!! If there is any hope to update this to XensServer 7?
Especially when you update the grub.
Sorry, I don’t have time to try XenServer 7 yet. I hope to get to it in the next month or so.
Is there any way to revert this process?
I have a XenServer 6.5 pool with 2 servers, both 6.5 with software raid.
Now I want to upgrade the pool, starting with the master. But starting a rolling pool upgrade from XenCenter end with “There was a failure communicating with the plugin”.
Starting the server from the install disk gives the message “No system found that could be upgraded”
I think this could be because starting from CD doesn’t start RAID?
I want to avoid installing a completely new pool and migrating all VMs to another pool.
Any help would be appreciated.
Daniel
It should be possible to follow the process in reverse; start with removing a drive from your RAID and partition it like the xenserver installer would. Then format the partitions, copy files from your RAID to the new disk, update boot loader data on the disk, and reboot (making sure to select the newly un-raided drive.)
Alternatively you can do what I did as outlined here : https://techblog.jeppson.org/2016/06/quickly-transfer-vms-xenserver-pools/ to ease the pain of migrating to a new xenserver pool.
OK, I think I’m making progress… XenServer bootet from non-raid-disk and appears online and healthy in XenCenter.
What I did:
Remove local SR:
xe sr-list (gives local storage-uuid)
xe pbd-list sr-uuid = (gives uuid)
xe pbd-unplug uuid =
xe sr-forget uuid =
Remove disk1 from RAID
– mdadm /dev/md0 –fail /dev/sda1 –remove /dev/sda1 (same for md1 and md2)
Boot from disk1 and install fresh XenServer6.5 on disk1 (to get partition scheme)
Boot back from disk2 (with RAID)
Mount root partition from disk1
mount /dev/sda1 /mnt
Copy files from RAID-disk to disk1 (overwrite, as the files exist from the fresh install I did to get the right partition scheme)
yes|cp -xR –preserve=all / /mnt
Boot from disk1
Now Xenserver booted from the non-raid disk. It shows online in XenCenter.
As I did this on a pool member while in maintenance mode, I’ll switch the pool master now and do the steps on my second server.
Then I’ll try to start the rolling pool update again.
I’ll keep you updated…