All posts by nicholas

Hotplug devices between Xen dom0, domU, and back again

August 17, 2014 nicholas Leave a comment

In my experiments with Xen to make dual booting obsolete, I’ve come across a need to hotplug PCI devices between dom0 and domU; Specifically, the SATA controller that my DVD-RW drive is connected to.

My DVD drive supports Lightscribe, which unfortunately is not nearly as strong in Linux as it is in Windows. You can get it to work but the label maker program is extremely basic. If I want to burn a lightscribe disc and have it look at all pretty it requires Windows.

The way I was doing PCI passthrough before was pretty inconvenient. It involved editing /etc/xen/pciback.conf and adding the bus:device.function (BDF) of the device I want to pass. This causes that device to be claimed by the pciback driver at boot time.

That’s all and well and good for the virtual machine, but what if you want your dom0 to use that device? You would have to remove the device from pciback.conf and reboot the machine.

As of Xen 4.2 there is now a better way. You can have the pciback driver claim a device and return it to its original driver at any time without having to reboot. The four magic commands are:

xl pci-assignable-add <BDF>
xl pci-attach <domain id / name> <BDF>
xl pci-detach <domain id / name> <BDF>
xl pci-assignable-remove -r <BDF>

The -r in pci-assignable-remove is necessary – it instructs xen to load the original driver that was loaded before we invoked pci-assignable-add. If you are using the xm toolstack instead, simply replace xl with xm.

Detaching from Dom0 and attaching to DomU

In my case I enter the following into a console whenever I want my Windows 8.1 virtual machine to have physical control of my DVD drive:

sudo xl pci-assignable-add 03:00.0 sudo xl pci-attach win8.1 03:00.0

Windows specific issues

It should have been as simple as that; Unfortunately, I ran into a road block. For some reason on the first try, Windows detected the drive but wouldn’t load any drivers for it (it thought none were necessary)

(this screenshot was taken when I was using a hard drive for troubleshooting, but the issue was the same with the DVD drive)

I tried ejecting the SATA controller and scanning for new devices as described on various forums, but that didn’t seem to work. The fix for me was to reboot the VM. Rebooting caused the PCI device to detach, so after the VM finished rebooting I had to re-issue “sudo xl pci-attach win8.1 03:00.0” to attach it again. Triumph!

I tried to make the second pci-attach command unnecessary by adding pci=03:00.0 to my virtual machine’s configuration file, but since I was passing a storage controller it kept trying to boot from drives attached to that controller instead of the virtual machine’s hard drive. I tinkered around with the config file for a while to try and get it to boot from the VMs hard drive again but couldn’t get it to work.

Since everything works by simply issuing pci-attach twice I gave up and just moved on. In one final bout of tinkering I discovered that if you issue pci-attach right after you boot the VM but before the OS finishes loading, it works on the first try. So the moral of the story here is Microsoft weirdness requires you to jump through some minor hoops to get this to work.

Returning to Dom0

When I want my dom0 to have the drive back I issue the following:

sudo xl pci-detach win8.1 03:00.0 sudo xl pci-assignable-remove -r 03:00.0

No complications here, although there is a funny bug. The file manager used in Linux Mint 16 gets confused and keeps adding CD ROM entries each time I pass the drive back and forth, but everything still works – it’s just a visual bug.

The drive is now accessible by dom0 once again. Success!

Virtualization

Xenserver and clock drift

August 15, 2014 nicholas Leave a comment

When it comes to a virtual machine’s clock my experience with other virtualization solutions has been that it’s automatically synchronized with the host machine. I didn’t notice until recently that this is not the case with Citrix Xenserver – at least when it comes to PVHVM machines.

I tried installing openntpd on each of my VMs and setting it to my internal NTP server (which in turn synchronizes with the web.) After a few days I was frustrated to see that the servers were still not in sync – some were minutes behind while others were inexplicably minutes ahead. Some of this might have to do with my experiments on live migrating these VMs a while back.

At any rate, it was clear that openntpd failed to do the job. Some research revealed that there is a bug where it reports adjusting the clock when the real status was an error. That little bug cost me an hour or two of digging and troubleshooting. Very frustrating.

I switched to plain old ntp instead and the problem was resolved within moments.

Moral of the story: Make sure you have a proper NTP setup for each of your VMs if you’re going to use Citrix Xenserver.

Analytics

Reverse search result order in Splunk

August 14, 2014 nicholas Leave a comment

When you run a query in Splunk it returns the most recent result at the top of the screen by default.

For far too long now I have been running queries in Splunk and then manually clicking back to the last page of results so that I can see the first time something happened.

It turns out there is a better way. Simply append ” | reverse” (without quotes) to the end of your search result. This will cause the earliest search result to be at the top, rather than the most recent. Handy.

Web

Changing permalink structure in WordPress

August 2, 2014 nicholas Leave a comment

After installing a fresh installation of WordPress on LAMP (Linux, Apache, MySQL, and PHP) I decided to change the new site’s permalink structure. The default structure on WordPress is so boring and uninformative – sitename/?p=(number). I prefer the blogger method – sitename/year/month/postname.

Changing the structure should be pretty straightforward; However, when I clicked that option and clicked save, every post greeted me with a wonderful 404 error page.

If you are running apache it turns out there are a few extra things you need to do that aren’t mentioned in WordPress’s 5 minute install guide. Detailed instructions are found here.

Enable the mod-rewrite apache plugin
Ensure the directory of your wordpress installation has a Directory entry and that AllowOverride All is enabled
Restart apache

After this, everything worked. The reason behind needing this change is WordPress modifies the .htaccess file for its installation folder with a simple rule:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ – [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

All it does is transparently prepend index.php to any web request it receives. Index.php is smart enough to direct pages wherever it needs to after that. If you don’t want to mess with enabling mod_rewrite, you can simply change your site structure to have /index.php/ before everything else. It will accomplish the same thing, only now index.php will show up in all your site URLs.

Virtualization

Migrating a Windows 8.1 VM from Xen to Xenserver

July 23, 2014 nicholas 2 Comments

Since Citrix recently released the entire Xenserver product to the world as free, open source software I thought I might give it a try. I have been pleased with the results and wanted to migrate my desktop VM over to it.

I’ve had a devil of a time getting my Windows 8.1 Professional virtual machine to migrate from plain Xen to Citrix Xenserver 6.2. My first mistake was not doing research before migrating hypervisor environments. While it is true that Citrix uses Xen as the underlying hypervisor, it turns out that there are still plenty of differences between the two environments.

I thought I would take the easy route by installing Citrix Xenconvert and converting my Xen Win8.1 VM to a format Xenserver likes. Although Xenconvert was designed for Physical to Virtual migration, I’ve found in the past that it works just as well for virtual to virtual migration.

After migrating to Xenserver I was greeted with the following friendly message:

INACCESSIBLE_BOOT_DEVICE windows 8.1 bluescreen — lovely

As far as I can tell it was the Xen GPLPV drivers that were the culprit. This leads me to my second mistake: not having a proper backup of the VM. I didn’t keep a backup of this VM in the Xen-friendly format after I migrated it to xenserver. This was mainly due to laziness – a classic example of “one ounce of laziness now produces one ton of hard work later.”

Instead of simply just booting the VM and removing the GPLPV drivers I had to attempt to do it via the Windows PE on the Windows 8.1 disc. I first tried running the GPLPV uninstall script from here, modifying it to point to the c:\ drive for both files and registry settings. Alas, that didn’t appear to do anything.

I then tried to go through the registry via the Windows PE and remove any references to Xen-anything. Success! Or so I thought. It turns out that blindly plowing through the Windows registry without an idea of exactly what you were doing has consequences. The VM would boot but I could not for the life of me get network drivers to work. As far as I can tell I corrupted something in the registry and despite my best efforts I couldn’t fix it.

At this point I had learned to back things up so I kept restoring from backups and messing with removing various registry keys. I continued this trial and error process for some time. After much weeping, wailing, and gnashing of teeth, I finally found the right combination of keys you must remove in order to boot again.

I took what I learned and updated the script from above to make it work with the WinPE environment . Download it here.

Boot into your PE environment of choice and run the script. When it’s finished, your VM will now be able to boot successfully.

The last step is to go into device manager and delete all xen-related drivers, then re-install them. After all that is said and done, your migration from xen to xenserver is complete. Repeat the exact same process to migrate from xenserver back to xen.

Technicus