Virtualization

PCI passthrough with Xenserver 6.2

Leave a comment

PCI passthrough is a great way to mix virtualization with bare metal hardware. It allows you to pass physical hardware to virtual machines. In order to do PCI passthrough you will need compatible hardware (a CPU and chipset that support it.) Intel’s nomenclature for this is VT-d; AMD’s is IOMMU. It’s difficult (although not impossible) to get consumer level hardware that supports this. It’s much easier to obtain with server grade hardware.

Why would you want to pass physical hardware to virtual machines? In my case, it’s to turn a single system into a super server. Instead of having separate physical systems for NAS, gaming, and TV recording (my three uses) you can have one physical system do all three. While this is possible with one single OS, it’s much easier to manage these functions if they are in their own separate OS (especially if you’re using appliance VMs such as FreeNAS.) PCI Passthrough allows you to get the best of both worlds – better security by isolating functions, easier backup/restore, and physical hardware access.

Citrix Xenserver 6.2 supports PCI passthrough beautifully. A great comprehensive guide on how to configure PCI passthrough can be found here.

Xenserver 6.2 no longer requires any configuration beforehand to get PCI passthrough to work. To pass a device to a VM all you need to do is obtain its the bus, device, function (B:D.F) via lspci, then pass that through to the VM.

lspci
<several lines deleted>
06:00.0 Ethernet controller: Atheros Communications AR8131 Gigabit Ethernet (rev c0)

The B:D.F of the above device (a network adapter) is 06:00.0. To then pass this device to a virtual machine we use the xe vm-param-set command with the other-config:pci= parameter, adding 0/0000: to the beginning of the B:D.F, then specifying the UUID of the VM in question.

xe vm-param-set other-config:pci=0/0000:06:00.0 uuid=db4c64e1-44ce-f9f3-3236-0d86df260249

If the VM is running when you issue that command, make sure to shut down (not reboot) the VM, then start it up again.

To add multiple devices to the same VM, simply separate each B:D.F with a comma, like so:

xe vm-param-set other-config:pci=0/0000:06:00.0, 0/0000:07:00.0 uuid=db4c64e1-44ce-f9f3-3236-0d86df260249

Sometimes if you pass multiple PCI devices to a single VM only one of those devices is recognized by the VM. If that is the case, try passing the B:D.F of each piece of hardware in a different order

If you ever want to remove a hardware mapping to a VM, issue the following:

xe vm-param-clear param-name=other-config uuid=<UUID of VM>

There is still a case where you want to modify Xenserver’s configuration with regard to PCI passthrough. On occasion you will have hardware that you do not want the hypervisor to ever know about (in the above example, the hypervisor can use the hardware until you power on a VM that has passthrough enabled for it.)

In my case, I don’t want the hypervisor to ever see the storage controller I’m passing to my NAS VM. I found this out the hard way. If you don’t modify your xenserver configuration to ignore storage controllers that you then pass through to a VM, the entire hypervisor will completely lock up if you happen to reboot that VM. Why? Because when that VM reboots it releases the storage controller back to the hypervisor, which promptly enumerates and re-names all of its attached drives. It often leads to a case of re-naming /dev/sda, promptly “losing” the root device, and kernel panicking.

So, if you are passing things you never want the hypervisor to see, you need to modify its boot configuration to “hide” those devices from it. Edit /boot/extlinux.conf and append pciback.hide=(B:D.F) to the Linux command line, right after the splash parameter

vi /boot/extlinux.conf 
<navigate to right after the word splash>
i
pciback.hide=(06:00.0)(01:00.0)
<esc> :wq
extlinux -i /boot

The above example excludes two devices. Multiple devices simply go next to each other in their own parenthesis, but the format is the same if you only passing a single device.

Reboot the hypervisor, and you are good to go. You can now pass hardware through to VMs to your heart’s content.

Analytics

Make stats in Splunk more meaningful with fillnull

Leave a comment

I mentioned in my last post about a common issue that I have with the stats command: items with empty values are simply excluded from the results. What if you want to include those empty results with the stats command?

The solution, which I found here, is to use the fillnull command.

<search query> | fillnull value=”-” | stats count by <field(s) which contain empty values>

It’s that simple! Now instead of excluding empty results, they are included and display as a dash. Brilliant.

Analytics

Perform DNS lookups on Splunk fields

4 Comments

I recently came across a very handy command in Splunk, the lookup command. Thanks to this website I was able to learn how to use the lookup command to give me more relevant results. Instead of Splunk listing a bunch of IP addresses, it now returns a column with everything it could resolve. Seeing resolved domain names alongside IP addresses gives much more meaning to the data.

The command is as follows:

<search> | lookup dnslookup clientip as <IP Field> OUTPUT clienthost as <Resolved Hostname>
  • <search> is your original search
  • <IP Field> is the field which contains the IP addresses you want to do name lookups on
  • <Resolved Hostname> is the name of the column which will contain your resolved hostnames.

You can order your search results in a table if you do the above command before your stats or table command. The example below is to parse some firewall logs from a single source host and perform lookups on them.

<search> | lookup dnslookup clientip as dstip OUTPUT clienthost as Resolved_hostname | stats count by dstip Resolved_hostname dstport proto action

Be careful when using the stats command, though. If the IP address is local it will have a blank resolved hostname, which will exclude it from the stats table.

CLI

Quickly create xen disk images with dd

Leave a comment

To this point I have been creating disk images with dd – pretty standard. The command I traditionally use is this:

sudo dd if=/dev/zero of=pc-bsd-10.1.img bs=1M count=40000

This command works but it takes some time as dd copies zeroes for every single part of the image.

It turns out that there is a much faster way to create a disk image, which I found out thanks to this article. Still using dd, you can simply use the -seek parameter to very quickly create your image

sudo dd if=/dev/zero of=pc-bsd-10.1.img bs=1 count=1 seek=40G

By setting bs and count to 1 but seek to 40G I have created a 40 gigabyte empty file image without having to wait for all those pesky zeroes to be written. Awesome.

Networking

Sophos UTM returns NXDOMAIN for valid domain names

Leave a comment

This issue took me a while to figure out. It’s actually been an issue for a while but I didn’t notice it until XBMC became Kodi. XBMC moved their domain name to kodi.tv and suddenly I was unable to access their site at all.

An nslookup returns NXDOMAIN immediately; however, querying a different server, say Google’s DNS, would return a valid address.

After scratching my head for weeks I came across this post which outlines the exact same problem – any .tv domains are instantly not resolved. I didn’t notice it until XBMC moved to kodi.tv because I don’t visit .tv domains.

The culprit: static DNS entries without a fully qualified domain name. I have plenty of these, and in this case, I have a computer named simply “tv” which Sophos translates into an internal DNS zone; Consequently it doesn’t even bother querying other DNS servers for anything ending in .tv.

The fix: make sure you have fully qualified domain names for all of your static DNS entries. This best practice will save you headaches in the long run.

Virtualization

FreeNAS PCI Passthrough dev_taste error message

Leave a comment

After getting my xenified FreeNAS up and running I noticed an oddity with disk reporting. When I pulled up the reports tab I noticed ada0 never showed any activity, despite my knowing that disk is doing plenty.

The mystery became greater when I noticed these error messages in my logs:

g_dev_taste: make_dev_p() failed (gp->name=ada0, error=17)

After some research I discovered here that disks passed through to a VM via Xen’s PCI Passthrough function present themselves to FreeBSD in a peculiar manner. In particular, the first disk in the passthrough array presents itself as ada0, despite the boot disk also having the name of ada0. With two disks named ada0 it’s a tossup on which one shows up in reporting, not to mention the strange errors above.

The fix is to add BSD parameter to not start disk numbering at ada0. For FreeNAS, you do this via the tunables section (System / Tunables / Add Tunable.) Add the following tunable:

variable: hint.ada.0.at
Value: scbus100
Comment: ada0 PCI passthrough fix
Enabled: true

Once that is configured, reboot FreeNAS. You will now have proper reporting of all your passthrough disks and the strange dev_taste errors will be gone.

CLI

Redshift – a better flux program for Linux

10 Comments

F.lux is a wonderful tool for helping eye strain. People who stare at computer screens all day (like myself) can experience quite a bit of eye strain due to the harsh lights screens emit. One solution is to wear yellow tinted gamer goggles. I chose the cheaper route, installing software to adjust the color temperature of your monitor. Flux does this beautifully.. for Windows, at least.

Linux is a different story. Its GUI is pretty flaky and appears to only work for one screen. Enter Redshift, an updated fork of the Linux port of F.lux, which properly supports dual monitors. Unfortunately, it is harder to configure than F.lux. It is a command line only tool (with a GUI indicator component) and it requires creating a manual configuration file.

On my Linux Mint system (Ubuntu based) I needed to install the following:

sudo apt-get install redshift gtk-redshift

I had a hard time getting day/night changes to work. Redshift allows you to specify several different location options, but none of them appeared to work for me. I then realized that I like the softer colors of redshift all the time so I simply set the same temperature for day or night. It now doesn’t matter what the latitude / longitude is.

I found it odd that the settings for flux and redshift don’t appear to be the same. I tweaked my config a little bit to best closely match my Windows f.lux setup. Below is my config file, placed in ~/.config/redshift.conf.

; Global settings for redshift
[redshift]
; Set the day and night screen temperatures

temp-day=4500
temp-night=4500

; Enable/Disable a smooth transition between day and night
; 0 will cause a direct change from day to night screen temperature.
; 1 will gradually increase or decrease the screen temperature
transition=1

; Set the screen brightness. Default is 1.0
;brightness=0.9
; It is also possible to use different settings for day and night since version 1.8.
;brightness-day=0.7
;brightness-night=0.4
; Set the screen gamma (for all colors, or each color channel individually)
gamma=0.8
;gamma=0.8:0.7:0.8

; Set the location-provider: 'geoclue', 'gnome-clock', 'manual'
; type 'redshift -l list' to see possible values
; The location provider settings are in a different section.
location-provider=manual

; Set the adjustment-method: 'randr', 'vidmode'
; type 'redshift -m list' to see all possible values
; 'randr' is the preferred method, 'vidmode' is an older API
; but works in some cases when 'randr' does not.
; The adjustment method settings are in a different section.
adjustment-method=randr

; Configuration of the location-provider:
; type 'redshift -l PROVIDER:help' to see the settings
; ex: 'redshift -l manual:help'
[manual]
lat=40
lon=110

; Configuration of the adjustment-method
; type 'redshift -m METHOD:help' to see the settings
; ex: 'redshift -m randr:help'
; In this example, randr is configured to adjust screen 1.
; Note that the numbering starts from 0, so this is actually the second screen.
[randr]
screen=0

After saving the config file you can add gtx-flux  gtk-redshift as a startup application to have it automatically load on startup. My eyes feel much more comfortable now.

 

 

Web

Add x-forwarded-for header to Apache

Leave a comment

If you happen to be running your site behind a web application firewall you will notice that initially you will not be able to determine the true source of traffic coming to your server. The default setup for Apache will only show traffic coming from the firewall itself.

To fix this, you need to tweak the LogFormat parameters in /etc/apache2/apache2.conf (for Debian distros) or wherever your apache config file is in other distros. Per here, you need to add

 %{X-Forwarded-For}i

to your config file. Here is an example setup successfully implementing X forwarded For as well as maintaining logging the IP of the WAF itself (in case you have more than one..)

LogFormat "%v:%p %{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Web

Configure Sophos UTM to forward e-mail for internal clients

Leave a comment

If you’re in a situation like mine you have an ISP which stubbornly blocks port 25. You can get around this by using SSL/TLS of course but configuring each server can be a pain. The solution, in my case, is to turn my UTM into an e-mail relay server and simply have each node point to it for SMTP.

The process wasn’t as straightforward as I had hoped. Below are the steps to accomplish this:

First, enable the SMTP proxy by going to E-mail Protection / SMTP.

Next, go to the Relaying tab and configure allowed relay hosts. For my setup I went to the Host-based relay section and added each individual host that I want to have use my UTM for e-mail relaying.

Lastly, you must configure your UTM to send e-mails out to your ISP’s mail relay since by default the UTM tries to send e-mails out itself on port 25, which in my case is blocked. Do this by going to the Advanced tab and scrolling down to smart host settings. Configure the mail relay server, credentials, and port required there.

That part was fairly straightforward; however the spam protection feature of the UTM was preventing any messages sent to the UTM from being sent to the internet.

After much weeping, wailing and gnashing of teeth (including adding an SPF record for my domain) I discovered that the solution was to ensure that outgoing mail shows the node’s name in the From: field. If your server’s name is server, the from should always be server@yourdomain.org. I haven’t dug into why, but oh well! It works.