All posts by nicholas

Networking

Sophos UTM returns NXDOMAIN for valid domain names

Leave a comment

This issue took me a while to figure out. It’s actually been an issue for a while but I didn’t notice it until XBMC became Kodi. XBMC moved their domain name to kodi.tv and suddenly I was unable to access their site at all.

An nslookup returns NXDOMAIN immediately; however, querying a different server, say Google’s DNS, would return a valid address.

After scratching my head for weeks I came across this post which outlines the exact same problem – any .tv domains are instantly not resolved. I didn’t notice it until XBMC moved to kodi.tv because I don’t visit .tv domains.

The culprit: static DNS entries without a fully qualified domain name. I have plenty of these, and in this case, I have a computer named simply “tv” which Sophos translates into an internal DNS zone; Consequently it doesn’t even bother querying other DNS servers for anything ending in .tv.

The fix: make sure you have fully qualified domain names for all of your static DNS entries. This best practice will save you headaches in the long run.

Virtualization

FreeNAS PCI Passthrough dev_taste error message

Leave a comment

After getting my xenified FreeNAS up and running I noticed an oddity with disk reporting. When I pulled up the reports tab I noticed ada0 never showed any activity, despite my knowing that disk is doing plenty.

The mystery became greater when I noticed these error messages in my logs:

g_dev_taste: make_dev_p() failed (gp->name=ada0, error=17)

After some research I discovered here that disks passed through to a VM via Xen’s PCI Passthrough function present themselves to FreeBSD in a peculiar manner. In particular, the first disk in the passthrough array presents itself as ada0, despite the boot disk also having the name of ada0. With two disks named ada0 it’s a tossup on which one shows up in reporting, not to mention the strange errors above.

The fix is to add BSD parameter to not start disk numbering at ada0. For FreeNAS, you do this via the tunables section (System / Tunables / Add Tunable.) Add the following tunable:

variable: hint.ada.0.at
Value: scbus100
Comment: ada0 PCI passthrough fix
Enabled: true

Once that is configured, reboot FreeNAS. You will now have proper reporting of all your passthrough disks and the strange dev_taste errors will be gone.

CLI

Redshift – a better flux program for Linux

10 Comments

F.lux is a wonderful tool for helping eye strain. People who stare at computer screens all day (like myself) can experience quite a bit of eye strain due to the harsh lights screens emit. One solution is to wear yellow tinted gamer goggles. I chose the cheaper route, installing software to adjust the color temperature of your monitor. Flux does this beautifully.. for Windows, at least.

Linux is a different story. Its GUI is pretty flaky and appears to only work for one screen. Enter Redshift, an updated fork of the Linux port of F.lux, which properly supports dual monitors. Unfortunately, it is harder to configure than F.lux. It is a command line only tool (with a GUI indicator component) and it requires creating a manual configuration file.

On my Linux Mint system (Ubuntu based) I needed to install the following:

sudo apt-get install redshift gtk-redshift

I had a hard time getting day/night changes to work. Redshift allows you to specify several different location options, but none of them appeared to work for me. I then realized that I like the softer colors of redshift all the time so I simply set the same temperature for day or night. It now doesn’t matter what the latitude / longitude is.

I found it odd that the settings for flux and redshift don’t appear to be the same. I tweaked my config a little bit to best closely match my Windows f.lux setup. Below is my config file, placed in ~/.config/redshift.conf.

; Global settings for redshift
[redshift]
; Set the day and night screen temperatures

temp-day=4500
temp-night=4500

; Enable/Disable a smooth transition between day and night
; 0 will cause a direct change from day to night screen temperature.
; 1 will gradually increase or decrease the screen temperature
transition=1

; Set the screen brightness. Default is 1.0
;brightness=0.9
; It is also possible to use different settings for day and night since version 1.8.
;brightness-day=0.7
;brightness-night=0.4
; Set the screen gamma (for all colors, or each color channel individually)
gamma=0.8
;gamma=0.8:0.7:0.8

; Set the location-provider: 'geoclue', 'gnome-clock', 'manual'
; type 'redshift -l list' to see possible values
; The location provider settings are in a different section.
location-provider=manual

; Set the adjustment-method: 'randr', 'vidmode'
; type 'redshift -m list' to see all possible values
; 'randr' is the preferred method, 'vidmode' is an older API
; but works in some cases when 'randr' does not.
; The adjustment method settings are in a different section.
adjustment-method=randr

; Configuration of the location-provider:
; type 'redshift -l PROVIDER:help' to see the settings
; ex: 'redshift -l manual:help'
[manual]
lat=40
lon=110

; Configuration of the adjustment-method
; type 'redshift -m METHOD:help' to see the settings
; ex: 'redshift -m randr:help'
; In this example, randr is configured to adjust screen 1.
; Note that the numbering starts from 0, so this is actually the second screen.
[randr]
screen=0

After saving the config file you can add gtx-flux  gtk-redshift as a startup application to have it automatically load on startup. My eyes feel much more comfortable now.

 

 

Web

Add x-forwarded-for header to Apache

Leave a comment

If you happen to be running your site behind a web application firewall you will notice that initially you will not be able to determine the true source of traffic coming to your server. The default setup for Apache will only show traffic coming from the firewall itself.

To fix this, you need to tweak the LogFormat parameters in /etc/apache2/apache2.conf (for Debian distros) or wherever your apache config file is in other distros. Per here, you need to add

 %{X-Forwarded-For}i

to your config file. Here is an example setup successfully implementing X forwarded For as well as maintaining logging the IP of the WAF itself (in case you have more than one..)

LogFormat "%v:%p %{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Web

Configure Sophos UTM to forward e-mail for internal clients

Leave a comment

If you’re in a situation like mine you have an ISP which stubbornly blocks port 25. You can get around this by using SSL/TLS of course but configuring each server can be a pain. The solution, in my case, is to turn my UTM into an e-mail relay server and simply have each node point to it for SMTP.

The process wasn’t as straightforward as I had hoped. Below are the steps to accomplish this:

First, enable the SMTP proxy by going to E-mail Protection / SMTP.

Next, go to the Relaying tab and configure allowed relay hosts. For my setup I went to the Host-based relay section and added each individual host that I want to have use my UTM for e-mail relaying.

Lastly, you must configure your UTM to send e-mails out to your ISP’s mail relay since by default the UTM tries to send e-mails out itself on port 25, which in my case is blocked. Do this by going to the Advanced tab and scrolling down to smart host settings. Configure the mail relay server, credentials, and port required there.

That part was fairly straightforward; however the spam protection feature of the UTM was preventing any messages sent to the UTM from being sent to the internet.

After much weeping, wailing and gnashing of teeth (including adding an SPF record for my domain) I discovered that the solution was to ensure that outgoing mail shows the node’s name in the From: field. If your server’s name is server, the from should always be server@yourdomain.org. I haven’t dug into why, but oh well! It works.

CLI

Manipulate EXIF data with jhead

Leave a comment

From time to time I find I need to edit the date taken metadata on pictures. I’ve discovered that jhead is a wonderful tool to accomplish this. It has many options, but the ones I use most frequently are the following:

  • –mkexif Create EXIF data for a picture that does not contain it
  • -ft Set picture’s filesystem modified date to match the EXIF date taken contained in the picture
  • –dsft Set the EXIF date taken date of the picture to match the modified time of the filesystem for that picture

You can use wildcards, which is super convenient. To create metadata for all JPG files in a current directory:

jhead -mkexif *.JPG

I like to use touch in conjunction with jhead to set exif picture taken times for files that don’t have any metadata but I know the date they were taken:

jhead -mkexif *.jpg
touch -t 201410201700 *.jpg
jhead -dsft *.jpg

For pictures which have correct metadata but incorrect modified date (downloaded pictures, for example) simply do the following:

jhead -ft *.jpg

Neat.

Update 7/15/2018

I came across this helpful article which outlines how to view exif information from the command line using the identify command. It requires that imagemagick be installed.

identify -verbose *.jpg | grep "exif:"

Update 12/9/2018

This is the syntax I use to set the time of the picture to a specific date:

jhead -tsYYYY:MM:DD-HH:mm:ss <filename>
CLI

Unzip multiple files into a single directory

Leave a comment

Occasionally I have a need to unzip multiple zip files into a single directory, renaming any files with duplicate names so all zip contents end up in the same directory. I accomplish this in a lazy fashion with find and unzip.

First, put all the zip files you need to extract in the same folder. I used find with the -ctime flag to find files created today (as those are the ones I want.) I use the -exec flag to execute a command on the resulting files; in this case, the unzip program with the -B flag, which doesn’t overwrite files with duplicate names, and the -d flag to specify which folder to extract to:

find *.zip -ctime 1 unzip -B {} -d example/ \;

This finds and unzips all my files, but there is a catch: files with the same filename have been renamed with tildes at the end of each file, for example pic1.jpg~. I do another quick find operation to simply tack .jpg to the end of each of these files

find example/ -name "*~*" -exec mv {} {}.jpg \;

The result is a directory full of the files you desire. My case is very simplistic as it assumes that all files in each zip file are of the same extension. You could use something like awk to parse the result of find command and re-add appropriate extensions, but I won’t detail that here (see laziness reference above.)

 

Virtualization

Convert xenserver .xva file to raw disk image

5 Comments

What if you want to migrate a VM that’s been living on Citrix Xenserver to a different linux machine running vanilla Xen? The process isn’t as straightforward as you might think. Fortunately thanks to Eriklax over at github there is a fairly easy way to convert xenserver’s .xva virtual machines to other formats, via xva-img.

The first step is to download and install xva-img from github.

wget https://github.com/eriklax/xva-img/archive/master.zip
unzip master.zip
cd xva-img-master
cmake .
sudo make install

When trying to compile this on my Linux Mint Cinnamon machine I ran into the following errors:

CMake Error: your CXX compiler: "/usr/bin/c++" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.
xva-img-master/src/sha1.cpp:20:25: fatal error: openssl/sha.h: No such file or directory
 #include <openssl/sha.h>

I had to install the build-essential and libssl-dev packages in order to successfully compile and install xva-img.

Now that it’s installed, create a directory and extract your .xva file into it.

mkdir my-virtual-machine 
tar -xf <.xva file> -C my-virtual-machine 
chmod -R 755 my-virtual-machine

Once that’s finished (it might take a while – it took over an hour for me) the last step is to convert the extracted directories into a raw disk file.

Note:  when you extract your VM tar creates subfolders for each hard disk attached to the VM. You will have to run this command for each Ref folder that was generated as part of the image extraction process.

xva-img -p disk-export my-virtual-machine/Ref\:1/ disk.raw

It took a while for some reason, but it did eventually generate the desired image.

Now that I have a raw disk image I can transfer it to an LVM partition for use with xen:

sudo dd if=win8.1.img of=/dev/desktop-xen/Win8.1 bs=64M

Success.

Virtualization

Convert xenserver installation to software RAID-1

6 Comments

Update 2/28/2015:  I have a newer article explaining how to do this in Xenserver 6.5.

 

After having a hard drive nearly die on me and threaten to obliterate the VMs living on it I realized it would be a good idea to have my xenserver installation live on a RAID array.

Following this guide I was able to successfully migrate my running xenserver installation to a software based RAID 1, with a few tweaks. In my case I wanted to migrate from a single old drive to two newer ones.

Below are the steps I took to accomplish this.

Partition the new drives

This assumes that your current drive resides on /dev/sda, and your two new drives are /dev/sdb and /dev/sdc.

sgdisk -p /dev/sda
sgdisk --zap-all /dev/sdb
sgdisk --zap-all /dev/sdc
sgdisk --mbrtogpt --clear /dev/sdb
sgdisk --mbrtogpt --clear /dev/sdc
sgdisk --new=1:34:8388641 /dev/sdb
sgdisk --new=1:34:8388641 /dev/sdc
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --typecode=1:fd00 /dev/sdc
sgdisk --attributes=1:set:2 /dev/sdb
sgdisk --attributes=1:set:2 /dev/sdc
sgdisk --new=2:8388642:16777249 /dev/sdb
sgdisk --new=2:8388642:16777249 /dev/sdc
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdc

The third partition (VM storage) had to be tweaked a bit since these are larger drives than the current xenserver installation. I simply used gdisk instead of sgdisk for this task.

gdisk /dev/sdb
n #create new partition
<enter> #accept defaults for partition number, first, and last sectors
<enter>
<enter>
t #select partition type
3 #select partition number 3
fd00  #set for raid
w   #write changes to disk

Repeat above steps for the other disk (/dev/sdc in my case)

Create the RAID arrays for each partition

mdadm --create /dev/md0 --level=1 --raid-devices=2  /dev/sdb1 /dev/sdc1
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 /dev/sdc2
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb3 /dev/sdc3

Watch array build (optional)

cat /proc/mdstat

Alternatively you can use the watch command to get a real time update of the raid build:

watch -n 1 cat /proc/mdstat

Format & mount the array

mkfs.ext3 /dev/md0
mount /dev/md0 /mnt

Copy the root filesystem to the new array

cp -vxpr / /mnt

Install bootloader on the new disks

mount --bind /dev /mnt/dev
mount -t sysfs none /mnt/sys
mount -t proc none /mnt/proc
chroot /mnt /sbin/extlinux --install /boot
dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb
dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdc

Generate new initrd image

chroot /mnt
mkinitrd -v -f --theme=/usr/share/splash --without-multipath /boot/initrd-`uname -r`.img `uname -r`
exit

Modify boot file

Edit /mnt/boot/extlinux.conf and replace every mention of the old root filesystem (root=LABEL=xxx) with root=/dev/md0.

vi /mnt/boot/extlinux.conf
:%s/LABEL=<root label>/\/dev\/md0/
:wq

Reboot

Keep the old drive in, but make sure to boot from either one of the member drives of your new array.

Create storage repository

Create new local storage repository with the new RAID array similar to here.

xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=<UUID of xenserver host> name-label="RAID-1" shared=false type=lvm

Migrate VMs / disks

Migrate any disk images or VMs living on the old drive to the new array.

If these VMs / disks are not powered on or being used, it is as simple as pulling up xencenter, right clicking on the VM and clicking move then  select new storage repository.

If the VMs are online you can live migrate them to a different xenserver, then live migrate them back to the proper storage repository.

Remove old storage repository

Following instructions found here.
Note: In my case the transfer returned a strange error but was still successful. I had to restart the XAPI toolstack in order for it to let me remove the old storage repository.

xe sr-list name-label="<name of SR to remove>"
xe pbd-list sr-uuid=<UUID of SR above>
xe pbd-unplug uuid=<UUID of pbd above>
xe sr-forget uuid=<UUID of SR>

Final reboot

Shutdown, disconnect the old drive, and boot back up from the new array. Success.

Configure e-mail alerts (optional)

Now that you have a working RAID array you might want to receive e-mail alerts if there are problems with the array.

First, build an mdadm.conf

mdadm --detail --scan > /etc/mdadm.conf

Modify mdadm.conf to add your desired e-mail address for notifications

sed -i '1i MAILADDR <e-mail address>' /etc/mdadm.conf

Thanks to this site for the sed -i 1i trick.

Lastly, enable the mdadm monitoring service. I found via this site that this is fairly easy to do.  Simply enter these two commands:

service mdmonitor start
chkconfig mdmonitor on

Xenserver uses ssmtp to send e-mail. You can follow this guide on how to set it up for SSL if you happen to have an ISP that blocks port 25 (as I do.) Otherwise modify /etc/ssmtp/ssmtp.conf to suit your needs.

You can generate a test event from mdadm to make sure e-mail is configured properly:

mdadm --monitor --test /dev/md0 --oneshot

To get e-mail alerts to work right I had to ensure that FromLineOverride was NOT set to yes (default). I also had to add this line to /etc/ssmtp/revaliases:

root:<e-mail address being sent from>

Update 02/03/2015:  A commenter made me realize I forgot a step – copying the Control Domain OS to the new Raid array. I’ve added that step above, after the “Format & Mount the array” section.

Update 02/17/2015: If you are using Xenserver 6.5 you might come across the following error message when trying to create RAID arrays:

mdadm: unexpected failure opening /dev/md0

If this happens, load the md kernel driver like so:

modprobe md

It should then let you create your arrays.