Configure WordPress to report true IP addresses when behind a reverse proxy

Placing WordPress behind a Web Application Firewall (reverse proxy) greatly enhances the security of your site; however, it can also be quite a pain.

My most recent experience with hosting WordPress behind a reverse proxy is that all traffic is reported as coming from that reverse proxy regardless of where it truly came from. This is a problem for site statistics, or for my specific example, seeing the source IP address of comment spam.

The fix, fortunately, is fairly simple. It requires tweaking the PHP code just a little bit to use the X_FORWARDED_FOR header, if it exists. Simply add the following to the top of your wp-config.php file (in the base install directory of your WordPress installation)

/** Reverse Proxy configuration
Configures WordPress to show correct source IP addresses when it's behind a reverse proxy
*/
if(isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
        $list = explode(',',$_SERVER['HTTP_X_FORWARDED_FOR']);
        $_SERVER['REMOTE_ADDR'] = $list[0];
  }

That’s all! Future traffic should properly report source IP addresses from that point on. Thanks to the WordPress support forum for helping me find this fix.

Configuring a Cisco 881w from scratch

Below are my rough notes for configuring a Cisco 881W router with a broken WAN port from scratch. The commands below will get it running from a clean slate to a running system.

Enable saving of running config 

Create WAN VLAN

  • int vlan 5
  • name WAN

Enable DHCP on WAN port

  • int vlan 5
  • ip address dhcp
  • no shut

Configure port Fa3 to be the new WAN port

  • int fa3
  • switchport mode access
  • switchport access vlan 5

Create LAN VLAN

Assign remaining ports to Vlan 1

  • int range fa0 – 2
  • switchport mode access
  • switchport access vlan 1
  • no shut

Configure IP address for vlan 1

  • int vlan 1
  • ip address 192.168.1.1 255.255.255.0

Enable ssh

  • hostname mythbox
  • ip domain-name mooo.com
  • aaa new-model
  • username cisco password 0 cisco
  • crypto key generate rsa
  • ip ssh time-out 60
  • ip ssh authentication-retries 2

Configure an enable password

  • enable secret cisco

Create access list allowing only internal subnet for SSH (optional)

  • access-list 23 permit 192.168.1.0 0.0.0.255
  • line vty 0 4
  • access-class 23 in
  • transport input ssh

enable DHCP

  • service dhcp
  • no ip dhcp conflict logging
  • ip dhcp pool myth
  • network 192.168.1.0 /24
  • dns-server 192.168.1.1
  • default-router 192.168.1.1

Enable DNS server

  • ip dns server
  • TODO: Only allow DNS from internal interfaces

Enable PAT overload (NAT) from WAN interface

Configure outside / inside interfaces

  • int vlan 5
  • ip nat outside
  • int vlan 1
  • ip nat inside

Configure access list (config mode)

  • access-list 101 permit ip any any
    (can be more strict with first any by putting local subnet instead)
  • ip nat inside source list 101 interface vlan 5 overload

Disable VTP and STP

Needed if you’re going to use this in an enterprise network

  • no spanning-tree vlan 1
  • no spanning-tree vlan 5
  • vtp mode transparent

Configure Wireless access point

Place wireless on same VLAN as wired network

  • interface wlan-ap0
  • ip unnumbered vlan 1
  • no shut

Switch wireless access point to autonomous mode

  • conf t
  • service-module wlan-ap 0 bootimage autonomous
  • service-module wlan-ap 0 reload

Reset wireless module to default

Necessary if you don’t know the enable password of the current config of the wireless controller

  • service-module wlan-ap0 reset default-config

 Flash wireless module IOS version

Copy IOS image from USB drive to internal flash

  • copy usbflash:<filename> flash:<filename>

Configure TFTP server for the flash file

  • tftp-server flash:<filename>

Console into wireless module

  • ^Z
  • service-module wlan-ap0 session
  • password cisco/cisco if prompted (this is the router user/pass combination)
  • enable password is Cisco
  • escape character is ctrl shift 6 and then the letter x. router prompt will show. type disconnect to exit the wireless console shell

Flash wireless ROM via TFTP

  • archive download-sw overwrite /reload tftp://192.168.1.1/<filename>

Configure Wireless Access Point

Enable HTTP management

  • ip http server

Enable wireless radio

  • int Dot11Radio0
  • no shut

Connect to wireless management page

Determine the IP address of the wireless access point and connect to it using your browser

  • ^Z
  • sh ip interfaces brief

The default WAP page username/password is admin / Cisco

Manual Wireless Configuration

The wireless management page is very clunky. Following guidance from this site, I modified the steps slightly to include WPA 2 with AES encryption to allow for faster wireless N speeds. Replace ssid with desired wireless SSID and testpassword with desired password.

configure terminal
interface Dot11Radio0
encryption vlan 1 mode ciphers aes-ccm
ssid mythbox1
no shutdown
station-role root
exit

dot11 ssid mythbox1
Vlan 1
authentication open
authentication key-management wpa version 2
wpa-psk ascii 0 testpassword
guest-mode
exit

dot11 network-map

interface Dot11Radio0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
bridge-group 1 spanning-disabled
exit

interface GigabitEthernet0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
no bridge-group 1 source-learning
bridge-group 1 spanning-disabled
exit

Splunk regex tips

I’ve spent some time playing around in Splunk trying to refine my dashboards and searches. Here is what I’ve learned (or re-learned) about Splunk and using regular expressions in your searches.

Field extraction syntax

The general formula for using regex to create field extractions is as follows:

(?i)Initial regex match(?P<FIELDNAME>Regex dictating how much to match after initial match)

Example:
(?i)Last Matched Message: (?P<message>(?:[^”]+))

This field extraction searched my logfiles for the string “Last Matched Message: ” It then kept matching every character until it reached a double quote ” and named the extraction “message”. I can now do a “| stats count by message” query in Splunk to cleanly see the values of “Last Matched Message” in my firewall logs.

Regex lookahead

The above regex string utilized a positive lookahead. The syntax for Splunk includes a question mark as expected, but also a colon for some reason (as opposed to an equal sign.) I haven’t looked into why.  Just put  (?:)  in front of your criteria (see above)

Eval

The eval parameter is handy if you want to take information in Splunk and make decisions on it, then display the results in its place. I use it to translate 6 to TCP and 17 to UDP in my firewall logs.

Example:
eval proto = case(proto=”6″,”TCP”,proto=”17″,”UDP”)

Regex: sed

You can use stream editor in Splunk just like you would in Linux. This allows you to modify the output of Splunk results, making them much more useful. The syntax is:
| rex field=fieldname mode=sed “sed syntax

Example:
| rex field=owncloud_file mode=sed “s/\&files\=/\//g”

In this example I take a field I had created (owncloud_file) and then instruct sed to search “s” then look for the string “&files” (with a proper escape character for the &), then replace that string with an equal sign. The g deletes the match so future matches can be made.

The field extraction I have for owncloud looks specifically for the output from the Files app so I can see which files have been downloaded:
(?i)\?dir=(?P<owncloud_file>(?:[^ \”]+))
The regex looks for ?dir= and then matches anything that’s not a double quote.

URLdecode

I made a few sed regex extractions to clean up URLs (replacing %20 for space, etc) when I realized there’s a much easier way to do this: the urldecode function. Simply append the following to your search:

| eval fieldname = urldecode(fieldname)

In my case all I had to do was append
| eval owncloud_file = urldecode(owncloud_file) and voila! all my results look nice and human readable. Magic.

Phew!

I think I’ll stop for now.

 

Xen HVM domU doesn’t synchronize with dom0 clock

After much research I’ve discovered that Xen does not synchronize the clock between dom0 and its HVM domUs. This poses a problem when you implement S3 sleep. Upon resume,  dom0 realizes how much time has passed but none of the domUs do. I realized this after a few days of successfully putting my Xen machine to sleep with running DomU virtual machines

The DomU in my case is a Windows 8.1 virtual machine. At first I thought that the standard Windows time service would take care of any clock discrepancies – it doesn’t. If your clock gets too far behind it simply refuses to update. My solution to this problem is two fold:

  1. Configure Windows to use my NTP server for clock updates
  2. Force Windows to check with the NTP server every minute and update its clock accordingly.

Fortunately the later Windows versions have an NTP client built in. Simply open an administrator command prompt and issue two commands:

w32tm /config /syncfromflags:manual /manualpeerlist:<hostname>

schtasks /create /sc minute /mo 1 /tn "NTP clock update" /tr "%WINDIR%\system32\w32tm.exe /resync /force" /RU SYSTEM

The first command configures your system with your NTP server of choice. Replace <hostname> with your desired hostname or IP address, minus the brackets. The second command creates a task which executes a command to force an NTP check every minute as the SYSTEM user (non-privileged users get an access denied message.) You can do it all with a GUI but the command line is so much more efficient 🙂

It works perfectly. My DomU now automatically checks if it has the correct time – very important if you ever put your dom0 to sleep while DomUs are running.

Fix images in wordpress posts after address change

In my experiments with WordPress I have discovered an annoying complication when it comes to changing servers. When you change the address of your wordpress site, links to images and other files inside posts are not updated to point to the new wordpress URL. Links to the posts themselves are all updated but for whatever reason WordPress ignores post content.

What this means is if you transform a staging server (say, http://server) into a “production” server (say, http://jeppson.org) posts containing links to any files on that server will now be (silently!) broken. The files are all still there, the posts just point to the old address instead of the new one.

There are numerous plugins for WordPress that can take care of this; however I am averse to installing random WordPress plugins due to security concerns. Fortunately, the fix for this vexing problem is a fairly simple one. It requires executing a command on the MySQL database hosting the wordpress site.

The table we are interested in is the <prefix>_posts table. <prefix> was defined when the wordpress site was set up. If you don’t remember what that prefix was, take a peek at /wp-config.php line 62

Capture

The field we are interested in is post_content – this is where all the links to pictures in your various posts resides. From here execute the update command and use the replace function to find all instances of the old URL and update them with the new URL

Capture

That’s it! Your images should now re-appear. The two commands I used in my specific example are below.  Be sure to replace the following :

  • wordpress with the name of your wordpress database
  • nm_posts with the name of your site’s post table,
  • http://server with your original wordpress URL and
  • http://jeppson.org with your new wordpress URL
mysql -u root -p wordpress
update nm_posts set post_content = replace(post_content, 'http://server' , 'http://jeppson.org');

 


 

Update: I’ve discovered a very handy script that will take care of this for you: wp-cli

To do the same thing as above with less steps, simply download wp-cli

curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar

Make the script executable and move it to /usr/bin

chmod +x wp-cli.phar
sudo mv wp-cli.phar /usr/local/bin/wp

Navigate to the directory of your wordpress install and run wp search-replace

cd /var/www/wordpress
wp search-replace 'http://server' 'http://jeppson.org'

Easy.

Capture

Put Xen dom0 to sleep with active pci passthrough VMs

Thanks to Xen 4.3, it is now possible to suspend / resume dom0 while domUs are running. Unfortunately, if you have a VM actively using pci passthrough, the whole machine completely locks up about 10 seconds after resuming from S3 sleep.

As Xen is a lot more geared to servers I realize I might be an edge case; However, I would really like to be able to suspend my entire machine to S3 with VMs actively using PCI passthrough (in my case, a video card and USB controller). For quite some time I thought I was out of luck. After learning about the hot swapping capabilities of Xen 4.2+ and its pciback driver, I thought I would take another whack at it.

My solution is to create a custom script which detaches all PCI passthrough devices on the VM before going to sleep. That same script would re-attach those devices to my VM on resume.

My dom0 is currently Linux Mint 16 so I placed the resulting script in the /etc/pm/sleep.d/ directory and named it 20_win8.1 . It works like a charm! I can suspend and resume to my heart’s content without having to worry about if I remembered to shut down my VM first.

My script is below. Be sure to modify it for the BDF of your devices and the name of your VM(s) if you decide to use it.

#!/bin/bash
#Sleep / hibernate script for Xen with active DomUs using PCI Passthrough
#This script is necessary to avoid freezing of dom0 on resume for Xen 4.3
#Modified 08/19/2014

#Name of the VM we're passing PCI things to
VM="win8.1"

#B:D.F of PCI devices passed through to VM
VIDCARD="01:00.0"
VIDCARDAUDIO="01:00.1"
FRONTUSB="00:1d.0"

#xen attach/detach commands. Replace with xm if you're using that toolstack instead
ATTACH="xl pci-attach"
DETACH="xl pci-detach"

case "$1" in
    hibernate|suspend)
        $DETACH $VM $VIDCARD
        $DETACH $VM $VIDCARDAUDIO
        $DETACH $VM $FRONTUSB
        ;;
    thaw|resume)
        $ATTACH $VM $VIDCARD
        $ATTACH $VM $VIDCARDAUDIO
        $ATTACH $VM $FRONTUSB
        ;;
    *)
        ;;
esac
exit $?

Hotplug devices between Xen dom0, domU, and back again

In my experiments with Xen to make dual booting obsolete,  I’ve come across a need to hotplug PCI devices between dom0 and domU; Specifically, the SATA controller that my DVD-RW drive is connected to.

My DVD drive supports Lightscribe, which unfortunately is not nearly as strong in Linux as it is in Windows. You can get it to work but the label maker program is extremely basic. If I want to burn a lightscribe disc and have it look at all pretty it requires Windows.

The way I was doing PCI passthrough before was pretty inconvenient. It involved editing /etc/xen/pciback.conf and adding the bus:device.function (BDF) of the device I want to pass. This causes that device to be claimed by the pciback driver at boot time.

That’s all and well and good for the virtual machine, but what if you want your dom0 to use that device? You would have to remove the device from pciback.conf and reboot the machine.

As of Xen 4.2 there is now a better way.  You can have the pciback driver claim a device and return it to its original driver at any time without having to reboot.  The four magic commands are:

xl pci-assignable-add <BDF>
xl pci-attach <domain id / name> <BDF>
xl pci-detach <domain id / name> <BDF>
xl pci-assignable-remove -r <BDF>

The -r in pci-assignable-remove is necessary – it instructs xen to load the original driver that was loaded before we invoked pci-assignable-add. If you are using the xm toolstack instead, simply replace xl with xm.

Detaching from Dom0 and attaching to DomU

In my case I enter the following into a console whenever I want my Windows 8.1 virtual machine to have physical control of my DVD drive:


sudo xl pci-assignable-add 03:00.0
sudo xl pci-attach win8.1 03:00.0


Windows specific issues

It should have been as simple as that; Unfortunately, I ran into a road block. For some reason on the first try, Windows detected the drive but wouldn’t load any drivers for it (it thought none were necessary)

Screenshot from 2014-08-17 15:08:08
(this screenshot was taken when I was using a hard drive for troubleshooting, but the issue was the same with the DVD drive)

I tried ejecting the SATA controller and scanning for new devices as described on various forums, but that didn’t seem to work. The fix for me was to reboot the VM. Rebooting caused the PCI device to detach, so after the VM finished rebooting I had to re-issue “sudo xl pci-attach win8.1 03:00.0” to attach it again.  Triumph!

Screenshot from 2014-08-17 15:20:17

I tried to make the second pci-attach command unnecessary by adding pci=03:00.0 to my virtual machine’s configuration file, but since I was passing a storage controller it kept trying to boot from drives attached to that controller instead of the virtual machine’s hard drive. I tinkered around with the config file for a while to try and get it to boot from the VMs hard drive again but couldn’t get it to work.

Since everything works by simply issuing pci-attach twice I gave up and just moved on. In one final bout of tinkering I discovered that if you issue pci-attach right after you boot the VM but before the OS finishes loading, it works on the first try. So the moral of the story here is Microsoft weirdness requires you to jump through some minor hoops to get this to work.

Returning to Dom0

When I want my dom0 to have the drive back I issue the following:


sudo xl pci-detach win8.1 03:00.0
sudo xl pci-assignable-remove -r 03:00.0


No complications here, although there is a funny bug. The file manager used in Linux Mint 16 gets confused and keeps adding CD ROM entries each time I pass the drive back and forth, but everything still works – it’s just a visual bug.

The drive is now accessible by dom0 once again. Success!

Screenshot from 2014-08-17 15:47:31

 

 

 

Xenserver and clock drift

When it comes to a virtual machine’s clock my experience with other virtualization solutions has been that it’s automatically synchronized with the host machine. I didn’t notice until recently that this is not the case with Citrix Xenserver – at least when it comes to PVHVM machines.

I tried installing openntpd on each of my VMs and setting it to my internal NTP server (which in turn synchronizes with the web.) After a few days I was frustrated to see that the servers were still not in sync – some were minutes behind while others were inexplicably minutes ahead. Some of this might have to do with my experiments on live migrating these VMs a while back.

At any rate, it was clear that openntpd failed to do the job. Some research revealed that there is a bug where it reports adjusting the clock when the real status was an error. That little bug cost me an hour or two of digging and troubleshooting. Very frustrating.

I switched to plain old ntp instead and the problem was resolved within moments.

clock

Moral of the story: Make sure you have a proper NTP setup for each of your VMs if you’re going to use Citrix Xenserver.

Reverse search result order in Splunk

When you run a query in Splunk it returns the most recent result at the top of the screen by default.

normal

For far too long now I have been running queries in Splunk and then manually clicking back to the last page of results so that I can see the first time something happened.

It turns out there is a better way. Simply append ” | reverse” (without quotes) to the end of your search result. This will cause the earliest search result to be at the top, rather than the most recent. Handy.

reverse

Changing permalink structure in WordPress

After installing a fresh installation of WordPress on LAMP (Linux, Apache, MySQL, and PHP) I decided to change the new site’s permalink structure. The default structure on WordPress is so boring and uninformative  – sitename/?p=(number). I prefer the blogger method – sitename/year/month/postname.

Changing the structure should be pretty straightforward; However, when I clicked that option and clicked save, every post greeted me with a wonderful 404 error page.

Screenshot from 2014-08-01 20:33:54

If you are running apache it turns out there are a few extra things you need to do that aren’t mentioned in WordPress’s 5 minute install guide. Detailed instructions are found here.

  1. Enable the mod-rewrite apache plugin
  2. Ensure the directory of your wordpress installation has a Directory entry and that AllowOverride All is enabled
  3. Restart apache

After this, everything worked. The reason behind needing this change is WordPress modifies the .htaccess file for its installation folder with a simple rule:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ – [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

All it does is transparently prepend index.php to any web request it receives. Index.php is smart enough to direct pages wherever it needs to after that. If you don’t want to mess with enabling mod_rewrite, you can simply change your site structure to have /index.php/ before everything else. It will accomplish the same thing, only now index.php will show up in all your site URLs.