Monthly Archives: April 2015

CLI

Use BASH to append an incremental suffix to directories

Leave a comment

In migrating Splunk indexes I came across the need to add an incrementing suffix of _(numbers) to a bunch of directories. For example, renaming directories

test
test1
test2
test3

to

test_100
test1_101
test2_102
test3_103

After a bunch of searching I settled on this approach of using a simple bash loop to accomplish this:

I=100; for F in db_*; do mv $F $F\_$I; I=$((I+1)); done

There is an initial variable, I, which is set to 100. The for loop goes through each file beginning with db_ and then renames it adding the proper suffix in the iteration (_###). The last step is to increment the value of I.

Replace I for whatever you want you suffix to begin with and change db_* with whatever the criteria for the files you want to rename are. Simple, but it works.

Web

Block annoying bots with Apache .htaccess

Leave a comment

Recently one of my sites has been having its database crash repeatedly. Investigation reveals it always happens while an aggressive bot is crawling it. Since the site is small it was causing the database to run out of memory and die.

The Web Application Firewall that this site is behind frustratingly does not have a feature for blocking user agents. I decided to resort to Apache on the webserver itself to serve as a gatekeeper. The user agent in question? flipboard proxy. It also conveniently appears to ignore robots.txt.

Thanks to this article I learned the details on how to get Apache to block this particular user agent. Creating an .htaccess file (if it doesn’t already exist) and putting it in the root directory of the website causes it to apply to the entire site. Within the .htaccess file, place the following directives:

#block bad bots with a 403
#SetEnvIfNoCase User-Agent "facebookexternalhit" bad_bot
SetEnvIfNoCase User-Agent "Twitterbot" bad_bot
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "MetaURI" bad_bot
SetEnvIfNoCase User-Agent "mediawords" bad_bot
SetEnvIfNoCase User-Agent "FlipboardProxy" bad_bot
<Limit GET POST HEAD>
 Order Allow,Deny
 Allow from all
 Deny from env=bad_bot
</Limit>

Save the file in the root of your website and make sure its permissions are such that your apache server can read the file. Success! Flipboard proxy (and other bad bots) no longer crashes the site. Instead, it gets served a 403 – Forbidden page for every request it makes.

Networking

Configure full VPN tunnel in Sophos UTM

Leave a comment

For years now I have had a successful split tunnel VPN with my Sophos UTM. Recently I’ve wanted to have a full tunnel option for greater security in remote areas (hotel wi-fi, etc.) Unfortunately setting up such a thing in Sophos is NOT straightforward.

The biggest problem I had was that no websites would work after the VPN was initiated. NSlookup was fine, connection was fine, even internal sites would load properly, but no external internet.

Thanks to this post I finally found the culprit: the pesky allowed networks feature for each UTM function. In my case, the VPN was allowing all the necessary traffic through but my transparent proxy was denying web access. I had to add my VPN pool to the list of allowed networks to my proxy.

To summarize, this is what you must do to have a full VPN tunnel:

  • Configure the desired method in the Remote Access section. Take note of whatever IP pool you use for your VPN. In my case I used VPN Pool (SSL)
  • Ensure that internet access is in the list of allowed networks for the user you’ve configured for VPN (Any, or Internet IPv4/6)
  • Add your VPN pool to the list of allowed networks for each service you use.
    • Network services / DNS
    • Web Protection / Web Filtering
  • Profit
OS, Virtualization

Slow Linux VM performance in VMware vSphere

Leave a comment

Recently I’ve been scratching my head over a particular performance issue with Linux VMs hosted on VMWare vSphere. Everything seemed to move at a glacial pace.

vmstat gave a few clues as to what was happening, although depending on what I read it still wasn’t clear:

vmstat

It became apparent that I was suffering from some kind of queuing problem. I wasn’t sure if it was CPU or disk related. I came across this post which has a lot of good performance tuning guides.This tip caught my eye:

 

7. Set your disk scheduling algorithm to ‘noop’

The Linux kernel has different ways to schedule disk I/O, using schedulers like deadline, cfq, and noop. The ‘noop’ — No Op — scheduler does nothing to optimize disk I/O. So why is this a good thing? Because ESX is also doing I/O optimization and queuing! It’s better for a guest OS to just hand over all the I/O requests to the hypervisor to sort out than to try optimizing them itself and potentially defeating the more global optimizations.

You can change the kernel’s disk scheduler at boot time by appending:

elevator=noop

to the kernel parameters in /etc/grub.conf.

Sure enough, I modified /boot/grub/grub.conf on my Centos 6 boxes and appended elevator=noop to the kernel line, then rebooted. It helped a lot! Performance no longer was pitiful. I’m not nearly as familiar with vmware as I am with Xenserver so this was a good hint.