CLI

Creating a ZFS RaidZ volume with different sized disks

Leave a comment

While I hear that “ZFS likes to use the whole disk” I wanted to experiment with creating a RaidZ pool with disks of different sizes. This requires partitioning the larger disks. The GUI in FreeNAS does not allow you to do this, so we must venture toward the command line. While these commands were run in FreeNAS they will work in any FreeBSD based system.

The commands below assume you are using the first four disks in the system for the RaidZ pool. I realize you can make these commands more efficient by using shell-fu but I will put them all here for completeness.

Partition the disks

Create GPT table for each disk

  • gpart create -s gpt ada1
  • gpart create -s gpt ada2
  • gpart create -s gpt ada3
  • gpart create -s gpt ada4

If gpart complains (probably due to the disk already having a GPT table) you can nuke the GPT setup and start over via the following commands, replacing ada2 with the stubborn drive:

  •  gpart destroy -F ada2
  • gpart create -s gpt ada2

Create partition for each disk

  • gpart add -s 232G -t freebsd-zfs -l test0 ada1
  • gpart add -s 232G -t freebsd-zfs -l test0 ada2
  • gpart add -s 232G -t freebsd-zfs -l test0 ada3
  • gpart add -s 232G -t freebsd-zfs -l test0 ada4

Create the pool

  • zpool create storage raidz ada1p1 ada2p1 ada3p1 ada4p1

Why does everyone call their pool “tank”? It must be in some documentation somewhere that everyone copies.

If you want to replace a failed disk in a pool after the faulty disk has been removed, issue the following command:

  • zpool replace storage <old/failed device name> ada2p1

Sometimes even though you’ve nuked the gpt data of the disk zpool will complain about the disk already being a member of a pool, e.g. “/dev/ada2p1 is part of active pool ‘storage'” Another scenario is if you have properly replaced the failed drive but the pool still shows degraded with a hash referring to the old drive showing in zpool status. To fix these issues use the zpool detach command

zpool detach storage ada2p1

 Check pool status

  • zpool status

If it comes out healthy you are good to go.

Import the raidZ into FreeNAS

With the pool manually created you can now import it into FreeNAS so it can be monitored / managed.

  • Click on Storage / Volumes / Auto Import Volume
  • Click No, skip to import
  • Wait a minute for it to scan, then click OK

Done.

Virtualization

Create local storage in Xenserver

1 Comment

For some reason the default installation of Xenserver on one of my machines did not create a local storage repository. I think it might be due to my having installed over an existing installation of Xenserver and the installer got confused.

I tried manually creating a storage repository by running the following command:

xe sr-create content-type=user device-config:device=/dev/disk/by-id/scsi-SATA_WDC_WD3200AAJS-_WD-WMAV2C718714-part3 host-uuid=9f8ddd87-0e83-4322-8150-810d2b365d37 name-label="Local Storage" shared=false type=lvm

Alas, it resulted in an error:

Error code: SR_BACKEND_FAILURE_55
Error parameters: , Logical Volume partition creation error [opterr=error is 5],

After much googling I came across this page, which has the explanation. Apparently you need to create an LVM physical volume on the desired partition by running the following command:

pvcreate /dev/disk/by-id/scsi-SATA_WDC_WD3200AAJS-_WD-WMAV2C718714-part3

WARNING: software RAID md superblock detected on /dev/disk/by-id/scsi-SATA_WDC_WD3200AAJS-_WD-WMAV2C718714-part3. Wipe it? [y/n] y

It seems the installer noticed an md superblock on this partition and freaked out, hence no local storage. Agreeing to wipe it created the storage repository. One last step: making it the default repository:

xe pool-param-set uuid=<pool UUID> default-SR=<SR UUID>

You can get the pool UUID by running: xe pool-list

Done.

Edit: 10/09/2014

I recently came across a new error message when trying to add a local repository:

The SR operation cannot be performed because a device underlying the SR is in use by the host.

Google searches didn’t reveal much. After a while I realized what was wrong: I had omitted the host-uuid: option. This option is required when you are a part of a pool, but not when you have a standalone xenserver. So, if your xenserver is a member of a pool, don’t forget the host-uuid parameter.

Virtualization

Manually apply patches to Citrix Xenserver

Leave a comment

Citrix Xenserver has many features, all of which are now free as of Xenserver 6.2. XenCenter, however, still expects a support license to use some of its features. One of those features is applying patches. Fortunately it’s easily done via the command line. Their site has documentation on how to do this. Below are my “cliff notes”

  1. xe patch-upload file-name=<filename>
    Note: .xsupdate is the extension of xenserver updates
  2. Wait a moment, then copy the UUID that it outputs
  3. xe host-list
  4. xe patch-apply uuid=<UUID copied from patch-upload>  host-uuid=<host UUID as out put from xe host-list>

If you’re in a pool, instead of xe patch-apply, you can do xe patch-pool-apply <UUID> to apply the patch to all pool members.

Web

Configure SSMTP to use SSL/TLS connections

1 Comment

SSMTP is a very simple SMTP mail program which is used to send e-mails to a target server. It’s not a fully feature e-mail server but simply passes e-mails on. I first became acquainted with it because it’s the only mail server you can install on Citrix Xenserver. I now use it with all my servers because it’s very easy to configure.

Simply install it via command line:

sudo apt-get install ssmtp

There is only one config file to worry about: /etc/ssmtp/ssmtp.conf. To configure it to use an SSL connection (for gmail or if, like me, your ISP blocks port 25), add the following options, changing the brackets with your mailserver, username, and password.

mailhub=<mailserver>:587
UseSTARTTLS=YES
AuthUser=<username>
AuthPass=<password>
AuthMethod=DIGEST-MD5

If you just pasted the above config into your ssmtp.conf be sure to check the resulting config file for duplicate entries.

It’s as simple as that. All outgoing mail will be sent to the server specified above.

Mobile

Rooting and flashing Verizon Galaxy S4 VRUFNC5

1 Comment

Below is my experience with rooting and flashing a newer ROM on my Verizon Wireless Samsung Galaxy S4. Thanks to a recently discovered kernel exploit (both scary and awesome) rooting was the easy part. Thanks to the encrypted bootloader on my phone and the KitKat update which made it impenetrable (at least for now) getting a new ROM on the phone was a little more difficult. Thanks to rootjunky.com for the informative video guide.

  1. Root the phone with towelroot.
    Simply navigate to the site, click the lambda to download the towelroot APK, then copy to your device and install it.
  2. Install SuperSU from the Google Play store
  3. Install Android Terminal Emulator
  4. Set selinux to permissive mode
    Open terminal editor and type the following:
    su
    setenforce 0
  5. Install busybox
  6. Install Safestrap 3.72 (the ATT version works fine)
  7. Reboot into safestrap and backup the current ROM (optional, but recommended)
  8. Install ROM of choice via Safestrap on the Stock ROM slot (other slots don’t work as of this writing.)
    I chose the hyperdrive ROM
  9. Flash safestrap KitKat module to fix Wireless functionality
  10. Remove “Press and Hold to add items to launcher screen” by dragging a widget from one screen to another

Success.

Screenshot_2014-08-26-08-36-16

 

 

Web

Configure WordPress to report true IP addresses when behind a reverse proxy

Leave a comment

Placing WordPress behind a Web Application Firewall (reverse proxy) greatly enhances the security of your site; however, it can also be quite a pain.

My most recent experience with hosting WordPress behind a reverse proxy is that all traffic is reported as coming from that reverse proxy regardless of where it truly came from. This is a problem for site statistics, or for my specific example, seeing the source IP address of comment spam.

The fix, fortunately, is fairly simple. It requires tweaking the PHP code just a little bit to use the X_FORWARDED_FOR header, if it exists. Simply add the following to the top of your wp-config.php file (in the base install directory of your WordPress installation)

/** Reverse Proxy configuration
Configures WordPress to show correct source IP addresses when it's behind a reverse proxy
*/
if(isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
        $list = explode(',',$_SERVER['HTTP_X_FORWARDED_FOR']);
        $_SERVER['REMOTE_ADDR'] = $list[0];
  }

That’s all! Future traffic should properly report source IP addresses from that point on. Thanks to the WordPress support forum for helping me find this fix.

Networking

Configuring a Cisco 881w from scratch

1 Comment

Below are my rough notes for configuring a Cisco 881W router with a broken WAN port from scratch. The commands below will get it running from a clean slate to a running system.

Enable saving of running config 

Create WAN VLAN

  • int vlan 5
  • name WAN

Enable DHCP on WAN port

  • int vlan 5
  • ip address dhcp
  • no shut

Configure port Fa3 to be the new WAN port

  • int fa3
  • switchport mode access
  • switchport access vlan 5

Create LAN VLAN

Assign remaining ports to Vlan 1

  • int range fa0 – 2
  • switchport mode access
  • switchport access vlan 1
  • no shut

Configure IP address for vlan 1

  • int vlan 1
  • ip address 192.168.1.1 255.255.255.0

Enable ssh

  • hostname mythbox
  • ip domain-name mooo.com
  • aaa new-model
  • username cisco password 0 cisco
  • crypto key generate rsa
  • ip ssh time-out 60
  • ip ssh authentication-retries 2

Configure an enable password

  • enable secret cisco

Create access list allowing only internal subnet for SSH (optional)

  • access-list 23 permit 192.168.1.0 0.0.0.255
  • line vty 0 4
  • access-class 23 in
  • transport input ssh

enable DHCP

  • service dhcp
  • no ip dhcp conflict logging
  • ip dhcp pool myth
  • network 192.168.1.0 /24
  • dns-server 192.168.1.1
  • default-router 192.168.1.1

Enable DNS server

  • ip dns server
  • TODO: Only allow DNS from internal interfaces

Enable PAT overload (NAT) from WAN interface

Configure outside / inside interfaces

  • int vlan 5
  • ip nat outside
  • int vlan 1
  • ip nat inside

Configure access list (config mode)

  • access-list 101 permit ip any any
    (can be more strict with first any by putting local subnet instead)
  • ip nat inside source list 101 interface vlan 5 overload

Disable VTP and STP

Needed if you’re going to use this in an enterprise network

  • no spanning-tree vlan 1
  • no spanning-tree vlan 5
  • vtp mode transparent

Configure Wireless access point

Place wireless on same VLAN as wired network

  • interface wlan-ap0
  • ip unnumbered vlan 1
  • no shut

Switch wireless access point to autonomous mode

  • conf t
  • service-module wlan-ap 0 bootimage autonomous
  • service-module wlan-ap 0 reload

Reset wireless module to default

Necessary if you don’t know the enable password of the current config of the wireless controller

  • service-module wlan-ap0 reset default-config

 Flash wireless module IOS version

Copy IOS image from USB drive to internal flash

  • copy usbflash:<filename> flash:<filename>

Configure TFTP server for the flash file

  • tftp-server flash:<filename>

Console into wireless module

  • ^Z
  • service-module wlan-ap0 session
  • password cisco/cisco if prompted (this is the router user/pass combination)
  • enable password is Cisco
  • escape character is ctrl shift 6 and then the letter x. router prompt will show. type disconnect to exit the wireless console shell

Flash wireless ROM via TFTP

  • archive download-sw overwrite /reload tftp://192.168.1.1/<filename>

Configure Wireless Access Point

Enable HTTP management

  • ip http server

Enable wireless radio

  • int Dot11Radio0
  • no shut

Connect to wireless management page

Determine the IP address of the wireless access point and connect to it using your browser

  • ^Z
  • sh ip interfaces brief

The default WAP page username/password is admin / Cisco

Manual Wireless Configuration

The wireless management page is very clunky. Following guidance from this site, I modified the steps slightly to include WPA 2 with AES encryption to allow for faster wireless N speeds. Replace ssid with desired wireless SSID and testpassword with desired password.

configure terminal
interface Dot11Radio0
encryption vlan 1 mode ciphers aes-ccm
ssid mythbox1
no shutdown
station-role root
exit

dot11 ssid mythbox1
Vlan 1
authentication open
authentication key-management wpa version 2
wpa-psk ascii 0 testpassword
guest-mode
exit

dot11 network-map

interface Dot11Radio0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
bridge-group 1 spanning-disabled
exit

interface GigabitEthernet0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
no bridge-group 1 source-learning
bridge-group 1 spanning-disabled
exit
Analytics

Splunk regex tips

Leave a comment

I’ve spent some time playing around in Splunk trying to refine my dashboards and searches. Here is what I’ve learned (or re-learned) about Splunk and using regular expressions in your searches.

Field extraction syntax

The general formula for using regex to create field extractions is as follows:

(?i)Initial regex match(?P<FIELDNAME>Regex dictating how much to match after initial match)

Example:
(?i)Last Matched Message: (?P<message>(?:[^”]+))

This field extraction searched my logfiles for the string “Last Matched Message: ” It then kept matching every character until it reached a double quote ” and named the extraction “message”. I can now do a “| stats count by message” query in Splunk to cleanly see the values of “Last Matched Message” in my firewall logs.

Regex lookahead

The above regex string utilized a positive lookahead. The syntax for Splunk includes a question mark as expected, but also a colon for some reason (as opposed to an equal sign.) I haven’t looked into why.  Just put  (?:)  in front of your criteria (see above)

Eval

The eval parameter is handy if you want to take information in Splunk and make decisions on it, then display the results in its place. I use it to translate 6 to TCP and 17 to UDP in my firewall logs.

Example:
eval proto = case(proto=”6″,”TCP”,proto=”17″,”UDP”)

Regex: sed

You can use stream editor in Splunk just like you would in Linux. This allows you to modify the output of Splunk results, making them much more useful. The syntax is:
| rex field=fieldname mode=sed “sed syntax”

Example:
| rex field=owncloud_file mode=sed “s/\&files\=/\//g”

In this example I take a field I had created (owncloud_file) and then instruct sed to search “s” then look for the string “&files” (with a proper escape character for the &), then replace that string with an equal sign. The g deletes the match so future matches can be made.

The field extraction I have for owncloud looks specifically for the output from the Files app so I can see which files have been downloaded:
(?i)\?dir=(?P<owncloud_file>(?:[^ \”]+))
The regex looks for ?dir= and then matches anything that’s not a double quote.

URLdecode

I made a few sed regex extractions to clean up URLs (replacing %20 for space, etc) when I realized there’s a much easier way to do this: the urldecode function. Simply append the following to your search:

| eval fieldname = urldecode(fieldname)

In my case all I had to do was append
| eval owncloud_file = urldecode(owncloud_file) and voila! all my results look nice and human readable. Magic.

Phew!

I think I’ll stop for now.

 

Virtualization

Xen HVM domU doesn’t synchronize with dom0 clock

Leave a comment

After much research I’ve discovered that Xen does not synchronize the clock between dom0 and its HVM domUs. This poses a problem when you implement S3 sleep. Upon resume,  dom0 realizes how much time has passed but none of the domUs do. I realized this after a few days of successfully putting my Xen machine to sleep with running DomU virtual machines. 

The DomU in my case is a Windows 8.1 virtual machine. At first I thought that the standard Windows time service would take care of any clock discrepancies – it doesn’t. If your clock gets too far behind it simply refuses to update. My solution to this problem is two fold:

  1. Configure Windows to use my NTP server for clock updates
  2. Force Windows to check with the NTP server every minute and update its clock accordingly.

Fortunately the later Windows versions have an NTP client built in. Simply open an administrator command prompt and issue two commands:

w32tm /config /syncfromflags:manual /manualpeerlist:<hostname>

schtasks /create /sc minute /mo 1 /tn "NTP clock update" /tr "%WINDIR%\system32\w32tm.exe /resync /force" /RU SYSTEM

The first command configures your system with your NTP server of choice. Replace <hostname> with your desired hostname or IP address, minus the brackets. The second command creates a task which executes a command to force an NTP check every minute as the SYSTEM user (non-privileged users get an access denied message.) You can do it all with a GUI but the command line is so much more efficient 🙂

It works perfectly. My DomU now automatically checks if it has the correct time – very important if you ever put your dom0 to sleep while DomUs are running.