Tag Archives: linux

Analytics, CLI, OS

Change the hostname on a Splunk Indexer

Leave a comment

Recently I set about to change the hostname on a Splunk indexer. It should be pretty easy, right? Beware. It can be pretty nasty! Below is my experience.

I started with the basics.

  • hostname command 
    hostname <newhostname>
  • Modify /etc/system/network to make it persistent (CentOS specific) 
    sed -i 's/<old hostname>/<new hostname>/g' /etc/system/network
  • Inform Splunk of the hostname change 
    sed -i 's/<old hostname>/<new hostname>/g' $SPLUNK_HOME/etc/system/local/server.conf
  • Restart Splunk

Sadly, that wasn’t the end of it. I noticed right away Splunk complained of a few things:

TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 300 seconds.
WARN TcpOutputFd - Connect to 10.0.0.10:9997 failed. Connection refused

Running

netstat -an | grep LISTEN

revealed that the server was not even listening on 9997 like it should be. I found this answer indicating it could be an issue with DNS tripping up on that server. I edited $SPLUNK_HOME/etc/system/local/inputs.conf with the following:

[splunktcp://9997]
connection_host = none

but I also noticed that after I ran the command a short time later it was no longer listening on 9997. Attempting to telnet from the forwarder to the indexer in question revealed the same results – works at first, then quit working. Meanwhile no events are getting stored on that indexer.

I was pulling my hair out trying to figure out what was happening. Finally I discovered this gem on Splunk Answers:

Are you using the deployment server in your environment? Is it possible your forwarders’ outputs.conf got deployed to your indexer?

On the indexer:
./splunk cmd btool outputs list –debug

Sure enough! after running

./splunk cmd btool outputs list --debug

I discovered this little gem of a stanza:

/opt/splunk/etc/apps/APP_Forwarders/default/outputs.conf [tcpout]

That shouldn’t’ have been there! Digging into my deployment server I discovered that I had a server class with a blacklist, that is, it included all deployment clients except some that I had listed. The blacklist had the old hostname, which meant when I changed the indexer’s hostname it no longer matched the blacklist and thus was deployed a forwarder’s configuration, causing a forwarding loop. My indexer was forwarding back to the forwarder everything it was getting from the forwarder, causing Splunk to shut down port 9997 on the offending indexer completely.

After getting all that set up I noticed Splunk was only returning searches from the indexers whose hostnames I had not changed. Everything looked good in the distributed search arena – status was OK on all indexers; yet I still was not getting any results from the indexer whose name I had changed, even though it was receiving events! This was turning into a problem. It was creating a blind spot.

Connections great, search status great, deployment status good.. I didn’t know what else to do. I finally thought to reload Splunk on the search head that had been talking to the server whose name I changed. Success! Something in the search head must have made it blind to the indexer once its name had changed. Simply restarting Splunk on the search head fixed it.

In short, if you’re crazy enough to change the name of one of your indexers in a distributed Splunk environment, make sure you do the following:

  • Change hostname on the OS
  • Change ServerName in Splunk config files
    • Add connection_host = none in inputs.conf (optional?)
  • Clean up your deployment server
    • Delete old hostname from clients phoning home
    • MAKE SURE the new hostname won’t be sucked up into an unwanted server class
  • Clean up your search head
    • Delete old hostname search peer
    • Add new hostname search peer
    • Restart search head
  • Profit
CLI, OS

Configure VMWare View Smartcard in Ubuntu

Leave a comment

Recently I’ve been required to use a smart card to log into some servers I manage. Configuring my Linux Mint 17.2 machine to pass my smartcard through to those machines via VMWare View has not been straightforward. This guide will walk you through how to get Smartcard redirection to work with VMWare View in Ubuntu 14.04 Trusty Tahr, which Linux Mint 17.2 is based off of. Enjoy.

Procedure

  1. Install the latest version of the VMWare View client (distro versions are often quite out of date) from here 
    chmod +x VMware-Horizon-Client-3.5.0-2999900.x64.bundle 
sudo ./VMware-Horizon-Client-3.5.0-2999900.x64.bundle
  2. Install necessary packages for CommonAccessCard (thanks to this helpful ubuntu writeup) 
    sudo apt-get install libpcsclite1 pcscd pcsc-tools
  3. (re)Start the pcscd daemon 
    sudo /etc/init.d/pcscd restart
  4. Ensure your smartcard reader is properly identified by running this command: 
    pcsc_scan

    If that command is stuck on “Waiting for the first reader…” then you need to install your smartcard drivers. If it sees your smartcard, skip this next step and proceed to step 6.

  5. Install your smartcard driver. This process is different for each card. For the card reader I have (the Identive SCR3500 A Contact Reader), I was able to obtain the drivers after much difficulty from here. The link to the drivers itself are here (alternate link). In my case I was able to untar and run the install script, which worked beautifully.
  6. Install 32 bit compatibility libraries (only applicable for 64 bit installations) thanks to this site for the answer and this one for clarification 
    sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install -y libxml2:i386 libssl1.0.0:i386 libXtst6:i386 libudev1:i386 libpcsclite1:i386 libtheora0:i386 libv4l-0:i386 libpulse0:i386
sudo ln -sf /lib/i386-linux-gnu/libudev.so.1 /lib/i386-linux-gnu/libudev.so.0
sudo ln -sf /lib/i386-linux-gnu/libssl.so.1.0.0 /lib/i386-linux-gnu/libssl.so.1.0.1
sudo ln -sf /lib/i386-linux-gnu/libcrypto.so.1.0.0 /lib/i386-linux-gnu/libcrypto.so.1 
sudo ln -sf /lib/$(arch)-linux-gnu/libudev.so.1 /lib/$(arch)-linux-gnu/libudev.so.0
  7. (re)Start the vmware-USBArbitrator and vmware-view-USBD services 
    sudo /etc/init.d/vmware-USBArbitrator start
sudo /etc/init.d/vmware-view-USBD start

    For some reason after I did all of this the vmware-view binary was nowhere to be found. It was quite strange. I fixed this issue by removing and re-installing the view client:

    sudo ./VMware-Horizon-Client-3.5.0-2999900.x64.bundle -u vmware-horizon-client
sudo ./VMware-Horizon-Client-3.5.0-2999900.x64.bundle

    After doing this the binary was there as expected.

  8. Create a config file to instruct the view client to redirect your smartcard reader. 
    echo 'viewusb.IncludeFamily = "smart-card"' >> /etc/vmware/config
echo 'viewusb.AllowSmartcard = "true"' >> /etc/vmware/config

    There is no graphical option to pass devices through like there is in the Windows client. I spent more time than I’d like to admit on this step. It turns out the name of the file is important – it has to simply be called “config.” Place this config file in ~/.vmware (it can also be placed in /etc/vmware/config and/or /usr/lib/vmware/config)

  9. Start vmware-view and enjoy your new smartcard capabilities 
    vmware-view

Troubleshooting

If it’s not working, make sure that these services are started

  • pcscd
  • vmware-USBArbitrator
  • vmware-view-USBD

One of these services have been known to crash if you attempt to connect while your smartcard is plugged in. The dance to get around this is to unplug your card reader, re-launch the above services, launch vmware-view, connect to your view server, and then only after you’ve logged in, plug in your card reader.

 

Update 2/25/2016: Here is the script I use to make my chromebook work beautifully for remoting into work:

sudo /etc/init.d/pcscd restart
sudo /etc/init.d/vmware-USBArbitrator restart
sudo /etc/init.d/vmware-view-USBD restart
setres 1600 1024
vmware-view
setres 2560 1700
OS, Virtualization

Fix No USB devices connected in Virtualbox

Leave a comment

Recently on my Linux Mint 13.2 system I was playing with Virtualbox. I wanted to pass through a USB device to a virtualbox VM but despite installing the appropriate extension pack, I was greeted with this lovely message in the USB menu:

No USB devices connected

I discovered here that you need to be a member of the vboxusers group. One quick command solved this:

sudo usermod -a -G vboxusers <username>

Once I did that I logged out and logged back in. Voila! USB passthrough worked.

Networking, OS

Fix POP/SMTP not working in Sophos UTM

Leave a comment

It all started with an innocent enough e-mail:

Data Disk is filling up - please check. Current usage: 100%

I couldn’t find any clear information about what to do about this on Sophos’ forums. My data disk was full. What to do?

I can tell you what not to do – delete random files. I thought my solution would be to log into the UTM’s console and run a du -hsx /* to see where the space was. I found a large folder inside /var/storage – /var/storage/cores/httpd.16438. I removed it, because why not?

It turns out that did some weird things to my UTM. After removing that folder I kept getting spammed with these e-mails, once every hour:

Pop3 proxy not running - restarted

It took me a while to realize, but this also meant all e-mails relayed to the UTM were not being delivered. The entire POP/SMTP subsystem of the Sophos UTM was hosed. I could not find anything on the Sophos forums. After scratching my head I decided to have a deeper look at the logs. From the command line I issued

ls -ltr /var/log

and began reading the most recent logs.

pop3.log let me know what the problem was:

pop3proxy[9270]: Can't connect to database, retrying in 10 seconds: could not connect to server: Connection refused

I could not find any useful fixes for this error. I kept digging.

selfmon.log wasn’t much help other than to confirm that pop3 was having some serious issues. It was an endless abyss of repeated error messages:

selfmonng[3818]: W NOTIFYEVENT Name=pop3proxy_running Level=INFO Id=117 suppressed
...
selfmonng[3818]: W actionCmd(+): '/var/mdw/scripts/pop3 restart'

system.log put me on the right track:

ulogd[5107]: pg1: connect: could not connect to server: No such file or directory

 

Finally, we’re getting somewhere! After some searching I learned that pg1 is the postgresql database Sophos uses. I found a way to rebuild from this forum post.

One simple command did the trick:

/etc/init.d/postgresql92 rebuild

This rebuilt the postgresql database that I apparently corrupted when I removed files with reckless abandon. My e-mails work again!

OS

Fix raidcheck error on Xenserver 6.5

Leave a comment

After configuring a software RAID to host my VMs on my Xenserver 6.5 instance I began receiving odd e-mails once a week. The e-mails simply said:

/usr/sbin/raid-check: line 62: declare: -A: invalid option 
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

It turns out /usr/sbin-raid-check is a bash script called from the file /etc/cron.d/raid-check. It’s a weekly cron job designed to “scrub” the RAID array. I was getting these e-mails because I had configured my xen server to e-mail me anything sent to root, which includes messages encountered during cron jobs.

There appears to be a typo in the raid-check script. Line 62 of raid-check reads:

declare -A check

After reading the syntax of the declare command, I believe the issue is the fact that the A is capitalized. I commented out that line and replaced it with

declare -a check

That seemed to work. No more weird errors coming from my xenserver.

CLI

Measure SSH transfer speeds

Leave a comment

SSH is a beautiful thing. In addition to remotely administering machines you can use it to transfer files. To do this one simply pipes the cat command on both ends. For example, to copy hello.txt on the source host to hi.txt on the destination host, the command would be:

ssh remote_host cat hello.txt | cat > hi.txt

The command takes the contents of hello.txt and pipes it over to the remote host. The cat command on the remote host takes what was piped to it as  input and the > sign instructs cat to take its input and output it to hi.txt.

A great way to measure transfer speeds using ssh between two hosts is to take /dev/zero on the source host and output it to /dev/null on the destination host. This bypasses any disk speed bottlenecks and only measures network throughput. Combine this with the pv command to get a nice graphical view of how fast the transfer is going.

ssh remote_host cat /dev/zero | pv | cat > /dev/null

The default options between my machines result in about a 65 megabytes a second transfer speed.

1

It turns out that the encryption cipher used makes a big difference on transfer speeds. Use the -c command to specify which cipher to use and see how much of a difference this makes. -o compression=no can also help with transfer speeds.

The fastest cipher I’ve found is arcfour. It’s touted as less secure, but for my local network I can accept the risk (thanks to slashdot for the discussion.)

ssh -c arcfour -o Compression=no remote_host cat /dev/zero | pv | cat > /dev/null

2

Using acrfour more than doubles the speed for me! Amazing.

CLI

Allow non-root users to mount disks

Leave a comment

I came across a need today to allow a regular (non-root) user to mount disks in Ubuntu 14.04 Trusty Tahr. I usually use sudo but in this case I needed to be able to run photorec as a regular user.

The way to accomplish this is to add the regular user to the disk group. To accomplish this, run this command:

sudo usermod -a -G disk <username>

If you are logged in as that user, you will have to log out and log back in to receive the permissions. Once this is done you should be able to mount disks without using sudo or being root.

OS

Install Cinnamon on a Chromebook with Crouton

35 Comments

I really love using Crouton on my Chromebook Pixel LS 2015. I was sad to see that there is no cinnamon desktop environment target with the latest versions of crouton. Below is what I did to get Cinnamon on my chromebook. Much of what I did was taken from https://gist.github.com/sohjsolwin/5939948

  1. Create a base chroot
  2. Enter your chroot
sudo apt-get update
sudo apt-get install software-properties-common python-software-properties
sudo add-apt-repository ppa:tsvetko.tsvetkov/cinnamon
sudo apt-get update
sudo apt-get install cinnamon

Once Cinnamon was installed I needed to know how to start it manually. Thanks to the Arch Linux forums for explaining it. You have to create a .xinitrc file in your home directory within your chroot.

echo "exec cinnamon-session" > ~/.xinitrc

Trying to manually start cinnamon by typing startx didn’t work – I got a blank screen and had to hard reset to get anything to come back. Thanks to github I learned you need to use xinit instead of startx.

Lastly, we need to create a suitable startcinnamon script.

wget https://gist.github.com/sohjsolwin/5934362/raw/f68fc0942798902a0bd48f40c17dc0cd5cf585ea/startcinnamon

Modify the file to remove the startx command with xinit. Also remove everything after xinit. My file is as follows:

APPLICATION="${0##*/}"

USAGE="$APPLICATION [options]

Wraps enter-chroot to start a Mint session.
By default, it will log into the primary user on the first chroot found.

Options are directly passed to enter-chroot; run enter-chroot to list them."

exec sh -e "`dirname "$0"`/enter-chroot" "$@" xinit

Make this file executable (chmod +x startcinnamon) and move it to the /usr/local/bin directory of your chromebook (not your chroot.) Now all you need to do is enter

sudo startcinnamon

and your cinnamon desktop should come up!

 

Update 2016-01-04

These two scripts seem to work a little bit better. Place this one within your chroot under /usr/local/bin/startcinnamon:

#!/bin/sh -e
# Copyright (c) 2015 The crouton Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

# Launches GNOME; automatically falls back to gnome-panel

exec crouton-noroot gnome-session-wrapper cinnamon

Place this one in /usr/local/bin outside your chroot (on your chromebook itself.)

#!/bin/sh -e
# Copyright (c) 2015 The crouton Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

set -e

APPLICATION="${0##*/}"

USAGE="$APPLICATION [options]

Wraps enter-chroot to start a GNOME session.
By default, it will log into the primary user on the first chroot found.

Options are directly passed to enter-chroot; run enter-chroot to list them."

exec sh -e "`dirname "\`readlink -f "$0"\`"`/enter-chroot" -t cinnamon "$@" "" \
    exec startcinnamon
OS

Install OTRW2 on DD-WRT

2 Comments

Optware done the right way 2 is a set of scripts designed to enhance the functionality of your DD-WRT router. I’ve recently installed it on my parents’ router so I can more or less have a full Linux box running in their house (it makes my life easier.)

The tutorial for install it is pretty comprehensive. These are my notes on the experience.

  • USB devices needs to be ext2 formatted (fat won’t do.) This is because the script makes a bunch of symlinks to that device.
  • Mount ext2 formatted drive as /opt (Services / USB / Disk Mount Point)
  • Reboot router if you made any changes to mountpoints.

SSH into the router and run the following:

wget -O /tmp/prep_optware http://dd-ware.googlecode.com/svn/otrw2/prep_optware
sh /tmp/prep_optware

Installation takes some time. Wait one minute after installation complete message and reboot router.

Once rebooted, use the service command to see which services are available. Green means the service is enabled.

service mypage on

Enables the mypage service, but you still have to reboot. Reboot router after any changes to services to enable / disable them.

Small overview of services as taken from dd-wrt forums:

rotate_log = move the log file to /opt/var/log/
pixelserv = addblocking on you network.
unfsd = nfs server
zabbix = zabbix client (useless since its included in kong build)
pound = reverse proxy which you can use since you host multiple sites
sshhack = block ips hammering ssh with incorrect credentials.
stophack = BLock ips which are trying to hack server (only combined with pound or vlighttpd)
stophammer = block ips which are hammering ports
nzbget & transmission for downloading.
fixtables rearranges the firewall entries

A full explanation on how otrw2 enhances your router and what each package does is located here.

All in all, pretty straightforward once you get the right filesystem on your media and have it mounted on the right mountpoint. OTRW2 gives your router a whole lot more usefulness and the ability to install a wide range of packages on it.

 

 

OS, Virtualization

Slow Linux VM performance in VMware vSphere

Leave a comment

Recently I’ve been scratching my head over a particular performance issue with Linux VMs hosted on VMWare vSphere. Everything seemed to move at a glacial pace.

vmstat gave a few clues as to what was happening, although depending on what I read it still wasn’t clear:

vmstat

It became apparent that I was suffering from some kind of queuing problem. I wasn’t sure if it was CPU or disk related. I came across this post which has a lot of good performance tuning guides.This tip caught my eye:

 

7. Set your disk scheduling algorithm to ‘noop’

The Linux kernel has different ways to schedule disk I/O, using schedulers like deadline, cfq, and noop. The ‘noop’ — No Op — scheduler does nothing to optimize disk I/O. So why is this a good thing? Because ESX is also doing I/O optimization and queuing! It’s better for a guest OS to just hand over all the I/O requests to the hypervisor to sort out than to try optimizing them itself and potentially defeating the more global optimizations.

You can change the kernel’s disk scheduler at boot time by appending:

elevator=noop

to the kernel parameters in /etc/grub.conf.

Sure enough, I modified /boot/grub/grub.conf on my Centos 6 boxes and appended elevator=noop to the kernel line, then rebooted. It helped a lot! Performance no longer was pitiful. I’m not nearly as familiar with vmware as I am with Xenserver so this was a good hint.