All posts by nicholas

Fix erroneous DM Splunk Missing Forwarders alert

For some time now Splunk has been alerting me to “missing” forwarders even though all of those forwarders are working perfectly fine. It turns out to be a glitch in the Deployment Monitor app. After much digging I found this Splunk article which explains it:

https://answers.splunk.com/answers/188784/after-update-to-splunk-enterprise-62-why-does-the.html

The fix is fairly simple, thankfully. You have to edit the macros.conf of the Deployment Monitor app to add this small snippet right before the first pipe:

NOT eventType=*

The default path for this configuration file is:

/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/macros.conf

The relevant stanza in my macros.conf is below:

[forwarder_metrics]
definition = index="_internal" source="*metrics.log" group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server build version os arch guid

Crop pictures with ImageMagick’s mogrify

I recently needed a quick and dirty way to crop the bottom chunk of a large batch of scanned photos. Thanks to Linux and FOSS, this is possible with a fantastic tool known as imagemagick.

Simply install imagemagick to get the necessary tools

#Assuming you have a redhat based distro
sudo yum install ImageMagick*

Once installed use the mogrify tool (part of ImageMagick) to quickly chop the bottom part off:

mogrify -chop 0x45+0+0 -gravity South *.jpg

The above example chops the bottom 45 pixels off of every picture in the directory you’re in. Thanks to this site for the info. Handy.

Install multiple xenserver patches at once

I came across a need to install multiple patches manually (via SSH) on one of my xenservers. It’s quite tedious to do this manually so I found a way to here.)

Download all the patch .zip files to a directory your xenserver can access. Then, extract them all with this command:

find *.zip -exec unzip {} \;

Next, upload all the .xsupdates:

find *.xsupdate -exec xe patch-upload file-name={} \;

This spits out a bunch of UUIDs. Make note of these. You will also need to get your host-uuid by using the

xe host-list

command.

Lastly, a quick for loop applies the patches we want (replace the UUIDs with those of the patches uploaded earlier and the host-uuid with yours)

for file in c3520494-be00-4133-afb3-adf8ab5edb11 7fea2d85-7ce1-428c-a92f-57e37551d6f1 d9862b7f-9be6-4672-b9a8-4f52f776fd03 a424dfe5-8be8-4bd6-a49e-62620e369a43 e28bb0ae-e43f-46d9-9147-c7dc712508eb; do xe patch-apply uuid=$file host-uuid=46f8ef28-8ee1-44b5-967c-b8e48585094b; done

That did the trick for me. After applying the patches I came across this post which appears to have a much better script. Whatever works.

Use OpenVPN from within crouton chroot

Update: Google released a ChromeOS update that broke the openvpn script. Find the updated version here.

Update 3/29/17: Added a DNS suffix line to fix broken DNS


I posted a little while ago about how to get openvpn working on your chromebook. That guide required that you run openvpn outside your chroot on the chromebook instance itself.

Lately I’ve been really feeling the need to have openvpn run within a crouton chroot instead. The solution is to take the script from the post above and divide it into two parts. One part you will still have to run outside your chroot but you will only have to do it once on each reboot. The other part can safely live inside your chroot.

First, on your chromebook itself (not in a chroot) make a small script to tell the shill service not to kill tun0:

sudo echo "
#!/bin/bash

#Allows the tun0 device to function
sudo stop shill
sudo start shill BLACKLISTED_DEVICES=tun0" > /usr/local/bin/shill

chmod +x /usr/local/bin/shill

Next, create this script within your chroot. Be sure to modify the environment variables to suit your setup.

#!/bin/bash

CONF_DIR="/path/to/directory/openvpn/config/is/in"
CONF_FILE="FILENAME_OF_OVPN_FILE"
NAMESERVER="IP_OF_DNS_SERVER_YOU_WANT_TO_USE"
SEARCH="DNS_SUFFIX_YOU_WANT_TO_USE"

cd "$CONF_DIR"

# Add google DNS on top of current ones, since openvpn command does not do it
sudo sed -i "1s/^/# new DNS\nsearch $SEARCH\nnameserver $NAMESERVER\n# old DNS\n/" /etc/resolv.conf

sudo openvpn --config "$CONF_FILE" --dev tun0

# When ctrl-c is hit remove tun0 and cleanup the DNS
sudo openvpn --rmtun --dev tun0
sudo sed -i '/# new DNS/,/# old DNS/d' /etc/resolv.conf
trap 2

Voila, we now have openvpn working inside our chroots again.

Install Guacamole 0.9.9 on Ubuntu 15.10

Lately I’ve been trying to upgrade my installation of Guacamole 0.9.8 to 0.9.9. You’d think it would be simple. It is not. I ended up just blowing up my 0.9.8 VM and starting over, this time with Ubuntu 15.10. I found this excellent guide which got me most of the way there, with one small hiccup that took way more time than it should have to figure out.

I will paste the guide I got from the above site for convenience with my added notes for clarification on parts that I had trouble with.

Installation

#!/bin/bash
# WORKING ON UBUNTU 15.10 WITH GUAC 0.9.9 AND TOMCAT8

#Update Everything
apt-get update && apt-get -y dist-upgrade

#Install Stuff
#You will be prompted for a mysql root password. Remember this for the configuration step; change MYSQLROOTPASSWORD to whatever you enter here.
apt-get -y install libcairo2-dev libpng12-dev libossp-uuid-dev libfreerdp-dev libpango1.0-dev libssh2-1-dev libtelnet-dev libvncserver-dev libpulse-dev libssl-dev libvorbis-dev libwebp-dev mysql-server mysql-client mysql-common mysql-utilities tomcat8

# Install libjpeg-turbo-dev
wget -O libjpeg-turbo-official_1.4.2_amd64.deb http://downloads.sourceforge.net/project/libjpeg-turbo/1.4.2/libjpeg-turbo-official_1.4.2_amd64.deb
dpkg -i libjpeg-turbo-official_1.4.2_amd64.deb

# Add GUACAMOLE_HOME to Tomcat8 ENV
echo "" >> /etc/default/tomcat8
echo "# GUACAMOLE EVN VARIABLE" >> /etc/default/tomcat8
echo "GUACAMOLE_HOME=/etc/guacamole" >> /etc/default/tomcat8

#Download Guacamole Files
wget -O guacamole-0.9.9.war http://downloads.sourceforge.net/project/guacamole/current/binary/guacamole-0.9.9.war
wget -O guacamole-server-0.9.9.tar.gz http://sourceforge.net/projects/guacamole/files/current/source/guacamole-server-0.9.9.tar.gz
wget -O guacamole-auth-jdbc-0.9.9.tar.gz http://sourceforge.net/projects/guacamole/files/current/extensions/guacamole-auth-jdbc-0.9.9.tar.gz
wget -O mysql-connector-java-5.1.38.tar.gz http://dev.mysql.com/get/Downloads/Connector/j/mysql-connector-java-5.1.38.tar.gz

#Extract Guac
tar -xzf guacamole-server-0.9.9.tar.gz
tar -xzf guacamole-auth-jdbc-0.9.9.tar.gz
tar -xzf mysql-connector-java-5.1.38.tar.gz

# MAKE DIRECTORIES
mkdir /etc/guacamole
mkdir /etc/guacamole/lib
mkdir /etc/guacamole/extensions

# Install GUACD
cd guacamole-server-0.9.9
./configure --with-init-dir=/etc/init.d
make
make install
ldconfig
systemctl enable guacd
cd ..

# Move files to correct locations
mv guacamole-0.9.9.war /etc/guacamole/guacamole.war
ln -s /etc/guacamole/guacamole.war /var/lib/tomcat8/webapps/
cp mysql-connector-java-5.1.38/mysql-connector-java-5.1.38-bin.jar /etc/guacamole/lib/
cp guacamole-auth-jdbc-0.9.9/mysql/guacamole-auth-jdbc-mysql-0.9.9.jar /etc/guacamole/extensions/

Configuration

# Configure guacamole.properties 
echo "mysql-hostname: localhost" >> /etc/guacamole/guacamole.properties 
echo "mysql-port: 3306" >> /etc/guacamole/guacamole.properties 
echo "mysql-database: guacamole_db" >> /etc/guacamole/guacamole.properties 
echo "mysql-username: guacamole_user" >> /etc/guacamole/guacamole.properties
 
# This is where you will want to change "PASSWORD" 
echo "mysql-password: PASSWORD" >> /etc/guacamole/guacamole.properties 
rm -rf /usr/share/tomcat8/.guacamole 
ln -s /etc/guacamole /usr/share/tomcat8/.guacamole 

# Restart Tomcat Service 
service tomcat8 restart

#Configure the MySQL database
#Make sure you change MYSQLROOTPASSWORD and PASSWORD
mysql -u root -pMYSQLROOTPASSWORD
create database guacamole_db;
create user 'guacamole_user'@'localhost' identified by 'PASSWORD';
GRANT SELECT,INSERT,UPDATE,DELETE ON guacamole_db.* TO 'guacamole_user'@'localhost';
flush privileges;
quit

#Populate the database
#Make sure you change MYSQLROOTPASSWORD
cat guacamole-auth-jdbc-0.9.9/mysql/schema/*.sql | mysql -u root -pMYSQLROOTPASSWORD guacamole_db

After that it should be as simple as logging into your shiny guacamale server as guacadmin/guacadmin.

It wasn’t that simple for me. When I tried to log in all I got was a blank page. Reading the log file /var/log/tomcat8/localhost.<date>.log revealed the following:

Error querying database. Cause: java.sql.SQLException: Access denied for user 'guacamole_user '@'localhost' (using password: YES)

I kept changing and double checking the password for guacamole and couldn’t figure out why it was getting access denied. I even manually logged into mysql with that username and password and it worked, yet guacamole would not load in the browser.

After staring at the log long enough I realized that there is a space before the closing tick on the username. Aha! There were pesky trailing spaces in my guacamole.properties. Removing those spaces did the trick. Always some mundane detail!

Finally I have guacamole 0.9.9 working.

Xenserver NFS SR from FreeNAS VM hack

I have a Citrix xenserver 6.5 host which hosts a FreeNAS VM that exports an NFS share. I then have that same xenserver host use that NFS export as a SR for other VMs on that same server. It’s unusual, but it saves me from buying a separate server for VM storage.

The problem is if you reboot the hypervisor it will fail to connect to the NFS export (because the VM hosting it hasn’t booted yet.) Additionally it appears Xenserver does not play well at all with hung NFS mounts. If you try to shutdown or reboot your FreeNAS VM while Xenserver is still using its NFS export, things start to freeze. You will be unable to do anything to any of your VMs thanks to the hung NFS share. It’s a problem!

My hack around this mess is to have FreeNAS, not Xenserver, control starting and stopping these VMs.

First, create public/private key pair for ssh into xenserver

ssh-keygen

This will generate two files, a private key file and a public (.pub) file. Copy the contents of the .pub file into the xenserver’s authorized_keys file:

echo "PUT_RSA_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys

Copy the private key file (same name but without .pub extension) somewhere on your FreeNAS VM.

Next, create NFS startup and shutdown scripts. Thanks to linuxcommando for some guidance with this.  Replace the -i argument with the path to your SSH private key file generated earlier. You will also need to know the PBD UUID of the NFS store. Discover this by issuing

xe pbd-list

Copy the UUID for use in the scripts.

vi nfs-startup.sh
#!/bin/bash
#NFS FreeNAS VM startup script

SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>"

#Attach NFS drive first, then start up NFS-reliant VMs
$SSH_COMMAND xe pbd-plug uuid=<UUID_COPIED_FROM_ABOVE>

sleep 10

#Issue startup commands for each of your NFS-based VMs, repeat for each VM you have
$SSH_COMMAND xe vm-start vm="VM_NAME"
...
vi nfs-shutdown.sh
#!/bin/bash
#NFS FreeNAS VM shutdown script
#Shut down NFS-reliant VMs, detach NFS SR

#Re-establish networking to work around the fact that Network goes down before this script is executed within FreeNAS
/sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % up
SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>"

#Issue shutdown commands for each of your VMs
$SSH_COMMAND xe vm-shutdown vm="VM_NAME"

sleep 60

$SSH_COMMAND xe pbd-unplug <UUID_OF_NFS_SR>

#Take the networking interfaces back down for shutdown
/sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % down

Don’t forget to mark them executable:

chmod +x nfs-startup.sh
chmod +x nfs-shutdown.sh

Now add the scripts as a startup task in FreeNAS  and shutdown task respectively by going to System / Init/Shutdown Scripts. For startup, Select Type: Script, Type: postinit and point it to your nfs-startup.sh script. For shutdown, select Type: Script and Type: Shutdown.

Success! Now whenever your FreeNAS VM is shut down or rebooted, things will be handled properly which will prevent your hypervisor from freezing.

 

Improve FreeNAS NFS performance in Xenserver

My home lab consists of a virtualized instance of freenas, Citrix Xenserver, and various VMs. Recently I wanted to migrate some of my VMs to an NFS export from FreeNAS. To my dismay, the speed was abysmal (3 MB/second write speeds.) This tutorial will walk you through how to improve FreeNAS NFS performance in Xenserver by adding an log device (ZIL) to your ZFS pool.

After much research I realized the problem lies with ZFS behind the NFS export. Xenserver mounts the NFS share in such a way that it constantly wants to synchronize writes, which slows things down.

The solution: add a ZIL device. Since my freeNAS is virtualized, I chose the route of adding a virtual disk that is attached to an SSD. This process wasn’t straightforward.  If you have a virtual FreeNAS this is how to improve NFS performance:

  1. Add a disk in xenserver. Rule of thumb for size is half the amount of system RAM. I added 16GB ZIL disk to be safe.
  2. Add the following tunables in FreeNAS (to allow the OS to properly see xen hard drives)
    1. hint.ada.0.at, scbus100 (for the FreeNAS OS disk)
    2. hint.ada.1.at, scbus100 (for the newly added ZIL disk)
  3. Reboot FreeNAS
  4. In the FreeNAS GUI, click the ZFS Volume Manager, select your volume to expand from the dropdown, and select the device to be a LOG volume (ZIL)

That’s it! Once I added an SSD based ZIL device for my ZFS pool, NFS writes went from 3 MB/s to 60 MB/s. Awesome.

Resizing LVM storage checklist

This is a short note of what to do when you change size of the physical disk an LVM setup, such as the default configuration in CentOS 7.

  1. Modify the physical disk size
  2. Modify the partition size
    1. I used fdisk to delete the partition, then re-create with a larger size
    2. Reboot
  3. Extend the physical volume size
    1. pvresize <path to enlarged partition>
  4. Extend the logical volume size
    1. lvextend <lv path> -l100%FREE
  5. Extend filesystem size
    1. resize2fs <lv path>
    2. #If you're running CentOS 7, the default filesystem is actually XFS, not ext4. In that case:
      xfs_growfs <lv path>
  6. Profit.

Fix inconsistent mouse cursors in Linux Mint

I love Linux Mint but a frustration of mine is the fact that the mouse cursors are inconsistent. If you change the mouse theme in the themes settings it will change for most windows, but certain windows such as chrome or wine revert to the system default mouse cursor.

I’ve finally found a fix courtesy of Ubuntu Forums. The problem lies with the x-cursor-theme being independent of the theme set in cinnamon. What you have to do is run a command to update the x-cursor-theme.

First, find the name of the mouse cursor you want from a list of your installed themes:

ls /usr/share/icons

Set an environment variable of the theme you want. Specify the folder name of the theme. For example, for DMZ-Black cursor:

CURSOR=DMZ-Black

Lastly run the command to update your cursor:

gsettings set org.gnome.desktop.interface cursor-theme "$CURSOR" && sudo update-alternatives --set x-cursor-theme /usr/share/icons/$CURSOR/cursor.theme

That’s it! You now have consistent mouse cursors. OCD demons satisfied.

Compile ffmpeg on CentOS 7

Recently I had to compile ffmpeg from scratch on CentOS 7. The reason? I wanted libfdk_aac support. Here are my notes on the procedure. The how-to on https://trac.ffmpeg.org/wiki/CompilationGuide/Centos was actually quite helpful and accurate.

Install necessary dependencies and set up build folder

yum install autoconf automake cmake freetype-devel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel
mkdir ~/ffmpeg_sources

Build necessary components
I only needed x264 and libfdk_aac, so that’s all I ended up doing:

#yasm
cd ~/ffmpeg_sources
git clone --depth 1 git://github.com/yasm/yasm.git
cd yasm
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin"
make
make install
make distclean
#libx264
cd ~/ffmpeg_sources
git clone --depth 1 git://git.videolan.org/x264
cd x264
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static
make
make install
make distclean
#libfdk_aac
cd ~/ffmpeg_sources
git clone --depth 1 git://git.code.sf.net/p/opencore-amr/fdk-aac
cd fdk-aac
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

Compile ffmpeg
I actually specified a git mirror because the sources at ffmpeg site were glacially slow.

cd ~/ffmpeg_sources
git clone https://github.com/FFmpeg/FFmpeg.git
cd FFmpeg
PKG_CONFIG_PATH="$HOME/FFmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/FFmpeg_build" --extra-cflags="-I$HOME/FFmpeg_build/include" --extra-ldflags="-L$HOME/FFmpeg_build/lib" --bindir="$HOME/bin" --pkg-config-flags="--static" --enable-gpl --enable-nonfree --enable-libfdk-aac  --enable-libx264
make
make install
make distclean
hash -r

Optionally, remove existing ffmpeg

sudo yum remove ffmpeg

That was it! After a bit of compile time ffmpeg worked with the items I wanted it to.