Category Archives: CLI

Find top 10 requests returning 404 errors

I had a website where I was curious what the top 10 URLs that were returning 404s were along with how many hits those URLs got. This was after a huge site redesign so I was curious what old links were still trying to be accessed.

Getting a report on this can be accomplished with nothing more than the Linux command line and the log file you’re interested in. It involves combining grep, sed, awk, sort, uniq, and head commands. I enjoyed how well these tools work together so I thought I’d share. Thanks to this site for giving me the inspiration to do this.

This is the command I used to get the information I wanted:

grep '404' _log_file_ | sed 's/, /,/g' | awk {'print $7'} | sort | uniq -c | sort -n -r | head -10

Here is a rundown of each command and why it was used:

  • grep ‘404’ _log_file_ (replace with filename of your apache, tomcat, or varnish access log.) grep reads a file and returns all instances of what you want, in this case I’m looking for the number 404 (page not found HTTP error)
  • sed ‘s/, /,/g’ Sed will edit a stream of text in any way that you specify. The command I gave it (s/, /,/g) tells sed to look for instances of commas followed by spaces and replace them with just commas (eliminating the space after any comma it sees.) This was necessary in my case because sometimes the source IP address field has multiple IP addresses and it messed up the results. This may be optional if your server isn’t sitting behind any type of reverse proxy.
  • awk {‘print $7’} Awk has a lot of similar functions to sed – it allows you to do all sorts of things to text. In this case we’re telling awk to only display the 7th column of information (the URL requested in apache and varnish logs is the 7th column)
  • sort This command (absent of arguments) sorts our results alphabetically, which is necessary for the next command to work properly.
  • uniq -c This command eliminates any duplicates in the results. The -c argument adds a number indicating how many times that unique string was found.
  • sort -n -r Sorts the results in reverse alphabetical order. The -n argument sorts things numerically so that 2 follows 1 instead of 10. -r Indicates to reverse the order so the highest number is at the top of the results instead of the default which is to put the lowest number first.
  • head -10 outputs the top 10 results. This command is optional if you want to see all the results instead of the top 10. A similar command is tail – if you want to see the last results instead.

This was my output – exactly what I was looking for. Perfect.

2186 http://<sitename>/source/quicken/index.ini
2171 http://<sitename>/img/_sig.png
1947 http://<sitename>/img/email/email1.aspx
1133 http://<sitename>/source/quicken/index.ini
830 http://<sitename>/img/_sig1.png
709 https://<sitename>/img/email/email1.aspx
370 http://<sitename>/apple-touch-icon.png
204 http://<sitename>/apple-touch-icon-precomposed.png
193 http://<sitename>/About-/Plan.aspx
191 http://<sitename>/Contact-Us.aspx

Compile 32bit WINE in 64bit CentOS 7

I was dismayed to find out that you can’t run 32 bit windows programs in 64bit CentOS 7 because the wine ti comes with will only execute 64bit windows programs. In order to get around this you must compile the 32 bit version of WINE. The easiest way to do this (in my opinion) is to download the 32bit version of CentOS 7 and throw it on a VM, build wine there, then copy the RPMs and install on your 64bit host.

I was able to accomplish this thanks to the wonderful guide on the  scientific linux forum. In my case I used a virtualbox VM of 32bit Centos7 (you can download the ISO here.) You could also use Docker or even a chroot but virtualbox was the easiest for me to set up.

In the 32bit Centos:

Install necessary packages

Basics

Make sure your system is up to date before you begin your compilation journey.

sudo yum -y update

Install the EPEL repository

sudo yum -y install wget
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y localinstall epel-release-latest-7.noarch.rpm

Install development packages

sudo yum -y groupinstall "Development Tools" 
sudo yum -y install glibc-devel.i686 dbus-devel.i686 freetype-devel.i686 pulseaudio-libs-devel.i686 libX11-devel.i686 mesa-libGLU-devel.i686 libICE-devel.i686 libXext-devel.i686 libXcursor-devel.i686 libXi-devel.i686 libXxf86vm-devel.i686 libXrender-devel.i686 libXinerama-devel.i686 libXcomposite-devel.i686 libXrandr-devel.i686 mesa-libGL-devel.i686 mesa-libOSMesa-devel.i686 libxml2-devel.i686 libxslt-devel.i686 zlib-devel.i686 gnutls-devel.i686 ncurses-devel.i686 sane-backends-devel.i686 libv4l-devel.i686 libgphoto2-devel.i686 libexif-devel.i686 lcms2-devel.i686 gettext-devel.i686 isdn4k-utils-devel.i686 cups-devel.i686 fontconfig-devel.i686 gsm-devel.i686 libjpeg-turbo-devel.i686 pkgconfig.i686 libtiff-devel.i686 unixODBC.i686 openldap-devel.i686 alsa-lib-devel.i686 audiofile-devel.i686 freeglut-devel.i686 giflib-devel.i686 gstreamer-devel.i686 gstreamer-plugins-base-devel.i686 libXmu-devel.i686 libXxf86dga-devel.i686 libieee1284-devel.i686 libpng-devel.i686 librsvg2-devel.i686 libstdc++-devel.i686 libusb-devel.i686 unixODBC-devel.i686 qt-devel.i686 cmake desktop-file-utils fontforge libpcap-devel fontpackages-devel ImageMagick-devel icoutils

Prepare the build environment

Create working directory

mkdir wine && cd wine

Download gcomes’ rpmbuild script

wget 'https://www.centos.org/forums/download/file.php?id=405' -O './rpmrebuild.gz' -c
gunzip ./rpmrebuild.gz ; chmod a+x rpmrebuild

Build chrpath

wget http://vault.centos.org/7.0.1406/os/Source/SPackages/chrpath-0.13-14.el7.src.rpm
./rpmrebuild chrpath-0.13-14.el7.src.rpm

Build openal-soft

wget http://dl.fedoraproject.org/pub/epel/7/SRPMS/o/openal-soft-1.16.0-3.el7.src.rpm
./rpmrebuild -e openal-soft-1.16.0-3.el7.src.rpm

Comment out BuildRequires:portaudio-devel, then save changes (esc:ZZ)

Save and install openal-soft 32-bit rpms (do no skip this step, rpmrebuild erases and restarts each time it is run):

cp rpmbuild/RPMS/i686/openal-soft{,-devel}-1.16.0-3.el7.centos.i686.rpm .

Build nss-mdns

wget http://dl.fedoraproject.org/pub/epel/7/SRPMS/n/nss-mdns-0.10-12.el7.src.rpm
./rpmrebuild nss-mdns-0.10-12.el7.src.rpm
cp rpmbuild/RPMS/i686/nss-mdns-0.10-12.el7.centos.i686.rpm .

Build WINE

With our prerequisites installed we now need to compile 32bit WINE.

wget https://dl.fedoraproject.org/pub/epel/7/SRPMS/w/wine-1.8.4-1.el7.src.rpm
./rpmrebuild -e wine-1.8.4-1.el7.src.rpm
#ZZ to exit, no changes required 

Copy RPMs

After the lengthy build process completes be sure to copy the RPMs that were generated. These are the RPMs you will need to copy over to your 64bit Centos 7 for installation.

cp rpmbuild/RPMS/*/* .

In the 64bit CentOS

Install the resulting RPMs by copying them to your 64bit system and using yum localinstall

sudo yum -y localinstall *.rpm

Install winetricks (optional)

wget  https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks
chmod +x winetricks
sudo mv winetricks /usr/local/bin

Success

Now that you’ve made it all the way through the tutorial I will provide a link for the lazy who don’t want to compile their own wine and instead just want to download the RPMs (assuming they want to trust my build). Download the RPMs here.

Script to change WordPress URL

I wrote up a little script to run when you migrate a wordpress installation from one host to another (hostname change.)  Once this script is run you can access the site via the hostname of the server it’s running on and then change the URL to whatever you like. This comes in handy for when you want to migrate one internal host to another, then specify an external hostname once things are looking how you like them.

Change SQL_COMMAND to reflect the name of the wordpress table in the destination server. Thanks to this site for the guidance in writing the script.

#!/bin/bash

#A simple script to update the wordpress database to reflect a change in hostname
#Run this after changing the hostname / IP of a wordpress server

#Prompt for mysql root password
read -s -p "Enter mysql root password: " SQL_PASSWORD

SQL_COMMAND="mysql -u root -p$SQL_PASSWORD wordpress -e"

#Determine what the old URL was and save to variable
OLD_URL=$(mysql -u root -p$SQL_PASSWORD wordpress -e 'select option_value from wp_options where option_id = 1;' | grep http)
#Get current hostname
HOST=$(hostname)

#SQL statements to update database to new hostname
$SQL_COMMAND "UPDATE wp_options SET option_value = replace(option_value, '$OLD_URL', 'http://$HOST') WHERE option_name = 'home' OR option_name = 'siteurl';"
$SQL_COMMAND "UPDATE wp_posts SET guid = replace(guid, '$OLD_URL','http://$HOST');"
$SQL_COMMAND "UPDATE wp_posts SET post_content = replace(post_content, '$OLD_URL', 'http://$HOST');"
$SQL_COMMAND "UPDATE wp_postmeta SET meta_value = replace(meta_value,'$OLD_URL','http://$HOST');"

Reboot a windows system remotely

I recently came across the need to reboot my Windows 10 workstation remotely. I couldn’t RDP into the box  – it would stay stuck on the disclaimer splash screen. There was an OK button but it did nothing when clicked.

My solution was to initiate a reboot command from another Windows PC on the same network as my Windows 10 workstation. I figured out how to do so thanks to this article.

To remotely reboot yourself, log onto another Windows box and open up a command shell with an account that has administrative permissions on the machine you want to reboot. Then, issue this command:

shutdown -r -f -m \\<remote machine name> -t 30 -c "<message>"

The above command forces a reboot command (shutdown -r -f) on the remote machine (-m \\machine name), causes a countdown timer for the number of seconds specified (-t 30) and displays a message on the screen (-c “message”).

It solved my issue and was pretty easy to do. Handy.

Rename files for proper sorting in Linux

I often come across files than are named 1..9 and then go to 10…99. The problem is many Linux programs begin with 1, then go to 10, etc. The sorting is wrong. Fortunately the rename command comes to our rescue:

rename 's/\d+/sprintf("%05d", $&)/e' *.jpg

Running the above command looks for numbers in the name of JPG files (in the current directory) and renames the file to ensure there are 5 digits in the filename. Now, instead of 1.jpg, your file will be named 00001.jpg. Handy.

Thanks to this forum for the information.

Updated crouton OpenVPN script

Updates to Chrome OS have broken the VPN script I had for crouton. I had to tweak a few things including a default route for my VPN server so that broken VPN connections could automatically reconnect.  I had to work around another new security feature:  selinux denying sed the ability to create temp files in /etc. The updated script works well, though… until Google updates ChromeOS again 🙂

See this post for instructions on how to implement the script.

Here it is:

#!/bin/bash

CONF_DIR="DIR_CONTAINING_OVPN_FILE"
CONF_FILE="NAME_OF_OVPN_FILE"
DNS_SERVER="ADDRESS_OF_VPN_DNS_SERVER"
VPN_SERVER=$(cat $CONF_DIR/$CONF_FILE | grep remote | awk '{print $2}')
DEFAULT_GW=$(sudo route | grep default | awk '{print $2}' )

cd "$CONF_DIR"

# Add google DNS on top of current ones, since openvpn command does not do it
sudo cp /etc/resolv.conf /tmp/resolv.conf
sudo sed -i "1s/^/# new DNS\nnameserver $DNS_SERVER\n# old DNS\n/" /tmp/resolv.conf
sudo cp /tmp/resolv.conf /etc/resolv.conf

sudo openvpn --config "$CONF_FILE" --dev tun0

# When ctrl-c is hit remove tun0 and cleanup the DNS
sudo openvpn --rmtun --dev tun0
sudo cp /etc/resolv.conf /tmp/resolv.conf
sudo sed -i '/# new DNS/,/# old DNS/d' /tmp/resolv.conf
sudo cp /tmp/resolv.conf /etc/resolv.conf
trap 2

Install WordPress on CentOS 7 with SELinux

I’ve been frustrated lately with the number of tutorials for installing WordPress on CentOS 7 that say to simply turn SELinux off. This is an unacceptable workaround. It’s simply lazy administration. SELinux is not designed to make your job harder (although it can do that pretty well), it’s there to make your system safer.

I have scoured the internet and pieced together bits of information from several sources – mainly here, here, and here to put together a tutorial that walks you through how to install WordPress in CentOS7 while keeping SELinux turned on.

It took a lot of reading to understand SELinux, but once you understand it, it makes a whole lot more sense. Go figure.

Apache

Install required packages:

sudo yum -y install httpd
sudo systemctl enable httpd

Modify apache config to allow mod_rewrite:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/AllowOverride None/AllowOverride All/g'

Open necessary firewall ports:

sudo firewall-cmd --add-service=http --permanent
sudo systemctl restart firewalld

Start apache:

sudo systemctl start httpd

Navigate to your new site to make sure the testing page comes up.

Update 11/4/2016: If you are behind a reverse proxy such as varnish or a web application firewall, you will want to modify your apache configuration to log x-forwarded-for IPs to make the logs more meaningful:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/%h/%{X-Forwarded-For}i/g'

MariaDB

Install:

sudo yum -y install mariadb-server mariadb
sudo systemctl enable mariadb

Run initial mysql configuration to set database root password

sudo systemctl start mariadb
sudo mysql_secure_installation

Create a wordpress database and user:

mysql -u root -p 
#enter your mysql root password here
create user wordpress;
create database wordpress;
GRANT ALL PRIVILEGES ON wordpress.* To 'wordpress'@'localhost' IDENTIFIED BY 'password';
quit;

WordPress

Install PHP and restart apache

sudo yum -y install php php-mysql php-gd php-ldap php-odbc php-pear php-xml php-xmlrpc php-mbstring php-snmp php-soap curl
sudo systemctl restart httpd

Configure base wordpress directory

Download, extract, and set permissions for your wordpress installation:

wget https://wordpress.org/latest.zip
sudo unzip latest.zip -d /var/www/html
sudo chown apache:apache -R /var/www/html/wordpress

Optional: Change Apache document root so you don’t need to tack /wordpress at the end of the url:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/DocumentRoot \"\/var\/www\/html/&\/wordpress/g'
sudo systemctl restart httpd

Configure upload directory

If you want users to upload content, then you will want to assign the http_sys_rw_content_t selinux security context for the wp-uploads directory (create it if it doesn’t exist)

sudo mkdir /var/www/html/wordpress/wp-content/uploads
sudo chown apache:apache /var/www/html/wordpress/wp-content/uploads
sudo semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html/wordpress/wp-content/uploads(/.*)?"
sudo restorecon -Rv /var/www/html/wordpress/wp-content/uploads

Run the wizard

In order for the wizard to run properly we need to temporarily give the wordpress directory httpd_sys_rw_content_t selinux context

sudo chcon -t httpd_sys_rw_content_t /var/www/html/wordpress/

Now navigate to your new website in a browser and follow the wizard, which will create a wp-config.php file inside the wordpress directory. Once your site is properly set up, restore the original security context for the wordpress directory:

sudo restorecon -v /var/www/html/wordpress/

Success! Everything is working within the proper SELinux contexts.

Troubleshooting

Permission denied when accessing /wordpress

The obvious thing to check is to make sure the directory /var/www/html/wordpress has the ownership set to apache. That didn’t fix my issue, though. Thanks to serverfault I narrowed this down to a selinux permissions issue.  Changing the selinux context to httpd_sys_content_t for the wordpress folder fixed the issue.

sudo semanage fcontext -a -t httpd_sys_content_t /var/www/html/wordpress/
sudo restorecon -v /var/www/html/wordpress/

Sorry, but I can’t write the wp-config.php file.

“You can create the wp-config.php manually and paste the following text into it.”

I had a suspicion that writing files was also a selinux issue. This was confirmed when I decided to tail /var/log/audit/audit.log and found this when the wordpress installer tried to write wp-config.php:

type=AVC msg=audit(1475596102.558:16868): avc: denied { write } for pid=5751 comm="httpd" name="wordpress" dev="dm-0" ino=68494934 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=dir
type=SYSCALL msg=audit(1475596102.558:16868): arch=c000003e syscall=21 success=no exit=-13 a0=7f9f4b931478 a1=2 a2=0 a3=1 items=0 ppid=5740 pid=5751 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)

This led me to find this great explanation from Francis Pereira on how to get wordpress and selinux to peacefully coexist. Also thanks to the excellent Redhat Manual for more information on what to do, combined with this very helpful guide from serverlab.

First, we need to temporarily grant httpd_sys_rw_content_t to the wordpress directory. This is to allow the initial wizard to create the wp-config.php file.

sudo chcon -t httpd_sys_rw_content_t /var/www/html/wordpress/

Now you can run the wizard and it will work properly.

Once your site is set up, restore the context to http_sys_content_t

sudo restorecon -v /var/www/html/wordpress/

Mountpoint check script

I wrote a simple script to check to see if a specific mountpoint on a Linux system is still live.  It does this by trying to read a specific file on the share, and if it cannot, write the event to a log, unmount, and then re-mount the folder. The need arose for instances where a file server has been rebooted and the linux system loses the connection to the share. This way it will automatically re-mount.

Modify the variables section as needed and then have a cron job run the script as root at whatever interval you want. Enjoy.

#!/bin/bash
#Script to monitor mount directories to ensure they are properly mounted
#Place a file with the word "mounted" in it inside all mounted directories
#The script will try to read the file and attempt to unmount and remount the folder if it fails to read the file
#Updated 8/30/2016 by Nicholas Jeppson

#---------Variable section------------#

#Place mount folder locations here, separated by space 
#Paths containing spaces need to have quotes around them
LOCATIONS=(/home/njeppson /home/njeppson/Desktop)

#Name of file to try to read
TEST_FILENAME="mountcheck"

#---------End Variable Section--------#
#-----Do not edit below this line-----#

#Read file, if contents don't contain "mounted" then attempt to unmount and re-mount the folder, output attempt to /var/log/mountcheck

for FOLDER in "${LOCATIONS[@]}"; do 
 if [[ $(cat $FOLDER/$TEST_FILENAME) != "mounted" ]]; then
 echo "$(date "+%b %d %T") $(hostname) $FOLDER Not mounted, remounting." >> /var/log/mountcheck 
 umount $FOLDER
 mount $FOLDER
 fi
done

FreeBSD: allow non-root processes to bind port 80

In experimenting with FreeNAS jails I wanted to allow a web service to use port 80. Normally 80 is a high order port reserved for root-level processes for security reasons. Since this is a FreeBSD jail and not a full on system I’m not worried about this.

The command to do so is fairly simple (thanks to this page for information)

sysctl net.inet.ip.portrange.reservedhigh=0

The above command is not permanent; to make it so add it to /etc/sysctl.conf:

echo "net.inet.ip.portrange.reservedhigh=0" >> /etc/sysctl.conf

FreeNAS unable to create jails fix

I recently got a shiny new FreeNAS Mini appliance. It’s the bee’s knees. Previously I was using a virtualized instance of FreeNAS that has served me admirably for two years now. During the migration I decided to start fresh with the jails configuration I had and deleted the entire jails dataset. This turned out to be a mistake. I suddenly found out that I couldn’t create any jails or plugins. The plugin download would hang for a long time and flash a brief message “Failed to download plugin.” Not helpful.

I tried changing the location of my jails in configuration to no avail. I even tried nuking my FreeNAS config entirely and starting from scratch. The error still happened! Somehow that configuration survived a factory restore.

I finally found this freenas forum entry that pointed me in the right direction. It suggested I use the warden command to delete the plugin jail template completely and re-install it. When I tried to I got this error:

 

[nicholas@freenas ~]$ sudo warden template delete pluginjail
ERROR: Not a ZFS volume: /mnt/storage/jails/.warden-template-pluginjail

It was still trying to install the plugin template in my non-existent dataset. I decided to try re-creating the missing dataset and then running the warden delete command again. Success!

[nicholas@freenas ~]$ sudo zfs create storage/jails/.warden-template-pluginjail
[nicholas@freenas ~]$ sudo warden template delete pluginjail

Once you delete the template jail via warden, you can re-create it in the right place after configuring the correct path in Jails / Configuration. Once you have the right place configured, issue the following:

warden template create -nick pluginjail -tar http://download.freenas.org/jails/9.3/x64/freenas-pluginjail-9.3-RELEASE.tgz

Plugins and jails work again! Success.