Category Archives: OS

OS, Web

Speed up WordPress in CentOS7 using caching

3 Comments

It has recently come to my attention that WordPress has a serious design flaw: it’s trivially easy to execute a denial of service attack against it in its default state. One simply has to hold down F5 to cause the site to slow to a crawl and in many cases crash entirely.

The reason behind this is that every single page request turns into the server parsing PHP code and executing a database call. Holding down F5 initiates hundreds of page view requests, which turn into hundreds of PHP code execution threads (each taking up CPU and memory) and hundreds of database calls. The server becomes quickly overwhelmed and runs out of resources.

The solution to this problem (on CentOS 7 at least) is a combination of php-fpm, zendopcache, APC, varnish and W3 Total Cache. It’s definitely more complicated but it eliminates this problem and massively increases site load times and general responsiveness.

Repositories

To install the required addons we will need to have the epel repository enabled:

yum -y install epel-release

zend-opcache

This caches PHP opcode to greatly speed up PHP code execution. It’s included in later versions of PHP but alas CentOS 7 is stuck on PHP 5.4, which does not include such caching. You have to install it manually. Thanks to this site for the information.

sudo yum -y install php-pecl-zendopcache

apc

This is another kind of of cache – this time for database operations.

sudo yum -y install php-pecl-apcu php-devel gcc
sudo pecl install apc
#accept defaults when prompted

php-fpm

php-fpm is a different way to serve up PHP code. Instead of apache running a module to interpret php code, it will send all php requests to a separate PHP server, optimized for speed. That php server will interpret the code and return the results to your browser.

sudo yum -y install php-fpm
sudo systemctl enable php-fpm

Modify your apache config to forward all php requests to php-fpm. Be sure to modify this to match your site URL setup:

sudo sh -c "echo '
    <LocationMatch \"^/(.*\.php(/.*)?)$\"> 
        ProxyPass fcgi://127.0.0.1:9000/var/www/html/\$1 
    </LocationMatch>' >> /etc/httpd/conf/httpd.conf"

Varnish

Varnish is a reverse proxy server optimized for caching pages. It sits between your visitors and your webserver and caches whatever it can to increase responsiveness and speed. This site pointed me in the right direction for configuring Varnish  in CentOS 7.

sudo yum -y install varnish
sudo systemctl enable varnish

Change apache to listen on port 8080 instead of port 80:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/Listen 80/&80/'

Change varnish to listen on port 80 instead of port 6081:

sudo sed -i /etc/varnish/varnish.params -e 's/VARNISH_LISTEN_PORT=6081/VARNISH_LISTEN_PORT=80/g'

Now we need to configure Varnish to properly cache wordpress sites. I found the configuration from this site to be the most helpful. I normally include code blocks to copy and paste but the configuration file is pretty big.

Instead, click here for the configuration code, then copy the whole page and paste it into your terminal.

Update 12/28/2016: I’ve updated the varnish configuration code slightly to allow the “purge all caches” button of W3 Total cache to work. Thanks to this site for pointing me in the right direction and this thread for getting me there.

After varnish has been configured, restart your new PHP / caching stack:

sudo systemctl restart httpd varnish php-fpm

Logging

Update: added this section on 11/4/2016

By default varnish does not log its traffic. This means that your apache log will only log things varnish does not cache. We have to configure varnish to log traffic so you don’t lose insight into who is visiting your site.

Update 2/14/2017:  I’ve discovered a better way to do this. The old way is still included below, but you really should use this other way.

New way:

CentOS ships with some systemd scripts for you. You can use them out of the box by simply issuing

systemctl start varnishncsa
systemctl enable varnishncsa

If you are behind a reverse proxy then you will want to tweak the varnishncsa output a bit to reflect x-forwarded-for header values (thanks to this github discussion for the guidance.) Accomplish this by appending a modified log output format string to /lib/systemd/system/varnishncsa.service:

sudo sed -i /lib/systemd/system/varnishncsa.service -e "s/ExecStart.*/& -F \'%%{X-Forwarded-For}i %%l %%u %%t \"%%r\" %%s %%b \"%%{Referer}i\" \"%%{User-agent}i\"\' /g"

Lastly, reload systemd configuration, enable, and start the varnishncsa service:

sudo systemctl daemon-reload
sudo systemctl enable varnishncsa
sudo systemctl start varnishncsa

Old way:

First, enable rc.local

sudo chmod +x /etc/rc.local
sudo systemctl enable rc-local #you can ignore errors here

Next, add this entry to the rc.local file:

sudo sh -c 'echo "varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid" >> /etc/rc.local'

If your varnish server is behind a reverse proxy (like a web application firewall) then modify the above code slightly (thanks to this site for the information on how to do so)

sudo sh -c "echo varnishncsa -a -F \'%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"\' -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid >> /etc/rc.local"

W3 Total Cache

The improvements above will greatly increase your speed and eliminate the F5 denial of service issue. The last bit to make it even sweeter is to install the W3 Total Cache wordpress plugin. Thanks to this site for the information that pointed me in the right direction.

There are a ton of options in W3 Total cache that are beyond the scope of this tutorial. For our purposes we will enable the following options in the General Settings tab of the plugin:

opcode-database-object

Opcode cache: Opcode:Zend Opcache

Database cache: Check enable, select Opcode: Alternative PHP Cache (APC / APCu)

Object cache: Check enable, select Opcode: Alternative PHP Cache (APC / APCu)

proxy

Reverse Proxy: Check “Enable reverse proxy caching via varnish”
Specify 127.0.0.1 in the varnish servers box. Click save all settings.

Full speed ahead

With all of these pieces into place your site is definitely more complicated, but it is also much faster to load. Enjoy.

Troubleshooting

If you go through all these steps only to see this very non-descriptive message:

File not found

it means you have PHP forwarding to the wrong directory. Modify the LocationMatch section you inserted at the bottom of /etc/httpd/conf/httpd.conf earlier to ensure the correct directory structure is passed for php files.

CLI, OS

Reboot a windows system remotely

Leave a comment

I recently came across the need to reboot my Windows 10 workstation remotely. I couldn’t RDP into the box  – it would stay stuck on the disclaimer splash screen. There was an OK button but it did nothing when clicked.

My solution was to initiate a reboot command from another Windows PC on the same network as my Windows 10 workstation. I figured out how to do so thanks to this article.

To remotely reboot yourself, log onto another Windows box and open up a command shell with an account that has administrative permissions on the machine you want to reboot. Then, issue this command:

shutdown -r -f -m \\<remote machine name> -t 30 -c "<message>"

The above command forces a reboot command (shutdown -r -f) on the remote machine (-m \\machine name), causes a countdown timer for the number of seconds specified (-t 30) and displays a message on the screen (-c “message”).

It solved my issue and was pretty easy to do. Handy.

CLI, OS

Updated crouton OpenVPN script

2 Comments

Updates to Chrome OS have broken the VPN script I had for crouton. I had to tweak a few things including a default route for my VPN server so that broken VPN connections could automatically reconnect.  I had to work around another new security feature:  selinux denying sed the ability to create temp files in /etc. The updated script works well, though… until Google updates ChromeOS again 🙂

See this post for instructions on how to implement the script.

Here it is:

#!/bin/bash

CONF_DIR="DIR_CONTAINING_OVPN_FILE"
CONF_FILE="NAME_OF_OVPN_FILE"
DNS_SERVER="ADDRESS_OF_VPN_DNS_SERVER"
VPN_SERVER=$(cat $CONF_DIR/$CONF_FILE | grep remote | awk '{print $2}')
DEFAULT_GW=$(sudo route | grep default | awk '{print $2}' )

cd "$CONF_DIR"

# Add google DNS on top of current ones, since openvpn command does not do it
sudo cp /etc/resolv.conf /tmp/resolv.conf
sudo sed -i "1s/^/# new DNS\nnameserver $DNS_SERVER\n# old DNS\n/" /tmp/resolv.conf
sudo cp /tmp/resolv.conf /etc/resolv.conf

sudo openvpn --config "$CONF_FILE" --dev tun0

# When ctrl-c is hit remove tun0 and cleanup the DNS
sudo openvpn --rmtun --dev tun0
sudo cp /etc/resolv.conf /tmp/resolv.conf
sudo sed -i '/# new DNS/,/# old DNS/d' /tmp/resolv.conf
sudo cp /tmp/resolv.conf /etc/resolv.conf
trap 2
CLI, OS, Web

Install WordPress on CentOS 7 with SELinux

14 Comments

I’ve been frustrated lately with the number of tutorials for installing WordPress on CentOS 7 that say to simply turn SELinux off. This is an unacceptable workaround. It’s simply lazy administration. SELinux is not designed to make your job harder (although it can do that pretty well), it’s there to make your system safer.

I have scoured the internet and pieced together bits of information from several sources – mainly here, here, and here to put together a tutorial that walks you through how to install WordPress in CentOS7 while keeping SELinux turned on.

It took a lot of reading to understand SELinux, but once you understand it, it makes a whole lot more sense. Go figure.

Apache

Install required packages:

sudo yum -y install httpd
sudo systemctl enable httpd

Modify apache config to allow mod_rewrite:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/AllowOverride None/AllowOverride All/g'

Open necessary firewall ports:

sudo firewall-cmd --add-service=http --permanent
sudo systemctl restart firewalld

Start apache:

sudo systemctl start httpd

Navigate to your new site to make sure the testing page comes up.

Update 11/4/2016: If you are behind a reverse proxy such as varnish or a web application firewall, you will want to modify your apache configuration to log x-forwarded-for IPs to make the logs more meaningful:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/%h/%{X-Forwarded-For}i/g'

MariaDB

Install:

sudo yum -y install mariadb-server mariadb
sudo systemctl enable mariadb

Run initial mysql configuration to set database root password

sudo systemctl start mariadb
sudo mysql_secure_installation

Create a wordpress database and user:

mysql -u root -p 
#enter your mysql root password here
create user wordpress;
create database wordpress;
GRANT ALL PRIVILEGES ON wordpress.* To 'wordpress'@'localhost' IDENTIFIED BY 'password';
quit;

WordPress

Install PHP and restart apache

sudo yum -y install php php-mysql php-gd php-ldap php-odbc php-pear php-xml php-xmlrpc php-mbstring php-snmp php-soap curl
sudo systemctl restart httpd

Configure base wordpress directory

Download, extract, and set permissions for your wordpress installation:

wget https://wordpress.org/latest.zip
sudo unzip latest.zip -d /var/www/html
sudo chown apache:apache -R /var/www/html/wordpress

Optional: Change Apache document root so you don’t need to tack /wordpress at the end of the url:

sudo sed -i /etc/httpd/conf/httpd.conf -e 's/DocumentRoot \"\/var\/www\/html/&\/wordpress/g'
sudo systemctl restart httpd

Configure upload directory

If you want users to upload content, then you will want to assign the http_sys_rw_content_t selinux security context for the wp-uploads directory (create it if it doesn’t exist)

sudo mkdir /var/www/html/wordpress/wp-content/uploads
sudo chown apache:apache /var/www/html/wordpress/wp-content/uploads
sudo semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html/wordpress/wp-content/uploads(/.*)?"
sudo restorecon -Rv /var/www/html/wordpress/wp-content/uploads

Run the wizard

In order for the wizard to run properly we need to temporarily give the wordpress directory httpd_sys_rw_content_t selinux context

sudo chcon -t httpd_sys_rw_content_t /var/www/html/wordpress/

Now navigate to your new website in a browser and follow the wizard, which will create a wp-config.php file inside the wordpress directory. Once your site is properly set up, restore the original security context for the wordpress directory:

sudo restorecon -v /var/www/html/wordpress/

Success! Everything is working within the proper SELinux contexts.

Troubleshooting

Permission denied when accessing /wordpress

The obvious thing to check is to make sure the directory /var/www/html/wordpress has the ownership set to apache. That didn’t fix my issue, though. Thanks to serverfault I narrowed this down to a selinux permissions issue.  Changing the selinux context to httpd_sys_content_t for the wordpress folder fixed the issue.

sudo semanage fcontext -a -t httpd_sys_content_t /var/www/html/wordpress/
sudo restorecon -v /var/www/html/wordpress/

Sorry, but I can’t write the wp-config.php file.

“You can create the wp-config.php manually and paste the following text into it.”

I had a suspicion that writing files was also a selinux issue. This was confirmed when I decided to tail /var/log/audit/audit.log and found this when the wordpress installer tried to write wp-config.php:

type=AVC msg=audit(1475596102.558:16868): avc: denied { write } for pid=5751 comm="httpd" name="wordpress" dev="dm-0" ino=68494934 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=dir
type=SYSCALL msg=audit(1475596102.558:16868): arch=c000003e syscall=21 success=no exit=-13 a0=7f9f4b931478 a1=2 a2=0 a3=1 items=0 ppid=5740 pid=5751 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)

This led me to find this great explanation from Francis Pereira on how to get wordpress and selinux to peacefully coexist. Also thanks to the excellent Redhat Manual for more information on what to do, combined with this very helpful guide from serverlab.

First, we need to temporarily grant httpd_sys_rw_content_t to the wordpress directory. This is to allow the initial wizard to create the wp-config.php file.

sudo chcon -t httpd_sys_rw_content_t /var/www/html/wordpress/

Now you can run the wizard and it will work properly.

Once your site is set up, restore the context to http_sys_content_t

sudo restorecon -v /var/www/html/wordpress/
Gaming, OS

Battle.net won’t update after copying from network folder

6 Comments

I ran into an issue recently where I tried to copy a battle.net game (Heroes of the Storm) from a backup folder on my NAS onto a new computer. Once the copy was completed I couldn’t get battle.net to update the game. It kept failing with error code:

BLZBNTAGT00000840

file update failed for an unknown reason.

After much digging I found this post which mentions it’s due to the fact that the updater apparertly can’t update files with the hidden attribute. The hidden file attribute gets applied by the NAS because the file in question has a dot in front of it in the filename. For some reason the updater can’t touch it.

The fix is to change all files in the game folder to not have the hidden attribute. The easiest way to do this is via the command line. Navigate to the folder of the game you copied over and execute the following:

attrib -H .* /S

Finally, I can copy Blizzard game backups without agonizing over why they won’t patch.

CLI, OS, Web

FreeBSD: allow non-root processes to bind port 80

2 Comments

In experimenting with FreeNAS jails I wanted to allow a web service to use port 80. Normally 80 is a high order port reserved for root-level processes for security reasons. Since this is a FreeBSD jail and not a full on system I’m not worried about this.

The command to do so is fairly simple (thanks to this page for information)

sysctl net.inet.ip.portrange.reservedhigh=0

The above command is not permanent; to make it so add it to /etc/sysctl.conf:

echo "net.inet.ip.portrange.reservedhigh=0" >> /etc/sysctl.conf
CLI, OS

FreeNAS unable to create jails fix

3 Comments

I recently got a shiny new FreeNAS Mini appliance. It’s the bee’s knees. Previously I was using a virtualized instance of FreeNAS that has served me admirably for two years now. During the migration I decided to start fresh with the jails configuration I had and deleted the entire jails dataset. This turned out to be a mistake. I suddenly found out that I couldn’t create any jails or plugins. The plugin download would hang for a long time and flash a brief message “Failed to download plugin.” Not helpful.

I tried changing the location of my jails in configuration to no avail. I even tried nuking my FreeNAS config entirely and starting from scratch. The error still happened! Somehow that configuration survived a factory restore.

I finally found this freenas forum entry that pointed me in the right direction. It suggested I use the warden command to delete the plugin jail template completely and re-install it. When I tried to I got this error:

 

[nicholas@freenas ~]$ sudo warden template delete pluginjail
ERROR: Not a ZFS volume: /mnt/storage/jails/.warden-template-pluginjail

It was still trying to install the plugin template in my non-existent dataset. I decided to try re-creating the missing dataset and then running the warden delete command again. Success!

[nicholas@freenas ~]$ sudo zfs create storage/jails/.warden-template-pluginjail
[nicholas@freenas ~]$ sudo warden template delete pluginjail

Once you delete the template jail via warden, you can re-create it in the right place after configuring the correct path in Jails / Configuration. Once you have the right place configured, issue the following:

warden template create -nick pluginjail -tar http://download.freenas.org/jails/9.3/x64/freenas-pluginjail-9.3-RELEASE.tgz

Plugins and jails work again! Success.

OS

Batch convert Global security groups to Universal

3 Comments

Recently I came across a need to batch convert global security groups into universal security groups in my work’s Active Directory domain. The reason for this is so I could then turn them into Mail Enabled security groups, which would enable mail to be delivered to members of these groups. Unfortunately all security groups at this organization are Global in scope.

Seeing as this is a one domain organization there is no harm in changing the scope to Universal. Doing this via mouse is very tedious; fortunately we can use a few basic command line tools to automate the task. Thanks to Jeff Guillet for outlining how to do this.

The three magic commands are: dsquery, dsget, and dsmod.

First I wanted to test out a single security group to make sure everything would work. I couldn’t convert it because it was a member of several global security groups. This rabbit hole went several levels deep. Piping together dsquery, dsget, and dsmod all together solved this problem instantly:

dsquery group -limit 0 -name "<Group Name>" | dsget group -memberof | dsmod group -c -q -scope u

The above command first gets the full name of the group specified by the -name command. The output is sent to the dsget command to query what groups that group is a member of. The output of that command is sent to the dsmod command, which does the work of actually changing each of those groups into a security group:

  • -c tells it to continue on error
  • -q tells it to not print successful changes.
  • -scope u instructs it to change the group’s scope to Universal.

Any errors will be printed to the console. Depending on how many levels of global groups there are you may have to run this command several times in order to convert the problematic groups to Universal scope.

Once that command finishes without error you can modify the group itself to be a universal group by simply omitting the middle dsget command:

dsquery group -limit 0 -name "<Group Name>" | dsmod group -c -q -scope u

After testing we are now ready to expand this to convert ALL Global security groups to be Universal in scope. If you would like a report of how many groups would be affected, run this command. It will output all groups from the query to the text file Groups.txt:

dsquery group -limit 0 | dsget group -samid -scope -secgrp > Groups.txt

To modify every group simply omit the “-name” parameter from the group command used above with our test group. This will iterate through every group in the directory and pass it on to dsmod which will modify the scope to be universal:

dsquery group -limit 0 | dsmod group -c -q -scope u

Some built-in groups can’t be converted due to their nature, so you will have to work around those (Domain Users being one example.) You will probably need to run the command a few times until no errors appear.

Profit.

 

CLI, OS

Mount folder from another system over SSH

Leave a comment

I recently had a need to mount a folder over SSH to allow my file manager to browse through the files on a remote system. Two great resources led me to the solution to this problem: sshfs

I first came across this little tutorial on how to install sshfs on my shiny new Linux Mint 18 box:

sudo apt-get install sshfs
sudo mkdir /mnt/droplet #<--replace "droplet" with whatever you prefer
sudo sshfs -o allow_other,defer_permissions root@xxx.xxx.xxx.xxx:/ /mnt/droplet

Pretty slick. If you want to use a keyfile instead of being prompted for a password, you can use the IdentityFile option:

sudo sshfs -o allow_other,defer_permissions,IdentityFile=~/.ssh/id_rsa root@xxx.xxx.xxx.xxx:/ /mnt/droplet

You can have this handled in /etc/fstab for automounting. Thanks to this Arch Linux guide for the info. (The command below requires systemd.)

user@host:/remote/folder /mount/point  fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/user/.ssh/id_rsa,allow_other,reconnect 0 0

I tweaked my /etc/fstab file a bit because it complained that allow_other required a configuration change. Since I’m the only user of this box it didn’t matter to me. Here is my configuration:

nicholas@remote:/ /home/desktop/remote fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/desktop/.ssh/keyfile,reconnect,allow_other 0 0

I’m mounting the root folder of my remote machine into a folder named remote on my desktop machine. I generated ssh keyfiles so that no password was required. Now the mount shows up under “Devices” in my file manager and a simple click mounts the folder gets me there. Sweet.

Edit 2/25/18: Added allow_other option per this article

 

OS

Unable to update crouton after installing VirtualBox

Leave a comment

When trying to update crouton I kept getting the following error message:

Preparing to unpack .../linux-image-3.8.11_20150314_amd64.deb ...
Ok, aborting, since modules for this image already exist.
dpkg: error processing archive /var/cache/apt/archives/linux-image-3.8.11_20150314_amd64.deb (--unpack):

After some digging I found this thread which explains how to fix it. The source of the problem was the fact that I had installed a modified kernel to allow VirtualBox to work properly. The way to update crouton is to remove the repository for that kernel

rm /etc/apt/sources.list.d/crouton-packages.list

After removing that repository the update completed successfully.