Category Archives: Web

Rewrite HTTP host request with Varnish

I had a stubborn WordPress plugin that somehow was remembering the URL of the site it was installed on. It became a problem when I changed the site URL. Despite changing the URL everywhere I could think of, this particular plugin was calling CSS files for the URL of the original site. I did a search and replace in the site database and searched all files for any reference to that site but couldn’t find anything. I never did find the culprit. My workaround was to use varnish to rewrite the request before it hit the browser.

Thanks to this answer by Jorge Nerin on Stack Overflow, I found my answer on how to do this.

backend www {
  .host = "www.example.com";
  .port = "http";
}

sub vcl_recv {
  if (req.http.host ~ "(?i)^(www.)?example.com$") {
    set req.backend_hint = www;
  }
}

In my case I had a default backend (no other backends configured) so my varnish config was simply adding these line in sub vcs_recv (varnish 4 syntax)

if (req.http.host ~ "(?i)^(www.)?old.host.name$") {
     set req.backend_hint = default;
}

That did the trick!

Fix wordpress PHP change was reverted error

Since WordPress 4.9 I’ve had a peculiar issue when trying to edit theme files using the web GUI. Whenever I tried to save changes I would get this error message:

Unable to communicate back with site to check for fatal errors, so the PHP change was reverted. You will need to upload your PHP file change by some other means, such as by using SFTP.

After following this long thread I saw the suggestion to install and use the Health Check plugin to get more information into why this is happening. In my case I kept getting this error message:

The loopback request to your site failed, this may prevent WP_Cron from working, along with theme and plugin editors.<br>Error encountered: (0) cURL error 28: Connection timed out after 10001 milliseconds

I researched what a loopback request is in this case. It’s the webserver reaching out to its own site’s url to talk to itself. My webserver was being denied internet access, which included its own URL, so it couldn’t complete the loopback request.

One solution, mentioned here, is to edit the hosts file on your webserver to point to 127.0.0.1 for the URL of your site. My solution was to open up the firewall to allow my server to connect to its URL. I then ran into a different problem:

The loopback request to your site failed, this may prevent WP_Cron from working, along with theme and plugin editors.<br>Error encountered: (0) cURL error 60: Peer's Certificate issuer is not recognized.

After digging for a while I found this site which explains how to edit php.ini to point to an acceptable certificate list. To fix this on my Cent7 machine I edited /etc/php.ini and added this line (you could also add it to /etc/php.d/curl.ini)

curl.cainfo="/etc/pki/tls/cert.pem"

This caused php’s curl module to use the same certificate trust store that the underlying OS uses.

Then restart php-fpm if you’re using it:

sudo systemctl restart php-fpm

Success! Loopback connections now work properly.


Update 7/16/2018: I still had a wordpress site that was giving me certificate grief despite the above fix. After MUCH frustration I finally found this post where André Gayle points out that wordpress ships with its own certificate bundle, independent of even curl’s ca bundle! It’s located in your wordpress directory/wp-includes/certificates folder.

My solution to this extremely frustrating problem was to remove their bundle and symlink to my own (Cent 7 box – adjust your path to match where your wordpress install and certificate trust store is located)

sudo mv /var/www/html/wordpress/wp-includes/certificates/ca-bundle.crt /var/www/html/wordpress/wp-includes/certificates/ca-bundle.crt.old
sudo ln -s /etc/pki/tls/cert.pem /var/www/html/wordpress/wp-includes/certificates/ca-bundle.crt

FINALLY no more loopback errors in the Health Check plugin, and thus the ability to edit theme files in the editor.

Make Java run on privileged ports in CentOS 7

I recently gnashed my teeth at trying to get java to directly bind to port 443 instead of using nginx to proxy to a java application I had to use. I was surprised at the complication of finding the solution, but I eventually did thanks to the following sites:

https://superuser.com/questions/710253/allow-non-root-process-to-bind-to-port-80-and-443/892391

https://github.com/kaitoy/pcap4j/issues/63

First, determine the full path of your current java install:

sudo update-alternatives --config java

In my CentOS 7 install, the java binary was located here:

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre/bin/java

Next, use setcap to configure java to be able to bind to port 443:

sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre/bin/java

Now, test to make sure java works:

java -version

java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

The above error means that after setting setcap, it breaks how java looks for its library to run. To fix this, we need to symlink the library it’s looking for into /usr/lib, then run ldconfig

sudo ln -s /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre/lib/amd64/jli/libjli.so /usr/lib/
sudo ldconfig

Now test Java again:

java -version

It took longer than I like to admit to get this working, but it it does indeed work this way.

Whitelist IPs with varnish

I recently needed to restrict which IP addresses can access wp-login.php for a wordpress site of mine. This site is sitting behind varnish cache for speed. Modifying htaccess doesn’t work in this case so I have to modify the varnish configuration in order to get this to work.

The varnish documentation is actually quite good at telling you what you need to do.  You have to first specify an acl and then in the vcl_recv function specify the action when these IPs are used.

I kept running into a problem where varnish wouldn’t compile. I kept receiving this error:

"Expected CSTR got 'admin_net'" (C String?)

It turns out my load balancer does not support the PROXY protocol, so client.ip is always the IP of the load balancer, not the IP of the person making the request.

The solution was finally found here where it was explained that in absence of PROXY protocol you can use the std.ip() function to convert a string containing an IP address to the value varnish expects an IP address to be, in order to check it against an ACL (see here for syntax reference.)

I then had to take it a step further to trim all the extraneous commas and quotes from X-Forwarded-For so that the std.ip() function would work:

# set realIP by trimming CloudFlare IP which will be used for various checks
set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", "");

With these three bits combined I was able to properly restrict access to wp-login.php to a specified whitelist of IP addresses:

acl admin {
 "10.0.0.0/24";
 "10.0.1.0/24";
 "10.0.2.0/24";
}

...

sub vcl_recv {
...

 # set realIP by trimming CloudFlare IP which will be used for various checks
 set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", "");

 #Deny wp-login.php access if not in admin ACL
 if ((std.ip(req.http.X-Actual-IP, "0.0.0.0") !~ admin) && req.url ~ "^/wp-login.php") {
  return(synth(403, "Access denied."));
 }
...
}

Success.

Fix WordPress “Sorry, you are not allowed to access this page.”

I recently came across an issue with my WordPress installation. It’s situated behind a load balancer where SSL is terminated. The load balancer takes HTTPS traffic, then forwards it as HTTP on port 80 to the wordpress server.

I was running issues with a redirect loop after installing wordpress. The solution was to add this bit of code to wp-config.php:

define('FORCE_SSL_ADMIN', true);
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
 $_SERVER['HTTPS']='on';

This solves the redirect loop issue but then I ran into a different problem. When I tried to sign into wp-admin I would get this message:

Sorry, you are not allowed to access this page.

After much digging I found this post which emphasizes that you must place that code BEFORE anything else in wp-config.php (except for the beginning PHP tag.) Success!

CentOS7, nginx, reverse proxy, & let’s encrypt

With the loss of trust of Startcom certs I found myself needing a new way to obtain free SSL certificates. Let’s Encrypt is perfect for this. Unfortunately SophosUTM does not support Let’s Encrypt. It became time to replace Sophos as my reverse proxy. Enter nginx.

The majority of the information I used to get this up and running came from digitalocean with help from howtoforge. My solution involves CentOS7, nginx, and the let’s encrypt software.

Install necessary packages

sudo yum install nginx letsencrypt
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=https --permanent
sudo systemctl enable nginx

Inform selinux to allow nginx to make http network connections:

sudo setsebool -P httpd_can_network_connect 1

Generate certificates

Generate your SSL certificates with the letsencrypt command. This command relies on being able to reach your site over the internet using port 80 and public DNS. Replace arguments below to reflect your setup

sudo letsencrypt certonly -a webroot --webroot-path=/usr/share/nginx/html -d example.com -d www.example.com

The above command places the certs in /etc/letsencrypt/live/<domain_name>

Sophos UTM certificates

In my case I had a few paid SSL certificates I wanted to copy over from Sophos UTM to nginx. In order to do this I had to massage them a little bit as outlined here.

Download p12 from Sophos, also download certificate authority file, then use openssl to convert the p12 to a key bundle nginx will take.

openssl pkcs12 -nokeys -in server-cert-key-bundle.p12 -out server.pem
openssl pkcs12 -nocerts -nodes -in server-cert-key-bundle.p12 -out server.key
cat server.pem Downloaded_CA_file.pem > server-ca-bundle.pem

Once you have your keyfiles you can copy them wherever you like and use them in your site-specific SSL configuration file.

Auto renewal

First make sure that the renew command works successfully:

sudo letsencrypt renew

If the output is a success (a message saying not up for renewal) then add this to a cron job to check monthly for renewal:

sudo crontab -e
30 2 1 * * /usr/bin/letsencrypt renew >> /var/log/le-renew.log
35 2 1 * * /bin/systemctl reload nginx

Configure nginx

Uncomment the https settings block in /etc/nginx/nginx.conf to allow for HTTPS connections.

Generate a strong DH group:

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Create SSL conf snippets in /etc/nginx/conf.d/ssl-<sitename>.conf. Make sure to include the proper location of your SSL certificate files as generated with the letsencrypt command.

ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_dhparam /etc/ssl/certs/dhparam.pem;

Here is a sample ssl.conf file:

server {
        listen 443;

        ssl_certificate /etc/letsencrypt/live/<HOSTNAME>/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/<HOSTNAME>/privkey.pem;
        ssl_dhparam /etc/ssl/certs/dhparam.pem;

        access_log /var/log/<HOSTNAME.log>;

        server_name <HOSTNAME>;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;

                proxy_pass http://<BACKEND_HOSTNAME>/;
        }
}

 

Redirect http to https by creating a redirect configuration file (optional)

sudo vim /etc/nginx/conf.d/redirect.conf
server {
	server_name
		<DOMAIN_1>
                ...
		<DOMAIN_N>;

        location /.well-known {
              alias /usr/share/nginx/html/.well-known;
              allow all;
	}
	location / {
               return 301 https://$host$request_uri; 
	}
}

 

Restart nginx:

sudo systemctl restart nginx

Troubleshooting

HTTPS redirects always go to the host at the top of the list

Solution found here:  use the $host variable instead of the $server_name variable in your configuration.

Websockets HTTP 400 error

Websockets require a bit more massaging in the configuration file as outlined here. Modify your site-specific configuration to add these lines:

# we're in the http context here
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {     proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}

 

Monitor your servers with phpservermonitor

I have a handful of servers and for years I’ve been wanting to get some sort of monitoring in place. Today I tried out php server monitor and found it was pretty easy to set up and use.

Download

The installation process was pretty straightforward.

  • Install PHP, mysql, and apache
  • Create database, user, password, and access rights for mysql
  • Download .tar.gz and extract to /var/www
  • Configure Apache site file to point to phpservermonitor directory
  • Navigate to the IP / URL of your apache server and run the installation script

The above process is documented fairly well on their website. I configured this to run on my Raspberry Pi 2. This was my process:

Install dependencies:

sudo apt-get install php5 php5-curl php5-mysql mysql-server

Configure mysql:

sudo mysql_secure_installation

Create database:

mysql -u root -p
create database phpservermon;
create user 'phpservermon'@'localhost' IDENTIFIED BY 'password';
 grant all privileges on phpservermon.* TO 'phpservermon'@'localhost'; 
flush privileges;

Extract phpservermon to /var/www and grant permissions

tar zxvf <phpservermon_gzip_filename> -C /var/www
sudo chown -R www-data /var/www/*

Configure php:

sudo vim /etc/php5/apache2/php.ini
#uncomment date.timezone and set your timezone
date.timezone = "America/Boise"

Configure apache:

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/phpservermon

#Modify /etc/apache2/sites-available/phpservermon server root to point to directory above, also add a ServerName if desired

sudo a2ensite phpservermon
sudo service apache2 reload

Configure cron (I have it check every minute but you can configure whatever you like)

*/1 * * * * /usr/bin/php /var/www/phpservermon/cron/status.cron.php

Navigate to the web address you’ve configured in apache and follow the wizard.

It’s pretty simple but it works! A nice php application to monitor websites and services.

 

Generate static html files from any website with httrack

I came across a need to take a cakePHP dynamically generated site and turn it into a collection of HTML files. After some trial and error I came across httrack which fit the need beautifully (thanks to this site for pointing me there.)

To install httrack run the following (for ubuntu-based distros)

sudo add-apt-repository ppa:upubuntu-com/web
sudo apt-get update
sudo apt-get install webhttrack httrack

Oncne httrack is installed simply launch it from the command line:

httrack

Follow the prompts and in no time you’ll have a folder with static HTML files for your entire website. Easy!

Purge Varnish cache by visiting URL

I came across a need to allow web developers to purge Varnish cache. The problem is the developers aren’t allowed access to the production machine and our web application firewall blocks purge requests. I needed for there to be a way for them to simply access a page hosted on the webserver and cause it to purge its own varnish cache.

I was able to accomplish this by placing a PHP file in the webserver’s directory and controlling access to it via .htaccess. Thanks to this site for the php script and this one for the .htaccess guidance.

Place this PHP file where the web devs can access it, making sure to modify the $hostname variable to suit your needs and to rename the file to have a .php extension.

<?php

#Simple script to purge varnish cache
#Adapted from the script from http://felipeferreira.net/index.php/2013/08/script-purge-varnish-cache/
#This script runs locally on the webserver and will purge all cached pages for the site specified in $hostname
#Modify the $hostname parameter below to match the hostname of your site

$hostname = $_SERVER['SERVER_NAME'];

header( 'Cache-Control: max-age=0' );

$port     = 80;
$debug    = false;
$URL      =  "/*";

purgeURL( $hostname, $port, $URL, $debug );

function purgeURL( $hostname, $port, $purgeURL, $debug )
{
    $finalURL = sprintf(
        "http://%s:%d%s", $hostname, $port, $purgeURL
    );

    print( "<br> Purging ${finalURL} <br><br>" );

    $curlOptionList = array(
        CURLOPT_RETURNTRANSFER    => true,
        CURLOPT_CUSTOMREQUEST     => 'PURGE',
        CURLOPT_HEADER            => true ,
        CURLOPT_NOBODY            => true,
        CURLOPT_URL               => $finalURL,
        CURLOPT_CONNECTTIMEOUT_MS => 2000
    );

    $fd = true;
    if( $debug == true ) {
        print "<br>---- Curl debug -----<br>";
        $fd = fopen("php://output", 'w+');
        $curlOptionList[CURLOPT_VERBOSE] = true;
        $curlOptionList[CURLOPT_STDERR]  = $fd;
    }

    $curlHandler = curl_init();
    curl_setopt_array( $curlHandler, $curlOptionList );
    $return = curl_exec($curlHandler);

    if(curl_error($curlHandler)) {
    print "<br><hr><br>CRITICAL - Error to connect to $hostname port $port - Error:  curl_error($curl) <br>";
    exit(2);
 }

    curl_close( $curlHandler );
    if( $fd !== false ) {
        fclose( $fd );
    }
    if( $debug == true ) {
        print "<br> Output: <br><br> $return <br><br><hr>";
    }
}
?>

<title>Purge cache</title>
Press submit to purge the cache
<form method="post" action="<?php echo $PHP_SELF;?>">
<input type="submit" value="submit" name="submit">
</form>

Place (or append) the following .htaccess code in the same directory you placed the php file:

#Only allow internal IPs to access cache purge page
<Files "purge.php">
    Order deny,allow
    Deny from all
    SetEnvIf X-Forwarded-For ^10\. env_allow_1
    Allow from env=env_allow_1
    Satisfy Any
</Files>

The above code only allows access to the purge.php page from IP addresses beginning with “10.” (internal IPs)

This PHP / .htaccess combo allowed the web devs to purge cache without any system access or firewall rule changes. Hooray!