In an effort to see how well my varnish configuration stood up to real world tests I came across a really cool piece of software: siege. You can use siege to benchmark / load test your site by having it hit your site repeatedly in a configurable fashion.
I played around with the siege configuration until I found one I liked. It involves using wget to spider your target site, varnishncsa and tee (if you use varnish), awk to parse the access log into a usable URL list, and then passing that list to siege to test the results.
varnishncsa
For my purposes I wanted to test how well varnish performed. Varnish is a caching server that sits in front of your webserver. By default it doesn’t log anything – this is where varnishncsa comes in. Run the following command to monitor the varnish log while simultaneously outputting the results to a file:
varnishncsa | tee varnish.log
If you don’t use varnish and rather serve pages up directly through nginx, apache, or something else, then you can skip this step.
wget
The next step is to spider your site to get a list of URLs to send to siege; wget fits in nicely for this task. This is the configuration I used (replace aoeu.1234.com with the domain of your web server)
wget -r -l4 --spider -D aoeu1234.com http://www.aoeu1234.com
-r tells wget to be recursive (follow links)
-l is the number of levels you want to go down when following links
— spider tells wget to not actually download anything (no need to save the results, we just want to hit the site to generate log entries)
-D specifies a comma separated list of domains you want wget to crawl (useful if you don’t want wget to follow links outside your site)
awk
Once the spidering is finished, we want to use awk to parse the resulting server logfile to extract the URLs that were accessed and pipe the results into a file named crawl.txt.
awk '{ print $7 }' varnish.log | sort | uniq > crawl.txt
varnish.log is what I named the output from the varnishncsa command above; if you use apache/nginx, then you would use your respective access log file instead.
siege
This is where the fun comes in. You can now use siege to stress test your website to see how well it does.
siege -c 550 -i -t 3M -f crawl.txt -d 10
There are many siege options so you’ll definitely want to read up on the man page. I picked the following based on this site:
-c concurrent connections. I had to tweak siege configuration to go up that high.
-i tells siege to access the URLs in a random fashion
-t specifies how long to run siege for in seconds, minutes, or hours
-f specifies a list of URLs for siege to use
-d specifies the maximum delay between user requests (it will pick a random number between 0 and this specified number to make requests more random)
siege configuration
I had to tweak siege a little bit to allow more than 256 connections at a time. First, run
siege.config
to generate an initial siege configuration. Then modify ~/.siege/siege.conf to increase the limit of concurrent connections:
sed -i ~/.siege/siege.conf -e 's/limit = 256/limit = 1000/g'
Note: the config says the following:
DO NOT INCREASE THIS NUMBER UNLESS YOU CONFIGURED APACHE TO HANDLE MORE THAN 256 SIMULTANEOUS REQUESTS
This was fine in my case because it’s hitting varnish, not apache, so we can really ramp up the simultaneous user count.
sysctl configuration
When playing with siege I ran into a few different error messages. Siege is a complicated enough program that it requires you to tweak system settings for it to work fully. I followed some advice on this site to tweak my sysctl configuration to make it work better. Your mileage may vary.
### IMPROVE SYSTEM MEMORY MANAGEMENT ### # Increase size of file handles and inode cache fs.file-max = 2097152 # Do less swapping vm.swappiness = 10 vm.dirty_ratio = 60 vm.dirty_background_ratio = 2 ### GENERAL NETWORK SECURITY OPTIONS ### # Number of times SYNACKs for passive TCP connection. net.ipv4.tcp_synack_retries = 2 # Allowed local port range net.ipv4.ip_local_port_range = 2000 65535 # Protect Against TCP Time-Wait net.ipv4.tcp_rfc1337 = 1 # Decrease the time default value for tcp_fin_timeout connection net.ipv4.tcp_fin_timeout = 15 # Decrease the time default value for connections to keep alive net.ipv4.tcp_keepalive_time = 300 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_intvl = 15 ### TUNING NETWORK PERFORMANCE ### # Default Socket Receive Buffer net.core.rmem_default = 31457280 # Maximum Socket Receive Buffer net.core.rmem_max = 12582912 # Default Socket Send Buffer net.core.wmem_default = 31457280 # Maximum Socket Send Buffer net.core.wmem_max = 12582912 # Increase number of incoming connections net.core.somaxconn = 4096 # Increase number of incoming connections backlog net.core.netdev_max_backlog = 65536 # Increase the maximum amount of option memory buffers net.core.optmem_max = 25165824 # Increase the maximum total buffer-space allocatable # This is measured in units of pages (4096 bytes) net.ipv4.tcp_mem = 65536 131072 262144 net.ipv4.udp_mem = 65536 131072 262144 # Increase the read-buffer space allocatable net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.udp_rmem_min = 16384 # Increase the write-buffer-space allocatable net.ipv4.tcp_wmem = 8192 65536 16777216 net.ipv4.udp_wmem_min = 16384 # Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1
