Using a real log file to load test a server

Last week I had to look into a problem with a Plone production server that had performed unsatisfactory a couple of times during the week under heavy load, with lagging response times. Now, how to trouble shoot what was the problem? I had a copy of the server that looked fine when clicking around in it, or when using the Apache benchmarking tool. The solution is of course to hit the testing server with lots of requests and see what actually happens.

However since the slowness only happened under certain load patterns, wouldn't it make sense to just use the log files from the times when the problems manifested themselves? I googled a bit and found repache, that is supposed to replay Apache log files to a server. It is written in C++ and does not compile on newer Ubuntus unless you add a couple of includes. Once that was done it promptly crashed on my access_log log file.

So, time to whip something together (note that the playlogfiletoserver github repository may have newer versions of the scipts than posted below). First a pre processor that takes a log file in the Apache combined log file format and outputs a python module containing a tuple of tuples with info on when to fire each request and what request that would be:

import datetime
import re

def get_timestamp(date):
    timestamp = datetime.datetime.strptime(date, '%d/%b/%Y:%H:%M:%S')
    return timestamp
    
def parseline(line):  
    linedict = {}
    line = line.replace('\\"', '%22')
    quoteprep = line.split('"')
    try:
        (linedict['ipnumber'], tmp, tmp, linedict['date'], tmp, tmp) = quoteprep[0].split(" ")
        (linedict['method'], linedict['path'], linedict['protocol']) = quoteprep[1].split(" ")
        (tmp, linedict['returncode'], linedict['bytestransferred'], tmp) = quoteprep[2].split(" ")
        linedict['url'] = quoteprep[3]
        linedict['date'] = linedict['date'][1:]
        
    except ValueError:
        print line
        raise
    return linedict
    
fi = open('access_log', 'r')
fo = open('generatedrequests.py', 'w')
fo.write('requests = (')

first_timestamp = 0
for line in fi:
    linedict = parseline(line)
    if not first_timestamp:
        first_timestamp = get_timestamp(linedict['date'])
        time_offset = 0
    else:
        time_offset = (get_timestamp(linedict['date']) - first_timestamp).seconds
    lineout = '''(%s,"%s"\n),''' % (str(time_offset), linedict['path'])
    fo.write(lineout)
    
fo.write(')')

Secondly, some kind of user agent is needed to do the actual requesting, and it needs to be able to run requests in parallel, lest we would not be load testing the server much if we just kindly wait for it to duly process each request in sequence. I found Pete Warden's pyparallelcurl that uses pycurl/libcurl to do its magic (pyparallelcurl is also included in the playlogfiletoserver github repository for convenience).

Thirdly and lastly the script to do the actual load testing:

import datetime
import time
from generatedrequests import requests as therequests
import pyparallelcurl
import pycurl

server = 'http://www.example.com'
speedup = 4
maxparallelrequests = 100

sleepduration = 0.5/speedup
t = datetime.datetime.now()
starttime = time.mktime(t.timetuple())


curloptions = {
    pycurl.SSL_VERIFYPEER: False,
    pycurl.SSL_VERIFYHOST: False,
    pycurl.USERAGENT: 'Play log file to server user agent',
    pycurl.FOLLOWLOCATION: True
}

#c.setopt(c.MAX_RECV_SPEED_LARGE, 10000)

parallel_curl = pyparallelcurl.ParallelCurl(maxparallelrequests, curloptions)

def go(request):
    parallel_curl.startrequest(server + request[1], on_request_done)
    return

   
def on_request_done(content, url, ch, search):
    
    if content is None:
        print "Fetch error for "+url
        return
    
    httpcode = ch.getinfo(pycurl.HTTP_CODE)   
    transfertime = ch.getinfo(pycurl.TOTAL_TIME)
    if httpcode != 200:
        print "%0.3f seconds fetch error %s for %s" % (transfertime, str(httpcode),url)
        return
    else:
        print "%0.3f seconds fetch worked %s for %s" % (transfertime, str(httpcode),url)
    print "********"


def timetogo(request):
    reqtime = request[0]
    t = datetime.datetime.now()
    now = time.mktime(t.timetuple())
    return reqtime  <= (now - starttime) * speedup


for request in therequests:
    while True:
        if timetogo(request):
            go(request)
            break
        else:
            time.sleep(sleepduration)
    
    

There are probably some faulty assumptions hiding in these few lines of code, feel free to point them out in the comments below in the issue tracker or fork the code on github.