
2010/04/02 § Leave a comment
I just pushed an update to labour which supports forking multiple HTTP clients, alongside with some other small improvements. For the sake of this update I wrote a simple module called multicall which enables a process to distribute a callable among several subprocesses (POSIX only). More elegant solutions exist (‘import multiprocessing’ come to mind), but I think it’s hard to beat multicall’s simplicity, and it doesn’t use Python threads /at all/ (unlike multiprocessing), which lands some performance improvement.
I think the next area which requires work is probably reporting, and I’m not entirely sure what kind of reports would be useful. Labour isn’t a performance oriented benchmark, but a bit more like a cross between a benchmark and a test-suite for WSGI servers. I’m not sure if the output should be a graph (using behaviour set ‘foo’, servers X, Y and Z reach A, B and C hits per second) or a table (server X handles SIGSEGV well, server Y doesn’t, server X doesn’t handle a slow worker well, server Y does).
Also, and this is something that’s been troubling me for a while, I think I know what the results of most benchmarks will be. Let’s start with the fact that all servers that don’t fork out processes will not survive pretty much all the wedging/SIGSEGVing/memory leaking benchmarks. Second, I’m pretty sure most servers will handle somewhat different scenarios (IO/CPU/sleep) in exactly the same way, which is, not notice anything went wrong and have the worker pool reduced by one.
Assuming this is indeed the way the result matrix will look like, and assuming the response of the respective WSGI servers’ programmers’ will be as I expect (“fix the workers”) – then why am I doing this benchmark, anyway?
The main answer which comes to mind is that this is an ‘everyone fails’ exam which opens the door to a future where some will fail and some will succeed (or fail more gracefully). I’m pretty sure this is a future we should be heading to. I never liked the ‘fix the workers’ response, we’re hackers, not mathematicians – failing well is part of the job. That said, I’m not sure Labour will have enough weight in the WSGI/Python Web community to make this change.
If anyone wants to chip in, or even just send moral support, go ahead. I’ll post updates about Labour the next time I revisit this project.
Workers of the world (wide web), unite!
2010/03/27 § 3 Comments
A few days ago I ran into an interesting post by Ian Bicking about a benchmark that Nicholas Piël ran on WSGI servers. Go ahead and read the original posts, but the the skinny is that Nicholas’ (and many others’) focus was about performance, wherein Ian (and I…) feel more attention should be given to the ‘robustness’ of the server, so to speak.
What’s robustness, really? Well, worker-code can misbehave in a lot of ways (and for a lot of reasons), and a robust server will deal gracefully with these misbehaviours. The puritans in the crowd may well argue that the proper solution is to fix the worker (or any of its hierarchy of dependencies). However, all too often purity is a luxury that a web programmer can’t afford (and maybe shouldn’t bother to afford, but maybe we can talk about that some other time).
So I sat down and jotted something called Labour which is a library/tool to ‘benchmark’ WSGI servers not on the merit of how many jiffies they take to parse HTTP headers and what kind of scheduler algorithm they got, but rather on how they’re handling bad workers (socialist pun intended but not necessarily achieved). Labour lets a test author write elegant (ahem) code to declare a certain mixture of bad worker behaviour, and then run it against any one of several WSGI servers.
Let’s look at the simplest example (you can find a few more in Labour’s repository):
from labour import servers from labour import client from labour import behaviours from labour import report driver = client.Client() driver.add_behaviour(behaviours.PlainResponse(), weight=99) driver.add_behaviour(behaviours.Sleeping(sleep_duration=1), weight=1) with servers.WSGIRef() as server: driver.execute(iterations=1000) report.trivial_report(driver)
At its current early stage, Labour supports these (self explanatory) behaviours: PlainResponse, Sleeping, IOBound, LeakMemory, PythonWedge and SIGSEGV, more can be added with ease. It also supports the following servers (at least to some extent): WSGIRef, Twisted, CherryPy, Cogen, FAPWS3, Gevent and Paste. Significant work should be done in the areas of forking test clients in parallel (the library is near meaningless without this), supporting more WSGI servers and better exploitation of those already supported, richer and better reporting and cleaning up the potential mess after a test.
This long list of caveats said – it is actually working code and it’s currently on github (link above). So either grab it while it’s hot or follow this space to see updates in the (near) future, I’d love to hear what you think about it.