I'm starting up a distributed computing project, somewhat like the various @home projects out there (though not doing simple scientific computing, but instead occasionally engaging the remote user in tasks involving presentation of audio and visual stimuli) and I need to get a sense of the relative system performance across machines that run my app so i can exclude data from machines that are very sub par (because these might not have presented the stimuli faithfully). The app is written in python, and I see that the pystone module provides a benchmark of sorts, but I also see that pystone has been disparaged as a benchmark in some cases. To my relatively novice understanding of benchmarking, pystone may not be good for general benchmarking because it collapses performance to a single score, but for my purposes where all I want is a single score to compare across machines, I think it should suffice. Are there any downsides I'm missing to using pystone for obtaining relative overall system performance?
The big problem with Pystone as a benchmark of anything (whether it be Python interpreter versions or the underlying hardware) is that it simply doesn't exercise enough different aspects of the computing environment.
Integer arithmetic, floating point arithmetic, vector operations, dedicated media hardware, memory throughput, I/O throughput, cache sizes, threading architecture, pipelining architecture... the list of hardware features that can vary across machines goes on and on, and is the biggest reason why the first question in reply to "Which is faster, A or B?" will usually be "Well, what do you plan to use them for?". The answer to the speed question is likely to be different depending on whether you're building a home media centre or a web server or a database server, etc.
Modern computer systems are complex beasts, and the layering of interpreter virtual machines with their own complex object and execution models on top don't make things any easier. A naive benchmark like Pystone will let you get a general idea of the basic computing grunt of the CPU, but won't tell you anything about the other potentially limiting factors of the machine.
What are the arguments against using pystone to estimate overall relative system performance across multiple systems?