I want to perform a comparison of multiple implementations of basically the same algorithm, written in Java, C++ and Python, the latter executed using Pypy, Jython and CPython on a Mac OS X 10.6.4 Macbook Pro with normal (non-SSD) HDD.
It's a "decode a stream of data from a file" type of algorithm, where the relevant measurement is total execution time, and I want to prevent bias through e.g. OS an HDD caches, other programs running simultaneously, too large/small sample file etc. What do I need to pay attention to to create a fair comparison?
These are difficult to do well.
In many cases the operating system will cache files so the second time they are executed they suddenly perform much better.
The other problem is you're comparing interpreted languages against compiled. The interpreted languages require an interpreter loaded into memory somewhere or they can't run. To be scrupulously fair you really should consider if memory usage and load time for the interpreter should be part of the test. If you're looking for performance in an environment where you can assume the interpreter is always preloaded then you can ignore that. Many setups for web servers will be able to keep an interpreter preloaded. If you're doing ad hoc client applications on a desktop then the start up can be very slow while the interpreter is loaded.
Performing unbiased program/script performance comparison