I'd like to compare the performance of different languages and/or different frameworks within the same language. This is aimed at server-side languages used for web development. I know an apples to apples comparison is not possible, but I'd like it to be as unbiased as possible. Here are some ideas :

Simple "Hello World" page
Object initialization
Function/method calls


Method bodies will range from empty to large

File access (read and write)
Database access

They can either be measured by Requests per second or I can use a for loop and loop many times. Some of these benchmarks should measure the overhead the language has (ie: empty function call) rather than how fast they perform a certain task. I'll take some precautions:

They'll run on the same machine, on fresh installations with as few processes on the background as possible.
I'll try and set up the server as officially recommended; I will not attempt any optimizations.

How can I improve on this?
What I have done is to write many unit tests so you can test the layers.
For example, write a SOAP web service in PHP, Python and C#.  
Write a REST web service in the same languages (same web services, just two ways to get to them). This one should be able to return JSON and XML as a minimum.
Write unit tests in C# and Python to serve as clients, and test the REST with the various result types (XML/JSON).  This is important as later you may need to test to see which is best end-to-end, and JSON may be faster to parse than XML, for you (it should be).
So, the REST/SOAP services should go to the same controller, to simplify your life.
This controller needs tests, as you may need to later remove it's impact on your tests, but, you can also write tests to see how fast it goes to the database.
I would use one database for this, unless you want to evaluate various databases, but for a web test, just do that for phase 2. :)
So, what you end up with is lots of tests, each test needs to be able to determine how long it took for it to actually run.
You then have lots of numbers, and you can start to analyze to see what works best for you.
For example, I had learned (a couple of years ago when I did this) that JSON was faster than XML, REST was faster than SOAP.
You may find that some things are much harder to do in some languages and so drop them from contention as you go through this process.
Writing the tests is the easy part, getting meaningful answers from the numbers will be the harder part, as your biases may color your analysis, so be careful of that.
I would do this with some real application so that the work isn't wasted, just duplicated.
How can I benchmark different languages / frameworks?