Here's my program:

Copy public class Benchmark { public static void main(String[] arg) { long before = System.currentTimeMillis(); int sum = 0; for (int index = 0; index < 10*1000*1000; index += 1) { sum += index; } long after = System.currentTimeMillis(); System.out.println("Elapsed time: " + Long.toString(after - before) + " milliseconds"); } }

You are writing a microbenchmark.

Remember how HotSpot works. It starts by running your program with an interpreter. When it discovers that some method is "hot" -- that is, executed a lot, either because it is called a lot or because it contains loops that loop a lot -- it sends that method off to be compiled. After that one of two things will happen, either the next time the method is called the compiled version will be invoked (instead of the interpreted version) or the currently long running loop will be replaced, while still running, with the compiled method. The latter is known as "on stack replacement", or OSR.

In the meantime, if you insist on using/writing microbenchmarks like this, you can work around the problem by moving the body of main to a new method and calling it once from main to give the compiler a chance to compile the code, then calling it again in the timing bracket to see how fast HotSpot is.

See also the JavaOne 2002 presentation S-1816 How NOT To Write A Microbenchmark

I'm trying to time method invocation time. I don't want there to be any extra work done, so I'm using an empty method. But when I run with HotSpot I get times that are unbelievably fast. Here's my code:

Copy public class EmptyMethod { public static void method() { } public static void runTest() { long before; long after; // First, figure out the time for an empty loop before = System.currentTimeMillis(); for (int index = 0; index < 1*1000*1000; index += 1) { } after = System.currentTimeMillis(); long loopTime = after - before; System.out.println("Loop time: " + Long.toString(loopTime) + " milliseconds"); // Then time the method call in the loop before = System.currentTimeMillis(); for (int index = 0; index < 1*1000*1000; index += 1) { method(); } after = System.currentTimeMillis(); long methodTime = after - before; System.out.println("Method time: " + Long.toString(methodTime) + " milliseconds"); System.out.println("Method time - Loop time: " + Long.toString(methodTime - loopTime) + " milliseconds"); } public static void main(String[] arg) { // Warm up the virtual machine, and time it runTest(); runTest(); runTest(); } }

Empty methods don't count. And you are also seeing that generated code is sensitive to alignment.

The call to the empty method is being inlined away, so there really is no call there to time. Small methods will be inlined by the compiler at their call sites. This reduces the overhead of calls to small methods. This is particularly helpful for the accessor methods use to provide data abstraction. If the method is actually empty, the inlining completely removes the call.

Code is generated into memory and executed from there. The way the code is laid out in memory makes a big difference in the way it executes. In this example on my machine, the loop that claims to call the method is better aligned and so runs faster than the loop that's trying to figure out how long it takes to run an empty loop, so I get negative numbers for methodTime-loopTime.

Okay, so I'll put some random code in the body of the method so it's not empty and the inlining can't just remove it. Here's my new method (and the call site is changed to call method(17)):

Copy public static void method(int arg) { int value = arg + 25; }

The HotSpot compiler is smart enough not to generate code for dead variables.

In the method above, the local variable is never used, so there's no reason to compute its value. So then the method body is empty again and when the code gets compiled (and inlined, because we removed enough code to make it small enough for inlining) it turns into an empty method again.

This can be surprising to people not used to dealing with optimizing compilers, because they can be fairly clever about discovering and eliminating dead code. They can occasionally be fairly stupid about it, so don't count on the compiler to do arbitrary optimizations of your code.

Dead code elimination also extends to control flow. If the compiler can see that a particular "variable" is in fact a constant at a test, it may choose not to compile code for the branch that will never be executed. This makes it tricky to make microbenchmarks "tricky enough" to actually time what you think you are timing.

Dead code elimination is quite useful in real code. Not that people intentionally write dead code; but often the compiler discovers dead code due to inlining where constants (e.g., actual parameters to methods) replace variables, making certain control flows dead.

I'm trying to benchmark object allocation and garbage collection. So I have harness like the one above, but the body of the method is:

Copy public static void method() { Object o = new Object(); }

That's the optimal case for the HotSpot storage manager. You will get numbers that are unrealistically good.

You are allocating objects that need no initialization and dropping them on the floor instantly. (No, the compiler is not smart enough to optimize away the allocation.) Real programs do allocate a fair number of short-lived temporary objects, but they also hold on to some objects for longer than this simple test program. The HotSpot storage manager does more work for the objects that are retained for longer, so beware of trying to scale up numbers from tests like this to real systems.