Question: I wrote a Java program. For a small input, after warm-up (the first 50 runs), its running time turns to be around 2 milliseconds. 

In this case, could we get any benefit if I use multiple threads version? If not, what are the overhead hidden the benefits of parallel execution?

Answer: 
Most likely the overhead of starting the threads would far outweigh any speedup. On windows you'll even have trouble measuring the execution time reliably for such fast programs.

But of course, you must ask yourself if you truly need any speedup for a program that runs in 2 ms.
Id depends on if 2ms is too slow. It also depends if you want to run the tasks concurrently.
Why not use an Executor to run your tasks then you can add threads as you need them.