Question: When I am pedaling hard and shift to a larger sprocket, the shift is not smooth or sometimes doesn't happen at all. Sometimes even double up-shifts don't work. There is no problem shifting when I test for it without riding. What could be the issue?

Answer: 
After a lot of tinkering and research I found the derailleur was in the need of [indexing adjustment](http://sheldonbrown.com/derailer-adjustment.html). Just twisted the barrel adjuster anticlockwise a little and the gears are shifting promptly now. The above link advises that indexing adjustment is the most frequently required adjustment for rear derailleur.
That is perfectly normal because when you are pedaling hard the chain is under tension and resists being moved by the derailleur. Before shifting to a higher gear slow your pedaling slightly.