Well, first I thought the problem was in distributed training. So, I disabled that and ran on single GPU. Then I thought it was the network architecture. I tried other architectures but same issue.
Finally, I concluded that this is not because of any of the above but because of the L2 regularization I was using with the Adam optimizer. I was using weight decay with 0.0005 as the weight but I think it was too large weight.