Solved by zshn25 in post #3 
              
              
                Well, first I thought the problem was in distributed training. So, I disabled that and ran on single GPU. Then I thought it was the network architecture. I tried other architectures but same issue. 
Finally, I concluded that this is not because of any of the above but because of the L2 regularizatio…