@apaszke, Could you please look into this. I think multiple people have this issue. I am getting 80% Training accuracy after 15th epochs using single GPU, but when I use multiple GPUs [4 in my case], after 15th epochs, the accuracy is 28%. I ran my experiments multiple times and the multi-GPU accuracy seems quite bad. Thank you.