Solved by cbalioglu in post #2 
              
              
                I assume you are using DataParallel. I would suggest using DDP as it calculates the loss separately on each worker after syncing the gradients.