Solved by Kushaj in post #5 
              
              
                Is there any specific reason for using DataParallel instead of DistributedDataParallel? I have experience with only single GPU machines so I don’t know about the details here.