Solved by JuanFMontesinos in post #12 
              
              
                Hi there @albanD, @Yuzhou_Song 
I noticed there is an small mistake with the code you provided: 
It’s necessary to unsqueeze loss inside forward pass to DataParallel were able to build loss back. Loss provided by PyTorch loss functions seems not to have dimensions, and DataParallel mount batch back …