It depends if the loss is inside the DataParallel or not.

In DataParallel you mean inside the forward function? But most time the forward function won’t contain loss computation right ?
I’m also confused about imbalance come from loss.backward() or from loss=criterion(ture, pred) ?
Thank you for your patience!