I don’t know if you can check the internals of DataParallel, but in my case I simply printed the loss value inside the forward method, which was a scalar, and the loss value outside the method (thus inside the training script), which was a vector.
Actually, just re-read the issue on github, and I realized that calling .mean() might be a weak solution, in case you have batch sizes that are not equally divided across the GPUs. The proper solution seems to be summing all outputs for the loss, and then dividing with the total batch size (after scaling the individual losses by their local batch sizes as well).