It’s about the optimizer you are using (I guess) and probably because you may be preallocating the whole batch in gpu 0.
DataParallel has a kind of “main gpu”. Even if the whole model is copied in each gpu, thing such us optimizer parameters remain in the main one. Let me know if using plain SGD you still observing such a big difference.