Did you double the batch size using nn.DataParallel or are you using a batch size <2x?
The batch will be chunked in dim0 and each chunk will be send to a specific GPU.
If you don’t increase the batch size, each GPU will use a smaller batch, which might decrease the performance e.g. due to noisy running estimates in batch norm layers.