It looks like you’re not quite getting linear scaling with the number of GPUs…one idea is to try DistributedDataParallel(), which is supposed to be more performant than DataParallel()
I would also profile your code with torch.utils.bottleneck and see if any optimizations help
Finally I would try this multi-epoch dataloader if you observe slowness at the start of each epoch (which would reduce the average images/second in the multi-GPU case): https://github.com/pytorch/pytorch/issues/15849 1