What is the batch size of the input you pass to the nn.DataParallel wrapped model?
The input is split as evenly as possible over the devices you want to use. But if the split is odd, if you’re splitting over 3 GPUs, then it is possible for a subset of GPUs to have suboptimal performance. For example, a batch size of 11 is going to be much worse than a batch of 8. If you don’t have enough data for an even split, where every GPU gets a power-of-two sized batch, you can always fill it back up with garbage tensors, since you’re only doing inference.