If you are using nn.DataParallel, and the model workload is small, you might see the overhead of the communication (scatter, gather of model parameters, data etc.).
We usually recommend to use DistributedDataParallel with a single process for each device.