nn.DataParallel will scatter and gather the parameters, gradients, etc. between the devices as explained in this blog post 16 and is thus adding some overhead. However, it should not slow down the execution assuming you’ve scaled up the batch size.
We generally recommend to use DistributedDataParallel with a single process per device for the best performance.