nn.DataParallel might create an overhead, thus we recommend to use DistributedDataParallel with a single process for each device.
This would be the fastest approach without creating the imbalanced memory usage.