I can’t speak to the specifics of the guidelines here. In my own usage, DataParallel is the quick and easy way to get going with multiple GPUs on a single machine.
However, if you want to push the performance, I’ve found that using the NVIDIA apex implementation of DistributedDataParallel with one GPU per process and a few of their other opts better saturates the GPUs on a single machine and usually results in approx 10-15% higher throughput.
I can’t speak to how apex DDP compares to the PyTorch native impl, I switched to apex because I’m also using AMP mixed precision from time to time.
Their example is a good sample: https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py 48