Solved by mrshenli in post #2 
              
              
                Yep, DistributedDataParallel (DDP) can utilize multiple GPUs on the same node, but it works differently than DataParallel (DP). DDP uses multiple processes, one process per GPU, while DP is single-process multi-thread. 
See this page for the comparison between the two: https://pytorch.org/tutorials/…