Hi Jim,
From docs 39
DistributedDataParallel can be used in the following two ways:
(1) Single-Process Multi-GPU
(2) Multi-Process Single-GPU
Second method the highly recommended way to use DistributedDataParallel, with
multiple processes, each of which operates on a single GPU. This is
currently the fastest approach to do data parallel training using PyTorch
and applies to both single-node(multi-GPU) and multi-node data
parallel training.
Which one are you using ?