Here is my understanding.
DistributedDataParallel generete one process on every gpu. During every batch, every process will work on batch_size/k samples (k is number of gpu), and they won’t be gathered until all batches are trained.
Is it right?
And correspondingly， how torch.nn.DataParallel works different?