xdwang0726:

model = torch.nn.parallel.DistributedDataParallel(model)


When doing the above without specifying a device_id, it will try to replicate the model to all visible devices in each process (unless the model is on CPU). Is this intentional? The recommended use of DDP is to let each process exclusively operate on one GPU.
Beside, before v1.7 DDP will create communication buckets. The total size of those buckets will be the same as model size. So the GPU memory size needs to be at least 3X of the model size.