Solved by mrshenli in post #2 
              
              
                DistributedDataParallel is multi-process parallelism, where those processes can live on different machines. So, for model = nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu]), this creates one DDP instance on one process, there could be other DDP instances from other processes in the …