Solved by mrshenli in post #7 
              
              
                Is the DDP process the only process using that GPU? The extra size ~500MB looks like an extra cuda context. Does this behavior still persist if you set CUDA_VISIBLE_DEVICES  env var properly (instead of using torch.cuda.set_device(rank)) before launching each process?