Solved by jaehyung.ca in post #3 
              
              
                Thank for the reply @mrshenli 
I haven’t set explicitly device cuda:0 at any point. 
And even in the official DDP example code shows the same unbalanced GPU memory consumption. 
I solved the issue by setting torch.cuda.set_device(args.local_rank) which works the same as setting CUDA_VISIBLE_DEVICES. …