Thank for the reply @mrshenli
I haven’t set explicitly device cuda:0 at any point.
And even in the official DDP example code 7 shows the same unbalanced GPU memory consumption.
I solved the issue by setting torch.cuda.set_device(args.local_rank) which works the same as setting CUDA_VISIBLE_DEVICES.