Hi, @mrshenli
Thanks a billion for your reply. I didn’t set CUDA_VISIBLE_DEVICES,  I set env as follows:
def train(args):
    torch.backends.cudnn.benchmark = True
    dist.init_process_group('nccl')
    torch.cuda.set_device(args.local_rank)
    device = torch.device('cuda', args.local_rank)
    ....
    model = model.to(device)
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank], output_device=args.local_rank)
    criterion = criterion.to(device)
    ...
    
        lr_imgs = lr_imgs.to(device)
        hr_imgs = hr_imgs.to(device)

simiar  settings in another model training,  gpu utilization is balance. It’s very strange.
I tried set CUDA_VISIBLE_DEVICES , it didn’t work.