@shaoming20798
Yes. my problem was the normal batch norm is not working with DDP so I replaced it with syncedbatchnorm 13
the way I used the synced batch norm
def train(rank, conf):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '8888'
    dist.init_process_group(backend='nccl',init_method='env://',world_size=world_size ,rank=rank)  
    torch.cuda.set_device(rank)   
    process_group = torch.distributed.new_group()
    model = Resnet18()
    model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model, process_group)   
    model.cuda(rank) 

I hope this solves your problem