Solved by nima_rafiee in post #10 
              
              
                @shaoming20798 
Yes. my problem was the normal batch norm is not working with DDP so I replaced it with syncedbatchnorm 
the way I used the synced batch norm 
def train(rank, conf):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '8888'
    dist.init_process_group(backend…