I’m using DDP(one process per GPU) to training a 3D UNet. I transfered all batchnorm layer inside network to syncbatchnorm with nn.SyncBatchNorm.convert_sync_batchnorm.
When doing validation at the end of every training epoch on rank 0, it always freeze at same validation steps. I think it is because of the syncbatchnorm layer. What is the correct way to do validation when DDP model has syncbatchnorm layer? Should I do validation on all ranks?
Code
for epoch in range(epochs):
    model.train()
    train_loader.sampler.set_epoch(epoch)
    for step, (data,  target) in enumerate(train_loader):
        # ...training codes
        train(model)
    if dist.get_rank() == 0:
        # ...validation codes
        model.eval()
        validate(model)
    dist.barrier()

Version / os

torch = 1.1.0
ubuntu 18.04
distributed backend: nccl

Similar question:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Freeze during validation with distributed training and model with batch normalization layers 29
    

    
      
        opened 
        
          
        
        May 16, 2019
      
      

        
          closed 
        
          
        
        May 21, 2019
      
        

      
        
          
          SweetVlad
        
      
    
  



  ❓ Questions and Help
Hi,
I got unexpected behavior during training with torch.distributed.DistributedDataParallel model on multiple GPUs.

I train my model with DistributedSampler and...



    oncall: distributed
    triaged