Hi,
Using multi-GPU will result in per GPU statistics. See discussion and why that makes sense here:

  
      github.com/pytorch/pytorch
  
  
    



Issue: Synchronize Batch Norm across Multi GPUs 15


	
	opened by ycszen
	on 2017-08-31
	
	
	


I find in some tasks , for example, semantic segmentation, detection, sync batch norm is crucial for performance.In these tasks, batch...


 	medium priority


  
  
    
    
  
  


However, there is a repo which implements synchronized batch norm:

  
      
      GitHub
  
  
    

vacancy/Synchronized-BatchNorm-PyTorch 16

Synchronized-BatchNorm-PyTorch - Synchronized Batch Normalization implementation in PyTorch.