Just wanted to leave a note regarding Batchnorm.  The running stats might differ too in case of multi gpu usage instead of single gpu.

  
      github.com/pytorch/pytorch
  
  
    



Issue: Synchronize Batch Norm across Multi GPUs 50


	
	opened by ycszen
	on 2017-08-31
	
	
	


I find in some tasks , for example, semantic segmentation, detection, sync batch norm is crucial for performance.In these tasks, batch...


 	medium priority