What I’m asking is which batch size is used in Batch Normalization, when there are more than one GPU? Does Pytorch do batch normalization per GPU, or it do batch normalization accross all GPUs?
For example, let’s assume batch size per GPU is 32, and there are 4 GPUs, so, total batch size is 32 * 4 =128. So, when Pytorch does batch normalization, which ‘batch’ does it use? 32 or 128?