When pytorch does the batch normalization in SGD, what batch size it use? Per GPU batch size, or total batch size over all GPUs?
For example, suppose I set the per-GPU batch size as 32, and I use 8 GPUs. When Pytorch does batch normalization, what batch size it use, 32 or 32 x 8?
My concern comes from a facebook’s paper “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”. If Pytorch’s implementation is the same with what this paper describes, I can modify the parameters of my model accordingly; if not, I need to know. Thanks!