The running stats of batchnorm layers are updated during training using the training batch stats and the formula mentioned in the docs 7, which can then be used during evaluation and makes the inference perform independent from the batch size. The BatchNorm paper 2 explains this in more detail.