Hi, @Zhang_Chi
Batch Normalization updates its running mean and variance every call of forward method.
Also, by default, BatchNorm updates its running mean by running_mean = alpha * mean + (1 - alpha) * running_mean (the details are here 307).
As to accumulating gradients, this thread “How to implement accumulated gradient？ 452” might help you.



 Zhang_Chi:

or example,now the batch size is 2, and i backward after five iterations, which is the same with a batch of 10.


As a side note, I don’t think the accumulated gradients and the gradient will be the same in the example.