hi
due to limited gpu memory , i want to accumulate gradients in some iterations and back propagate to work as large batch.  for example,now the batch size is 2, and i backward after five iterations, which is the same with a batch of 10. However, what is running mean of BN layer in this process?  Will pytorch average the 10 data or only take the average of the last mini-batch (2 in this case ) as the running mean?