Hi, @ptrblck. Yes, this seems to be the probable reason, thank you! I use Roberta as an encoder, so there are some BatchNorm modules in the encoder, but, as far as I can see, there are no batch-size dependent modules in the rest of the model (by the way, I’m curious now, if there any other popular batch-size dependent modules like batch norm).
But I didn’t quite understand, how momentum can help here? Because the momentum term is used for accelerating gradient using gradients from previous batches, but in the scope of several accumulation steps that we have in one batch it doesn’t influence on gradient computation in any way.