Normalization layers might be batch-dependent and of course some custom layers.
I’m not sure if there are any other “core layers”. I guess RNN training would also be different, if you pass the samples sequentially or in a batched way, but this doesn’t seem to be your use case.



 mathcircler:

But I didn’t quite understand, how momentum can help here? Because the momentum term is used for accelerating gradient using gradients from previous batches, but in the scope of several accumulation steps that we have in one batch it doesn’t influence on gradient computation in any way.


Sorry, I meant the batchnorm momentum, which is used to update the running stats.
If your current batch statistics are noisy, you might change the momentum to decrease the current updates.