johncookds:

So this suggests its a scaling issue and tuning the learning rate should solve it rather than a more fundamental issue where the mean of the minibatch losses doesn’t equal the total batch loss.


My above example is very simple and as a result such scaling might resolve the issue. However for a complex model, it might not be clear how to scale grads/learning rate since it could contain a large number of complex functions of the form f(x + y) ! = f(x) + f(y) which result in this inconsistency.