@MORI @smth Yeah, that may be the subtle reason. Thanks for pointing it out.
But actually, I don’t really understand. From my perspective, loss is calculated for each mini-batch samples. The gradients are accumulated if we don’t reset it, but the loss is not accumulated. Is my understanding correct? Or the loss is also summed for all the iterations inside the for loop?
So if the loss is not accumulated, why do we have to divide it by the iter_size? Thank you so much if you can explain more to help.