So I have been looking at this for the past day and a half.
Here in the code 28 the gradient in gradient accumulation is averaged.
Please explain to me. Gradient accumulation should accumulate (i.e. sum) the gradient, not average it, right? That makes this scaling plain wrong? Am I missing something?
Same holds for  multi-gpu parallel training, where the mean() is used directly. Both cases would be closer to their 1 GPU and full batch size equivalent if just a sum was used, right?