Gradient accumulation averages over gradients
https://github.com/huggingface/transformers/blob/121c24efa4453e4e726b5f0b2cf7095b14b7e74e/src/transformers/trainer.py#L1118

So I have been looking at this for the past day and a half. Please explain to me. Gradient accumulation should accumulate the gradient, not average it, right? That makes this scaling plain wrong? Am I missing something? 
Hi @MarktHart do you mind posting this on the forum rather? It's here: https://discuss.huggingface.co

We are trying to focus the issues on bug reports and features/model requests.

Thanks a lot.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
