Hi, thanks for that. I was thinking along those lines, but still unsure about the maths, if it is purely additive, wouldnt the performance not be much worse If you use a batch size of 1 with 64 accumulation steps, is the worse performance about about numerical stability, or are you adding up lots of variances you wouldnt have by taking the gradient in one go?
Maybe you are right and it is only the batchnorm layers, so if you had no batch norm in your model it would be the same.
Anyway I ask as its taking me ages to train one model, so dont want to try out too many learning rates:
Sam Shleifer here said here (transformers/examples/research_projects/seq2seq-distillation at master · huggingface/transformers · GitHub 10) about tips for Pegasus: "* reduce batch size, and increase gradient accumulation steps so that the product gpus * batch size * gradient_accumulation_steps = 256 . We used--learning-rate = 1e-4 * gradient accumulation steps. "
So that’s why I was wondering if you need to increase the learning rate by that equation.