I notice that when I accumulate gradients there is consistently worse generalization.
(by small number), e.g perplexity 12.24 vs 12.02
This happens especially when combining it with checkpointing, I did not notice checkpointing alone harms generalization for my models.
I checked this for several tasks:
I tried it in deterministic and non-deterministic cudnn modes, I tried it with LayerNorm to avoid batch norm problems.
I wonder if it could be  about LayerNorm epsilon, maybe I should change it when I scale the loss for gradient accumulation? or maybe its an overflow?
Any suggestions would be appreciated