kmityagin:

My basic guess right now is that the problem is connected with unavoidable sync between GPUs during gradient accumulation stage.


There shouldn’t be any sync between the GPUs going on if you’ve disabled gradient synchronization with no_sync context manager. You can verify this by broadcasting gradients across all ranks and observing that they are different.
When switching to distributed training there may be a need to tune certain parameters to improve the accuracy. Have you tried tuning gradient sync interval, batch size, learning rates, etc?