I have the same issue when I use multi-host training (2 multigpu instances) and set up gradient_accumulation_steps to 10.
I don’t install transformers separately, just use the one that goes with Sagemaker.
I wonder, if gradient_accumulation_steps is not compatible with multi-host training at all, or there are other parameters I need to tweak?