Sorry, I think I misunderstood your question initially.
(thanks @rvarm1 for suggestion!)
Do you do validation on rank 0? If so, you can compute val_acc on rank 0 and broadcast it to all ranks at which point you can run scheduler.step independently, so all ranks are consistent. note that broadcast operation will synchronize the ranks for you.
(by @wayi)
maybe you can also consider using something like https://arxiv.org/pdf/2007.05105.pdf 4
and
GitHub - facebookresearch/fairscale: PyTorch extensions for high performance and large scale training. 4