paepcke:

The difference ends up to be multiple epochs, even though both processes do finish.


The way that NCCL allreduce calls (basically the backward sync point you mention)  work is by enqueuing the ops on GPU, and then the CPU host can continue, so while unlikely this desynchronization can occur.
Are you noticing any significant performance issues when this happens? If you do need to synchronize the GPUs, you can use torch.cuda.synchronize() or dist.barrier(), though this might affect the performance, especially if called very frequently.