Matthew_Landen:

In the above code, the loss.backwards() is hit at different learning_steps_completed without the barrier but not with.


Could you explain what you mean by this? Are you printing self.learning_steps_completed after loss.backward on each process and you see different values across trainers where some are running ahead of others?
Note that if you are using GPUs here, gpu execution is async so essentially your training loops is just enqueuing work to the GPU and when loss.backward returns it means the work has been enqueued to the GPU but not necessarily finished. When you do dist.barrier() it forces a CUDA synchronization and blocks the host until all GPU work is done.