pritamdamania87:

Could you explain what you mean by this? Are you printing self.learning_steps_completed after loss.backward on each process and you see different values across trainers where some are running ahead of others?


Yes, I’m seeing different learning steps on different ranks at once. A max difference is about 40.
I am not using a GPU at all, just cpu. Does the not ddp distributed calls cause problems for the context manager of join()?