Matthew_Landen:

Yes, I’m seeing different learning steps on different ranks at once. A max difference is about 40.
I am not using a GPU at all, just cpu.


Could it be possible this is just an output buffering issue? Could you add a sys.stdout.flush() line after your print statements and see if the output makes sense?

Does the not ddp distributed calls cause problems for the context manager of join()?

Not sure I followed this, are you referring to lines like self.agent_model.join() in your code? If so, is this the ddp.join() method or is this join() method doing something else?