Matthew_Landen:

Yes, it is the ddp.join(). Because there could be three model training together, I have to enter the the model join contexts.


If you are using ddp.join(), I’m assuming this is because you have uneven data across your trainers. If so, isn’t it expected to see different number of training steps across different trainers?
Could you also provide a complete repro script that we can try on our end and also the output that you are seeing on your end. Would be much easier to troubleshoot this issue for us that way.