Solved by rvarm1 in post #2 
              
              
                we just want the training to continue normally for next iterations 

Assuming you mean continue training with DDP, this exception basically indicates this is not possible - one rank has finished all its inputs while others have not. As a result, if you continue training DDP will hang (probably at S…