Looks like the program somehow desynchronized (different processes wants to sync different amount of parameters or run different numbers of iterations). Unfortunately, DDP does not have a debug mode for now. Can you share some code snippet? Does your code tries to handle any errors in backward pass on its own (say catch OOM and rertry)? Does all processes see exactly the same amount of input data?