Realize that the root cause is that, your print statement only requires the input from host, not from device. If you print results from CUDA tensors, you should see synced outputs.
This is because although DDP syncs across devices at each step, the allreduce communication from the host perspective is just a non-blocking enqueue operation. There isn’t anything wrong with your DDP code. Just your print statement gives you an illusion caused by the enqueue operation from the host.