Solved by wayi in post #4 
              
              
                Realize that the root cause is that, your print statement only requires the input from host, not from device. If you print results from CUDA tensors, you should see synced outputs. 
This is because although DDP syncs across devices at each step, the allreduce communication from the host perspective …