Solved by mrshenli in post #2 
              
              
                DistributedDataParallel (DDP) does supports CUDA. The comment suggests extra care might be necessary when backward run on non-default stream. Actually, even if backward occurs on non-default streams it should be fine for most use cases. Below is why: 
background: I learned from @albanD that autograd…