Hi Deepali,
@Deepali  Thanks for your reply. Does DistributedDataParallelCPU only work for modules with all variables on CPU as its name implied? Or we can also use it for the network resides on GPU? I’m asking so because in the previous version DistributedDataParallel also works for GPU network + MPI backend.