Solved by mrshenli in post #2 
              
              
                QUESTION: Suppose each process has a different random generator state, when DistributedDataParallel is initialized does each process need to have the same parameter values? 

No. Rank 0 will broadcast model states to all other ranks when you construct DDP. Code for that is here. 

In order to evalu…