Solved by mrshenli in post #2 
              
              
                Hey @Vibhatha_Abeykoon 
That _sync_params call is there for two purposes: 

Intra rank/process parameter sync: this is only for the legacy single-process multi-device use case, where each process operates on multiple model replicas. And this is not a recommended way to use DDP.
Inter rank/process bu…