Solved by mrshenli in post #2 
              
              
                Hey @Trinayan_Baruah 
Quote a recent discussion: Comparison Data Parallel Distributed data parallel 
Please also see this brief note and this full paper 

What is the behavior of using DistributedDataParallel without running the training dataset using the DistributedSampler? Will it mean that the mo…