someone answered the question on stack overflow. Basically over trying things out apparently a different batch_size slipped into the two different versions. Correcting that gives the same result… pytorch - Hello World aka. MNIST with feed forward gets less accuracy in comparison of plain with DistributedDataParallel (DDP) model with only one node - Stack Overflow 1