Solved by ptrblck in post #4 
              
              
                No, each device will get a batch containing 2 samples. 

DDP should be faster as it reduces the communication overhead in nn.DataParallel. 
The details of the latter (including the scatter/gather calls) are described in this blog post.