Solved by ptrblck in post #4 
              
              
                nn.DataParallel will create model replica on all specified devices. During the forward pass, the data will be split in dim0 and each chunk will be sent to the corresponding device. 
This tutorial gives you more information regarding the shapes and this blog post explains the overall work flow as wel…