Solved by ptrblck in post #2 
              
              
                nn.DataParallel splits the input batch in dim0 and pushes each chunk onto a GPU. 
Each GPU will call the forward method thereafter using the current data chunk. 
In your case it looks like you are using 3 GPUs. 
Since you are apparently using data samples with the batch size in dim1, this will also …