Solved by ptrblck in post #2 
              
              
                A quick test shows that the data is split sequentially: 
import torch
import torch.nn as nn

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()

    def forward(self, x):
        print(x, x.device)
        return x

model = MyModel()
model = nn.DataParallel(mod…