Solved by ptrblck in post #7 
              
              
                You could check the internal implementation and probably use the functional distributed API to send the appropriate splits to the desired device. 
From a general point of view: nn.DataParallel will split the tensors in dim0, so would it be possible to make sure the data and target tensors are constr…