Finally I have solved. nn.DataParallel moves to the correct gpu only tensors, if you have list of tensors as input of your model forward() method, you need to move one by one tensors in the list on the correct gpu. The correct gpu can be retrieved by accessing the .device attribute of a tensor automatically moved by the nn.DataParallel on the correct gpu. Never force a .to(device) with the wrong device!