Solved by ptrblck in post #6 
              
              
                I would recommend to setup the model before wrapping it into nn.DataParallel. 
The new creation of resnet via: 
modules = list(resnet50.children())[:-1]
resnet = nn.Sequential(*modules)

might have side effects and I’m not sure how this would interact with nn.DataParallel.