@dsuess Thanks for the response!
I’ve actually followed the example here 2 for DataParallel.
Here’s a MWE (trying to avoid putting lot’s of code here)
model = VGG16
 dataset = CIFAR10

def main():
    model = create_model()
    train_loader = torch.utils.data.DataLoader(...CIFAR10...)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05, weight_decay=5e-4, momentum=0.9)
     if torch.cuda.device_count() > 1:
         model = torch.nn.parallel.DataParallel(model).to(device)
     else:
         model.to(device)
     train()
     test()




 dsuess:

It could be, e.g. that you don’t pass the DataParallel’s parameters to the params argument of the optimizer


I think that might be the case since in the MWE I’m creating the optimizer before putting the model on DataParallel?
But, it doesn’t explain why the same code works when everything else is constant in the MWE and just swap VGG16 for ResNet50?
Let me give it a try and rearrange the optimizer order after model is sent to DataParallel.
Last question, do you by any chance have any insight on this problem 2?
Thanks!