Hi everyone,
I came across a problem and can not figure out which one is right and the reason:
model = nn.DataParallel(model)
optimizer = optim.SGD(model.parameters())
##training code
optimizer = optim.SGD(model.parameters())
model = nn.DataParallel(model)
##training code
I guess the first one is logically right, since pytorch is going to optimize the parallelized model’s parameters. However, in my experiment, both worked.
I wonder which one is the right way to set up the training and why.
Thanks a lot!!