According to the torch.optim docs 347 it is recommended to move a model to GPU before constructing an optimizer i.e.
model.cuda()

optimizer = optim.SGD(model.parameters(), lr = 0.01)

# Do optimization

I only became aware of this recently and have often constructed the optimizer before moving the model to GPU, i.e.
optimizer = optim.SGD(model.parameters(), lr = 0.01)

model.cuda()

# Do optimization

However, I’ve never noticed anything going “wrong” when doing this and I have always got the results I expected in those situations.
So my question, is what can, in practice, go wrong when moving the model to GPU after constructing an optimizer? And why is this behaviour not recommended?
Thank you!