Hi,
When you do net.cuda(), you should do it before you give the parameters to the optimizer.
And the conversion to cuda is actually not done in a differentiable manner, so the Parameters remains leaf (but now on a different device).