Solved by ptrblck in post #2 
              
              
                w doesn’t seem to be passed to the optimizer, so that even if this parameter gets valid gradients, it won’t be updated. 
Also, call cuda() on the tensor, not the nn.Parameter, as this operation will create a non-leaf variable.