Solved by SimonW in post #2 
              
              
                This is expected. In your first code piece, the optimize-able tensor is tensor([0.], requires_grad=True), after you do .to('cuda') you get a new Tensor that is a computed result of the original tensor, i.e., an intermediate step.