Solved by richard in post #2 
              
              
                Please see the comment here: ValueError: can't optimize a non-leaf Tensor?. The main problem you are running into is that 
auto a = torch::ones({2, 2}, torch::TensorOptions().dtype(torch::kFloat).requires_grad(true)).cuda();  
is NOT a leaf node, so autograd does not accumulate gradients into a.grad.…