I played a bit with torch::autograd::grad.
if x,y in R^3 and f = dot(x,y), then f : R^3 -> R.
f = x1y1 + x2y2 + x3y3
Now I want grad_x f (sensitivity of f with respect to x), which is a scalar by vector derivative, and I expect to get
grad_x f = [ df / dx1 df/dx2 df/dx3 ]^T = [y1 y2 y3]^T
which makes sense because dot(x,y) is linear in x, and y is kept constant with grad_x f.
Here’s the code with x = y,
    auto x = torch::linspace(0, 1, 3, torch::requires_grad()); 
    auto y = torch::linspace(0, 1, 3, torch::requires_grad()); 
    auto f = dot(x,y); 

    std::cout << "x = \n" << x << std::endl;
    std::cout << "y = \n" << y << std::endl;
    std::cout << "f = dot(x,y) = " << f << std::endl;
    
    auto grad_f = torch::autograd::grad(
        {f}, {x}, {torch::ones_like(x)}
    );

    std::cout << "grad_f = \n" << grad_f << std::endl; 

and the output for grad_f is
x = 
 0.0000
 0.5000
 1.0000
[ CPUDoubleType{3} ]
y = 
 0.0000
 0.5000
 1.0000
[ CPUDoubleType{3} ]
f = dot(x,y) = 1.25
[ CPUDoubleType{} ]
grad_f = 
 0.0000
 1.5000
 3.0000
[ CPUDoubleType{3} ]


which is not y, why? I seem to still be misunderstanding the J^T v calculation done in autograd.