[ 1. calculate gradient via backward() ]
The following code generates the gradient of the output of a row-vector-valued function y with respect to (w.r.t.) its row-vector input x, using the backward() function in autograd.
(Strictly speaking, x and y are both 1xN matrix, and this is why the Jacobian matrix is a 1x2x1x2 tensor before it is “squeezed” as shown below in section 2, because this is a Jacobian of one matrix w.r.t. another matrix)
x = torch.tensor( [[2, 3]], dtype=torch.float, requires_grad=True)

def func(x):
    y = torch.zeros(1, 2) 
    y[0, 0] = x[0, 0]**2 + 3*x[0, 1] 
    y[0, 1] = x[0, 1]**2 + 2*x[0, 0]
    return y 

y = func(x)

y.backward(gradient=torch.ones_like(y))

x.grad

The output is:
tensor([[6., 9.]])
[ 2. calculate gradient manually via Jacobian-vector product ]
However, I’m unable to obtain the gradient of x mannually using the Jacobian-vector product method as shown below, i.e. conducting matrix multiplication between the transpose of the Jacobian matrix and a vector of “ones” in the same shape of y (i.e. a 1x2 row vector):
x = torch.tensor( [[2, 3]], dtype=torch.float, requires_grad=True)

def func(x):
  y = torch.zeros(1, 2)
  y[0, 0] = x[0, 0]**2 + 3<em>x[0, 1]
  y[0, 1] = x[0, 1]**2 + 2</em>x[0, 0]
  return y

y = func(x)

J = torch.squeeze(torch.autograd.functional.jacobian(func, x))

x_grad = torch.matmul(
  torch.transpose(J, 0, 1) , 
  torch.ones_like(y)
)

x_grad

This is because the 2x2 Jacobian matrix of y w.r.t. x cannot multiply a 1x2 row vector of ones in the same shape of y, as indicated in the error message, which is understandable.
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2 and 1x2)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-3ebd031d82e3> in <module>
     11 J = torch.squeeze(torch.autograd.functional.jacobian(func, x))
     12 
---> 13 x_grad = torch.matmul(
     14     torch.transpose(J, 0, 1),
     15     torch.ones_like(y)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2 and 1x2)

[ 3. My Question ]
So, if the gradient is indeed calculated by autograd using the Jacobian-vector-product method, does this mean that internally autograd will convert(transpose) the input row vector of ones to a column vector in the same length so that the matrix multiplication can be conducted correctly, and the resulting column vector will be transposed so that the final output is a row vector in the same shape as x, as shown below?
x = torch.tensor( [[2, 3]], dtype=torch.float, requires_grad=True)

def func(x):
    y = torch.zeros(1, 2) 
    y[0, 0] = x[0, 0]**2 + 3*x[0, 1] 
    y[0, 1] = x[0, 1]**2 + 2*x[0, 0]
    return y 

y = func(x)

J = torch.squeeze(torch.autograd.functional.jacobian(func, x))

x_grad = torch.transpose(torch.matmul(
    torch.transpose(J, 0, 1),
    torch.transpose(torch.ones_like(y), 0, 1) # transpose the row vector of ones to a column vector
), 0, 1)  # the result is transposed back to a row vector in the same shape as x

x_grad

… which does generate the correct results:
tensor([[6., 9.]])