I am trying to obtain hard-to-symbolically-compute derivatives of matrices, so I thought that I can use PyTorch’s autograd features.
I have a vector y=f(x) where y is a vector (n,1) and x is a matrix of (p,q). y is composed of a lot of operations involving x (multiplications, vertical stacking, etc …).
I’m trying to find the gradient of dy/dx (which I suppose needs to have the dimensions [n by p by q]. Not exactly sure what v 1 to write in the y.backward(v) function to get the correct results because if I use v = torch.ones_like(y) then I get the gradient x.grad as a matrix [p,q] where I would expect it to become [n,p,q] (and then again, maybe I have a different problem?).
Here’s a sample code:
import torch

n, p, q = 9, 2, 3
x = torch.randn(p, q, requires_grad=True)
A = torch.randn(q, p)
z = torch.randn(q, 1)

f = A@x
y = None
for k in range(int(n/q)):
    if(y is None):
        y = f @ z
    else:
        y = torch.vstack( ( y, f @ z ) )

    # prepare for next time step
    f = (A@x) @ f

v = torch.ones_like(y)
y.backward(v, retain_graph=True)
dydx = x.grad.clone()

print('dydx with autograd: \n', dydx)
>>> tensor([[13.4633, -3.9368,  7.8990],
            [21.1977, -4.3395, -5.6655]])