Hi Rishi!



 snau:

I tried your version and getting the same error.


The problem is that you have to rebuild the “computation graph”
before you call .backward() for a second time.  Both indexing
into a and setting that “value” to param count as part of the
computation graph.
Consider:
>>> import torch
>>> print (torch.__version__)
1.9.0
>>>
>>> param = torch.ones (1, requires_grad = True)
>>>
>>> a = torch.zeros (2, 2)
>>> a[0,0] = param
>>> a[1,1] = 2
>>>
>>> a.sum().backward()
>>>
>>> a[0,0] = param
>>> a.sum().backward()   # fails
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path_to_pytorch_install>\torch\_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "<path_to_pytorch_install>\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.
>>>
>>> b = torch.zeros (2, 2)
>>> b[1,1] = 2
>>>
>>> a = b.clone()
>>> a[0, 0] = param
>>> a.sum().backward()
>>>
>>> a = b.clone()
>>> a[0, 0] = param
>>> a.sum().backward()   # works

Applying this in a tweaked version of your code:
>>> import torch as t
>>> import torch
>>>
>>> param = torch.ones (1, requires_grad = True)
>>> a = torch.zeros (2, 2)
>>> # a[0, 0] = param   # we will do this with a_clone
... a[1, 1] = 2
>>>
>>> b = torch.tensor([[5], [6]], dtype=t.float, requires_grad = True)
>>> c = torch.tensor([[50], [12]], dtype=t.float, requires_grad = True)
>>>
>>> def calc_loss(A, B, C):
...     X =  torch.sqrt(torch.mean((torch.matmul(A,B)-C)**2))
...     return t.abs(X)
...
>>> optimizer = t.optim.Adam([param], lr=1)
>>> n = 10
>>>
>>> values = t.zeros(n, dtype=t.float)
>>> for i in range(n):
...     optimizer.zero_grad()
...     a_clone = a.clone()     # re-clone a in forward pass
...     a_clone[0, 0] = param   # re-"copy-slices" in forward pass
...     # loss = calc_loss(a, b, c)
...     loss = calc_loss (a_clone, b, c)  # use cloned copy in forward pass
...     print(f"calculated loss value is {loss}")
...     loss.backward()
...     print(f"gradient of the complex number are {param.grad}")   # move to after .backward()
...     optimizer.step()
...     values[i] = param.detach()
...
calculated loss value is 31.819805145263672
gradient of the complex number are tensor([-3.5355])
calculated loss value is 28.284271240234375
gradient of the complex number are tensor([-3.5355])
calculated loss value is 24.748737335205078
gradient of the complex number are tensor([-3.5355])
calculated loss value is 21.21320343017578
gradient of the complex number are tensor([-3.5355])
calculated loss value is 17.677669525146484
gradient of the complex number are tensor([-3.5355])
calculated loss value is 14.142135620117188
gradient of the complex number are tensor([-3.5355])
calculated loss value is 10.60660171508789
gradient of the complex number are tensor([-3.5355])
calculated loss value is 7.071067810058594
gradient of the complex number are tensor([-3.5355])
calculated loss value is 3.535533905029297
gradient of the complex number are tensor([-3.5355])
calculated loss value is 0.0
gradient of the complex number are tensor([nan])

Best.
K. Frank