albanD:

In pytorch, it is different to have different Tensor and have different memory.
When you do b = a.detach() , a and b are two completely different Tensors that look at the same memory.
Just like b = a.view(-1) are two different Tensors that look at the same memory.


That was really helpful. Thank you.
I guess to answer my own question, I do not need to call clone it seems. The new tensor tracks its own memory space for its gradients automagically (without interference, so it doesn’t collect the same gradients for both graphs in the same place):
import torch

a = torch.tensor([2.0], requires_grad=True)
b = a.detach()
b.requires_grad = True

la = (5.0 - a)**2
la.backward()
print(f'a.grad = {a.grad}')

lb = (6.0 - b)**2
lb.backward()
print(f'b.grad = {b.grad}')

result:
a.grad = tensor([-6.])
b.grad = tensor([-8.])

of course you guys would have thought of a good implementation of this! Not surprised!