Thanks for the explanation! They are pretty clear but I still have some questions about it because the result is kind of weird to me:
# scenario 1
>>> a = torch.rand(3,2,requires_grad=True)
>>> b = torch.rand(2,3)
>>> loss = 10 - (a @ b).sum()
>>> loss.backward()
>>> a.grad  
tensor([[-0.9418, -1.4957],
        [-0.9418, -1.4957],
        [-0.9418, -1.4957]])  # expected, has gradient
>>> b = torch.rand(2,3)   # expected, no gradient
>>>
 
...
# scenario 2
>>> a = torch.rand(3,2,requires_grad=True).cuda()
>>> b = torch.rand(2,3).cuda()
>>> loss = 10 - (a @ b).sum()
>>> loss.backward()
>>> a.grad   # expected since operation `cuda` makes it a non-leaf, but why?
>>> b.grad  # expected, no gradient
>>>

...
# scenario 3
>>> ll = nn.Linear(3,3).cuda()
>>> inp = torch.rand(3).cuda()
>>> loss = (10 - ll(inp)).sum()
>>> loss.backward()
>>> ll.weight.grad
tensor([[-0.8497, -0.1128, -0.8081],
        [-0.8497, -0.1128, -0.8081],
        [-0.8497, -0.1128, -0.8081]], device='cuda:0')  # why it has gradient?


As you see in scenario 2, by moving the tensor cuda I was not able to get the gradient without using a.retain_grad() or device='cuda'. In scenario 3 moving the layer to cuda the gradient of the weight is still there. May I know why it is designed this way? I know that result of an operation like multiplication does not requires its gradient to be calculated since intermidate values’s gradient usually not useful, but why does operation like cuda also cause the tensor to be non-leaf but stay as leaf for weight in layer?