ZhengRui:

inp = torch.rand(3, requires_grad=True).to(0)


Here is your problem. Your graph of the 1st example is
rand_result[leaf, requires_grad=True] ==> inp[intermediate result from rand_result.to()] ==> ...

And intermediate result do not keep gradient by default.
You should instead do
device = torch.device('cuda')
inp = torch.rand(3, device=device, requires_grad=True)