No it’s a new variable, that is on the gpu and that contains the same thing as the original cpu tensor.
Is the following sample making it clearer?
import torch

a = torch.rand(1, requires_grad=True)
b = a.cuda()
c = 3*b

c.sum().backward()
print(c.grad) # None:        Not a leaf
print(b.grad) # None:        Not a leaf
print(a.grad) # tensor([3]): A leaf


b = torch.rand(1, requires_grad=True).cuda()
c = 3*b

c.sum().backward()
print(c.grad) # None:        Not a leaf
print(b.grad) # None:        Not a leaf