I don’t know where the overhead comes from.
There is a flat cost of ~20us for any operation on a cuda tensor, vs an unmeasurable small time on CPU.
E.g. use
x = torch.linspace(-1, 1, 128)
%timeit torch.sin(x)

for me, that is ~2us.
repeat with
x = x.cuda()
%timeit torch.sin(x)

=> ~21us.
The computational overhead isn’t putting x and y on the GPU.