These are quite orthogonal things.
If you’re using python, the Tensor is a python object. But if the Tensor is a GPU tensor, then the memory it works with is on the GPU. So when you perform operations on it, they will run on the GPU.
This discussion here about the GIL is unrelated. You can use the underlying cpp Tensor object and use it without touching the python object as well.