Arbitrary user-specified forward/backwards Python functions are run on the CPU. If you use the PyTorch API to operate on your Tensors, each operation has a CUDA version that will be called if your tensor is on a GPU.