Since the calculations should be performed on the GPU, you would have to push the parameters and data to the device at some point.
Are you running out of memory on the GPU?
If so, you could try to lower your batch size or use torch.utils.checkpoint to trade compute for memory.