You need to make sure that the data type and device is the same for every inputs.
Then whether you model should on cpu or gpu I don’t know. That is your choice. If the rest of the model is on gpu, I would say run everything on gpu.