The input and grid should be on the same device.
If you are creating one of these tensors manually in the forward or pass it to the forward method, make sure to transfer it to the same device, e.g. by using:
grid = grid.to(x.device)