Could you check, if you are creating tensors inside the forward method using .cuda() or to('cuda')?
If so, make sure these new tensors are created on the currently used device by using the .device attribute of any parameter or the input via .to(x.device).