Can you try removing the lr_scheduler()? I was having issues with that before. You may also need to consider adding .detach() to your model outputs before any evaluation metrics. Additionally, in an RNN, if I recall, you should be detaching the hidden layers between runs or the graph keeps getting expanded.
Lastly, is there a reason for moving the evaluation to the cpu?