Two quick comments from a quick look (it might not be all):

You shuffle your datapoints for batching, that probably mixes badly with the statefulness you mention.
The retain_graph thing you mention also comes from the statefulness. You could solve that using “detach”, but then you’d not get back propagation through timesteps (or any signal for the hidden hidden connections).

What you probably want to do is to feed several timesteps into your RNN and then extract one prediction.
I would recommend to look at the Time series prediction example 129 or look at how fast.ai do that in their language modelling 38 (keep in mind that fast.ai’s intention is to provide a efficient access to their excellent ideas rather than be a showcase of PyTorch coding style).
Best regards
Thomas