Thanks for the great example! I am now able to train the network. But I still have a question about detach() and retain_graph:
The current prediction should depend on all previous timesteps. If I dont detach the hidden state and train the network with backward(retain_graph=True)  will the updated parameters not be updated to a better value as gradient descent can use the history from previous timesteps? From testing this the results are quite similar but still vary a little bit.
Just trying to clear up a few things in my head. Is the answer for this question different if I would use a simple RNN as it differs a lot from a LSTM?
I can probably research these questions myself but I just dont have enough time at the moment and it bothers me to use something I dont understand.
Thanks in advance!