Thanks again for the quick reply!
The reason I dont feed the whole sequence at the same time to the network is that I want to try this out as “online training” where I would recieve one timestep-value at a time. As the sequence of data I have to evaluate my model on is already complete, would I get the same results if I just passed the whole sequence instead of one timestep at a time (with retain_graph=True in the “one step at a time” version)? I still want the prediction for each timestep so I can evaluate how long it takes for the network before it makes accurate predictions.
In keras I did this by giving the LSTM the argument return_sequences=True. This is information I can probably easily find so no need to answer this if it would take you too much time.
And if it would work just as well to send in the whole sequence I would still have to go back to the model that performs one timestep at a time if I would try it out with online training. But I would rather do this if it is faster than using the “online version” for evaluating my model on complete sequences.

The gradients will be more accurate (correct gradient wrt full sequence)


That said, I still am not sure why you want to backward for each single each time step. It may save you some memory, but it takes the same, if not longer, time to do so. And it gives incorrect gradients.

I dont quite get this part. Do you mean when using detach() instead of retain_graph=True in the second quote? Because then I understand!
Thanks again for all the help!