pancho111203:

The last thing I tried is setting seq_len and batch_size to 1, and passing the hidden states on every iteration:


A good idea, but the weights get updated at every timestep and as this involves backpropagating all the way back to the sequence start, it is gets pretty slow.
Might I suggest a compromise? Keep batch_size=1 and instead of setting seq_len to total_seq_len or to 1, try some value in between. E.g. if you set seq_len to total_seq_len/4, then the weights will be updated four times per epoch which will run a lot faster than once per timestep. Note that the fourth batch will run noticeably slower than the first because it has further to backpropagate.



 pancho111203:

In Keras this works without having to do anything


I don’t think Keras does backpropagate right back to the sequence start unless you feed in the whole sequence at once. I am pretty sure it only ever backpropagates to the beginning of each batch.
I think the common approach is to set the seq_len to be relatively large, to call backward without the retain_graph option, and to hope for the best. Keeping the hidden state from one batch to the next will provide a summary of the previous history that the model can learn to use. So the model can learn some long term relationships, but it can’t learn them as finely as it might if it could backpropagate all the way through.
Another approach (which will be a little harder to implement) is described in this paper https://arxiv.org/abs/1705.08209 6 Among other things it involves randomising the seq_len for each batch.
If you need to run in an online learning setting, i.e. updating the weights at every timestep in an efficient manner, then the following paper can help https://arxiv.org/abs/1702.05043 4