I like to make sure that the subsequences in each batch are the continuations of the subsequences in the previous batch. That way I can use the final hidden state of one batch as the initial hidden state of the next batch.
If you do that then you must either


detach 16 the hidden state between batches in order to cut off backpropagation. The fact that the hidden state is retained allows the model to learn something about the influence of previous history, but not as much as full BPTT would allow.
or use the retain_graph=True option when calling .backward(), but in this case each batch will take longer than the previous batch because it will be backpropagating all the way through time to the beginning of the first batch.