Thanks for the answer. I’m using time series data rather than LM but it sounds like the 2nd schema is still preferred.
Is it important that the sequences are trained in order so that the relevant hidden state is reused, or is it common to shuffle them? It seems the hidden state may not be used anyways if they’re fed in parallel batches.



 jekbradbury:

and is used in the word_language_model example


Ok, I see. I was just looking at the way get_batch 28 was indexing, but it looks like in training/evaluation it loops using range with a bptt stepsize.