Hi, thanks for your response. I am sure I can avoid using a DataLoader but it makes my framework simpler (unless it really does “complicate the state assignments”). Everything in the DataLoader is part of the preprocessing of data and that includes the LSTM. The task is simple; for some point in space, I accumulate information temporally in the forms of vectors. The LSTMs task is to give me one feature vector summarizing this sequence. The result is my training data ready for my actual net responsible for making the predictions. The two main reasons why I want my framework to work as described:


DataLoader makes it convenient to parallelize the preprocessing, and that includes the LSTM.
As this will be computationally intensive, I do not want to load all relevant vectors of the sequences into memory; instead I want to do this incrementally.

I believe I have found the answer to my original question. I had not realized what I was looking for was called “Statefule LSTM”, and once I had realized this it was easy to search for examples on how to do this. However, you mentioned that there are difficulties in executing the LSTM in the DataLoader. Is there I way to do this?