Thanks for your reply! Yes, it is important for me to be able to use multiple workers in the DataLoader. I am not sure I see your point. I will write out what I think is happening so that you can tell me where I am going wrong.
What I imagined:

DataLoader workers with LSTM load data [data1, …, datan], and then these tensors would have the LSTM autograd states [LSTM1, …, LSTMn].
DataLoader returns [(data1 w/ LSTM1), …, (datan w/ LSTMn)], where (datai w/ LSTMi) just means LSTMi states are stored in the datai tensor.
Then through SomeNet: [(data1 w/ LSTM1), …, (datan w/ LSTMn)] → [(data1 w/ [LSTM1, SomeNet1]), …, (datan w/ [LSTMn, SomeNetn])

So if this works the optimizer optimizing over [LSTM.parameters(), SomeNet.parameters()] should have access to the full history of autograd states. Considering the data generated by each worker get passed independently through SomeNet, it should be okay that the autograd states of datai are independent of those fo dataj. Which step/assumption is incorrect?
Perhaps there was a misunderstanding? When I talked about passing sequences incrementally through an LSTM what I meant was within one worker, I am loading in data incrementally and thus the LSTM must be stateful, and I didn’t mean multiple workers work on different sections of the same sequence.
Looking forward to your reply.