The idea of creating the sequences inside the Dataset is valid and you could either stick to your loop or have a look at e.g. this simple post 3 to see, how to create “sliding windows”.
I would not recommend to put an nn.LSTM into an nn.Sequential container, since it’s working with multiple inputs and outputs.
Create a custom nn.Module instead so that you could use e.g. the last time step of the lstm output to feed into the linear layer.



 Laura_Montalvo:

And when the Linear layer is commented then the CrossEntropyLoss cant be applied because as the input is two sequences is a 2D tensor, while CrossEntropyLoss expects just 1D tensor…


I assume you are using the complete hidden state as the model output, which won’t work in this case.
If each sequence has a single target, you could e.g. use the last time step of the lstm output or reduce the temporal dimension in any other way (e.g. taking the mean).