Solved by tom in post #4 
              
              
                So one thing you need to do to get it to work is to pass batch_first to the LSTM instantiation if that is what you want. 
While taking the last timestep (as you do with lstm_out[:, -1, :]) is certainly a common way to set up sequence-to-one problems (assuming your inputs are of the same length), I w…