Solved by vdw in post #2 
              
              
                The encoder outputs ARE the hidden states at each time step. Look at the following sketch of an LSTM cell: 
 [1024px-The_LSTM_cell] 
There is not “extra” output, it’s just the current hidden state h_t. For example, if you define an lstm with num_layers=0 and batch_first=False, and given, say, 
outpu…