I am not sure if LSTM is the correct module for something like that directly.
But maybe the last output of the LSTM contains the information enough to compute the “single vector of length 256”. You could try index_select 4 and having several linear modules in order to map the sizes of the vectors.