It may somewhat defeat the purpose of using an LSTM if you are tossing the cell state. Have you checked GRU? Or you may try just using 2 linear layers and see if it gets the same or better results. Since you’re not passing the hidden state anyway, and since you’re not positionally encoding the data.