Solved by alorentemur in post #2 
              
              
                Are you making use of the hidden state? 
Maybe you could use the lstm like it’s done in this tutorial: 
https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html