Thank you for your response. I changed the loss function to CrossEntropyLoss and removed the softmax activation in the forward method, however, I am getting more or less the same results. It is around the same train/val accuracy as before (around 0.74), however, the loss is now considerably higher hovering around 0.55 as opposed to .12. I am thinking this has something to do with the LSTM layers. Do I have to pass randomized hidden states to the layers, or does PyTorch deal with this automatically. Unfortunately, a lot of the implementation details are abstracted away when using Keras, so I am not entirely sure how they deal with this either.