So in case someone else ever has this same question, this is what a month’s worth of research has led me to believe:
LSTM’s are prone to overfitting unless you have a lot of data. This Kaggle competition data includes ~7600 tweets, and I just don’t think that’s enough.
What leads me to believe this is that I could perfectly overfit 20 samples, 10 with positive and 10 with negative labels. But when training on the whole dataset, I had two options: Overfit or Underfit.

Simplify the model to the extreme. With the following parameters, the model did not overfit but severely underperformed (barely better than a flip of the coin).

Embedding size: 10
Lstm hidden dimension: 8
Added learning rate decay with a factor of 0.1 and patience of 5 epochs.




image729×265 27.1 KB



A slight improvement was to train a nn.EmbeddingBag layer as in my original post, then transfer the trained weights over to the LSTM’s nn.Embedding layer.


Adding any more complexity to the model resulted in overfitting after ~5-30 epochs depending on the amount added.


What I will try next is to make the model bidirectional, but I have no idea how to mask the loss of a bidirectional LSTM. I guess that’s a different question.