Answers.Body: <p>The conclusion that I finally arrived at was that I don't have enough data. The reason I believe this is true is that I slowly lowered model complexity and watched as overfitting decreased and underfitting increased.</p>
  <p>If the model was not overfitting, i.e. if the validation loss decreased along with the training loss, then the model underperformed. It was little better than a flip of a coin on binary classification.</p>
  <p>Increasing model complexity at all from this point lead to immediate overfitting.</p>
  <p>I believe that the solution is to find more data, or use a model that generalizes better with less data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T13:08:41.043Z
Answers.LastActivityDate: 2021-08-26T13:08:41.043Z
Answers.UserDisplayName: rocksNwaves
Body: <p>I've been working on learning about NLP via a beginners competition on Kaggle.</p>
  <p>I first trained a model with an embedding layer and then a simple linear layer. I actually got way better than a flip of the coin with this model, so I decided to try to step it up with an LSTM.</p>
  <p>What happened was that training loss decreased and then palteaued while validation loss never decreased at all.</p>
  <p>In the case of overfitting, I would expect validation loss to decrease for a while but then either remain steady or perhaps even increase as the model starts to overfit.</p>
  <p>I can't find any reason for the strange loss curves I'm seeing:</p>
  <p><a href="https://i.sstatic.net/kubq8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kubq8.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/UFb6a.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UFb6a.png" alt="enter image description here" /></a></p>
  <p><strong>What could cause such a phenomenon?</strong></p>
  <p>I would be happy to share my network architecture and training code if there isn't a straightforward answer (I know there usually isn't).</p>
  
Tags: <natural-language-processing><long-short-term-memory>
Title: What is the reason for a training loss that drops but validation that NEVER does
UserDisplayName: rocksNwaves
ThreadText: # What is the reason for a training loss that drops but validation that NEVER does
  
  (Asked by: username_0 on 2021-08-16)
  
  I've been working on learning about NLP via a beginners competition on Kaggle.
  
  I first trained a model with an embedding layer and then a simple linear layer. I actually got way better than a flip of the coin with this model, so I decided to try to step it up with an LSTM.
  
  What happened was that training loss decreased and then palteaued while validation loss never decreased at all.
  
  In the case of overfitting, I would expect validation loss to decrease for a while but then either remain steady or perhaps even increase as the model starts to overfit.
  
  I can't find any reason for the strange loss curves I'm seeing:
  
  [![enter image description here](https://i.sstatic.net/kubq8.png)](https://i.sstatic.net/kubq8.png)
  [![enter image description here](https://i.sstatic.net/UFb6a.png)](https://i.sstatic.net/UFb6a.png)
  
  **What could cause such a phenomenon?**
  
  I would be happy to share my network architecture and training code if there isn't a straightforward answer (I know there usually isn't).
  
  **Tags:** natural-language-processing, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-08-26. Score: 0 | Accepted answer
  
  The conclusion that I finally arrived at was that I don't have enough data. The reason I believe this is true is that I slowly lowered model complexity and watched as overfitting decreased and underfitting increased.
  
  If the model was not overfitting, i.e. if the validation loss decreased along with the training loss, then the model underperformed. It was little better than a flip of a coin on binary classification.
  
  Increasing model complexity at all from this point lead to immediate overfitting.
  
  I believe that the solution is to find more data, or use a model that generalizes better with less data.
  
  
  ---
  
  