Answers.Body: <p>The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen.  This means that your model has sufficient capacity to learn (i.e., it is not underfitting) and that increasing the model's capacity (i.e., increasing the number of layers or increasing the number of neurons per layer) is unlikely to help.</p>
  <p>Instead, you should work towards reducing overfitting.  The best way to do this is by collecting more real data.  You can also consider Data Augmentation or simulating synthetic data.  Other approaches such as dropout and L1 and/or L2 regularization may also be helpful.  This article <a href="https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/" rel="nofollow noreferrer">&quot;An Overview of Regularization Techniques in Deep Learning (with Python code)&quot;</a> is a nice overview of these methods.  Even though you have used dropout, you can tune the dropout rate and include dropout at one or more layers in your model.</p>
  <p>Finally, ensemble methods have shown success in improving accuracy in many (but not all) settings, compared to single models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T14:50:05.793Z
Answers.LastActivityDate: 2022-10-25T14:50:05.793Z
Answers.UserDisplayName: Snehal Patel
Answers.Body: <p>I fully agree with @Snehal Patel's comment but want to expand on the capacity of the model a bit.</p>
  <p>The easiest approach to reduce overfitting is to reduce the capacity of the model, i.e. the size of its layers.</p>
  <p>Therefore, in your first sequential model, instead of three linear with [1024, 512, 256] nodes, it would probably be better to have just two with [512, 256] nodes.</p>
  <p>The same applies to the second sequential model. Just using one linear layer probably reduces the capacity of the model to overfit to the data.</p>
  <p>I hope those changes work out for you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T21:16:15.497Z
Answers.Text: Good call, @RolandSt!  I agree that is the high yield move.
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T21:12:08.623Z
Answers.LastActivityDate: 2022-10-25T21:12:08.623Z
Answers.UserDisplayName: RolandSt
Body: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use <code>nn.CrossEntropyLoss()</code> as my loss function. This is my network’s configuration</p>
  <pre><code>Model(
    (fc): Sequential(
      (0): Linear(in_features=800, out_features=1024, bias=True)
      (1): ReLU()
      (2): Linear(in_features=1024, out_features=512, bias=True)
      (3): ReLU()
      (4): Linear(in_features=512, out_features=256, bias=True)
      (5): ReLU()
    )
    (lstm): LSTM(256, 1024, bidirectional=True)
    (hidden2tag): Sequential(
      (0): Linear(in_features=2048, out_features=1024, bias=True)
      (1): ReLU()
      (2): Linear(in_features=1024, out_features=500, bias=True)
    )
  )
  </code></pre>
  <p>This is what my loss looks like. It increases after reaching a particular value no matter what setting I use. I’ve used k-fold cross–validation but the loss/accuracy across all folds stays the same, so I’m assuming that there’s no issue in the distribution of train/val splits.</p>
  <p>From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every val epoch))</p>
  <p><a href="https://i.sstatic.net/DM3gZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DM3gZ.png" alt="Loss wrt. iteration" /></a></p>
  <p>I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve</p>
  <p><a href="https://i.sstatic.net/MLnNF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MLnNF.png" alt="lr change" /></a></p>
  <p>I’ve tried using dropout but that gives higher validation loss values instead</p>
  <p>What can be done here in order to further decrease the loss?</p>
  
Tags: <deep-learning><classification><long-short-term-memory><pytorch><loss>
Title: Val loss doesn’t decrease after a certain number of epochs
UserDisplayName: helloworld
ThreadText: # Val loss doesn’t decrease after a certain number of epochs
  
  (Asked by: username_0 on 2022-10-25)
  
  I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use `nn.CrossEntropyLoss()` as my loss function. This is my network’s configuration
  
  ```
  Model(
    (fc): Sequential(
      (0): Linear(in_features=800, out_features=1024, bias=True)
      (1): ReLU()
      (2): Linear(in_features=1024, out_features=512, bias=True)
      (3): ReLU()
      (4): Linear(in_features=512, out_features=256, bias=True)
      (5): ReLU()
    )
    (lstm): LSTM(256, 1024, bidirectional=True)
    (hidden2tag): Sequential(
      (0): Linear(in_features=2048, out_features=1024, bias=True)
      (1): ReLU()
      (2): Linear(in_features=1024, out_features=500, bias=True)
    )
  )
  
  ```
  
  This is what my loss looks like. It increases after reaching a particular value no matter what setting I use. I’ve used k-fold cross–validation but the loss/accuracy across all folds stays the same, so I’m assuming that there’s no issue in the distribution of train/val splits.
  
  From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every val epoch))
  
  [![Loss wrt. iteration](https://i.sstatic.net/DM3gZ.png)](https://i.sstatic.net/DM3gZ.png)
  
  I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve
  
  [![lr change](https://i.sstatic.net/MLnNF.png)](https://i.sstatic.net/MLnNF.png)
  
  I’ve tried using dropout but that gives higher validation loss values instead
  
  What can be done here in order to further decrease the loss?
  
  **Tags:** deep-learning, classification, long-short-term-memory, pytorch, loss
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-10-25. Score: 2 | Accepted answer
  
  The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen. This means that your model has sufficient capacity to learn (i.e., it is not underfitting) and that increasing the model's capacity (i.e., increasing the number of layers or increasing the number of neurons per layer) is unlikely to help.
  
  Instead, you should work towards reducing overfitting. The best way to do this is by collecting more real data. You can also consider Data Augmentation or simulating synthetic data. Other approaches such as dropout and L1 and/or L2 regularization may also be helpful. This article ["An Overview of Regularization Techniques in Deep Learning (with Python code)"](https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/) is a nice overview of these methods. Even though you have used dropout, you can tune the dropout rate and include dropout at one or more layers in your model.
  
  Finally, ensemble methods have shown success in improving accuracy in many (but not all) settings, compared to single models.
  
  
  ---
  
  ### Answer by username_2 on 2022-10-25. Score: 1
  
  I fully agree with @username_1's comment but want to expand on the capacity of the model a bit.
  
  The easiest approach to reduce overfitting is to reduce the capacity of the model, i.e. the size of its layers.
  
  Therefore, in your first sequential model, instead of three linear with [1024, 512, 256] nodes, it would probably be better to have just two with [512, 256] nodes.
  
  The same applies to the second sequential model. Just using one linear layer probably reduces the capacity of the model to overfit to the data.
  
  I hope those changes work out for you.
  
  > **Comments:**
  > **username_1**: Good call, @username_2!  I agree that is the high yield move.  
  
  ---
  
  