Body: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use <code>nn.CrossEntropyLoss()</code> as my loss function. I noticed that my model was overfitting (validation loss increased after some epochs) and hence I used a dropout (p=0.4) in the last FCN layer. I’m now seeing a higher loss (for both train and val)</p>
  <p>The green line denotes training without dropout</p>
  <p><a href="https://i.sstatic.net/yX8y2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yX8y2.png" alt="enter image description here" /></a></p>
  <p>From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every train epoch))</p>
  <p>Is this behavior justified or am I doing something wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T03:02:30.900Z
Comments.Text: What is the plot all the way on the left?  Can you display curves for training and validation on the same plot (one for with and one for without dropout)?  As an aside, I have never seen anything like the validation loss with dropout that you are showing.  Very bizarre pattern!
Comments.UserDisplayName: Snehal Patel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T03:19:53.900Z
Comments.Text: Thanks for clarifying that the iter loss is the loss recorded after every batch, but you should specify that it is the `train loss` (recorded after every batch).
Comments.UserDisplayName: Snehal Patel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T03:28:31.387Z
Comments.Text: Sorry for the confusion. I've updated my question. The logs were rotated. I'll re-run the experiments and update them with the plots you've asked
Comments.UserDisplayName: helloworld
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-27T03:08:45.523Z
Comments.Text: I'm stumped as to what is happening with your learning curves, but you may try moving the dropout layer around and tuning the dropout rate to see how the behavior changes.  Dropout is supposed to reduce / slow down overfitting.  They can slow down learning altogether, but this is very bizarre, especially the validation loss after adding dropout.
Comments.UserDisplayName: Snehal Patel
Tags: <deep-learning><classification><pytorch><regularization><dropout>
Title: Higher validation loss after using Dropout
UserDisplayName: helloworld
ThreadText: # Higher validation loss after using Dropout
  
  (Asked by: username_0 on 2022-10-25)
  
  I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use `nn.CrossEntropyLoss()` as my loss function. I noticed that my model was overfitting (validation loss increased after some epochs) and hence I used a dropout (p=0.4) in the last FCN layer. I’m now seeing a higher loss (for both train and val)
  
  The green line denotes training without dropout
  
  [![enter image description here](https://i.sstatic.net/yX8y2.png)](https://i.sstatic.net/yX8y2.png)
  
  From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every train epoch))
  
  Is this behavior justified or am I doing something wrong?
  
  **Tags:** deep-learning, classification, pytorch, regularization, dropout
  
  > **Comments:**
  > **username_1**: What is the plot all the way on the left?  Can you display curves for training and validation on the same plot (one for with and one for without dropout)?  As an aside, I have never seen anything like the validation loss with dropout that you are showing.  Very bizarre pattern!  
  > **username_1**: Thanks for clarifying that the iter loss is the loss recorded after every batch, but you should specify that it is the `train loss` (recorded after every batch).  
  > **username_0**: Sorry for the confusion. I've updated my question. The logs were rotated. I'll re-run the experiments and update them with the plots you've asked  
  > **username_1**: I'm stumped as to what is happening with your learning curves, but you may try moving the dropout layer around and tuning the dropout rate to see how the behavior changes.  Dropout is supposed to reduce / slow down overfitting.  They can slow down learning altogether, but this is very bizarre, especially the validation loss after adding dropout.  
  
  ---
  
  ## 0 answers
  
  