Answers.Body: <p>Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question.</p>
  <p>I think some of the answers misunderstood the OP's intention.</p>
  <p>Over fitting is used as a means to test the complexity of the model - if a model cannot overfit a small dataset then it's likely not able to generalize well.</p>
  <p>It's not that OP misunderstood the meaning of over fitting.</p>
  <p>For Instance, I think this discussion is relevant: <a href="https://stats.stackexchange.com/questions/492165/what-to-do-when-a-neural-network-cannot-overfit-one-training-sample">https://stats.stackexchange.com/questions/492165/what-to-do-when-a-neural-network-cannot-overfit-one-training-sample</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T02:59:14.393Z
Answers.LastActivityDate: 2021-02-22T02:59:14.393Z
Answers.UserDisplayName: Tom Charles Zhang
Answers.Body: <p>Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered.</p>
  <p>Naturally, you do not want this situation. What you want is a high training accuracy and a very low validation loss, which implies a high validation accuracy.</p>
  <p>The first task is to ensure that your model gets a high training accuracy. Once that is accomplished, you can work on getting a low validation loss.</p>
  <p>If your model is overfitting, there are several ways to mitigate the problem. First, start out with a simple model. If you have a lot of dense layers with a lot of neurons, reduce the hidden dense layers to a minimum. Typical just leave the top dense layer used for final classification. Then see how the model trains. If it trains well, look at the validation loss and see if it is reducing in the later epochs. If the model does not train well, add a dense layer followed by a dropout layer. Use the level of dropout to adjust for overfitting. If it still trains poorly, increase the number of neurons, and train again. If that fails, add another dense hidden layer with fewer neurons than the previous layer followed by another dropout layer.</p>
  <p>Another method to combat overfitting is to add regularizers to the dense layers. Documentation for that is <a href="https://keras.io/api/layers/regularizers/" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-04T17:06:31.720Z
Answers.LastActivityDate: 2020-10-04T20:25:43.173Z
Answers.LastEditDate: 2020-10-04T20:25:43.173Z
Answers.UserDisplayName: Gerry P
Answers.Body: <p>I think that your misuse of the term <em>over-fitting</em> made the question vague. In layman terms, over-fitting means that a model fails to generalize to real-world scenarios, but is accurate with the training set.</p>
  <p>Using a dropout layer means that the network cuts down on neurons that are used for training, in this case, 50%.</p>
  <p>Recommendations for improving training accuracy would be:</p>
  <ul>
  <li>Transfer learning</li>
  <li>Adding more layers to the network (also shifting number of neurons helps)</li>
  <li>Adding epochs</li>
  <li>Changing optimizer (Adam and RMSProp are some of my suggestions)</li>
  <li>Adding activation layers</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-06T07:15:42.820Z
Answers.Text: How did you determine your network was over fitting? Did you use a validation set and if you did did the validation error start to increase as you increased the number of epochs? You need to be sure you are over fitting before you start to change things. Can you provide the training loss and validation loss data for each epoch. That will help to tell what is going on.
Answers.UserDisplayName: Gerry P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-17T14:39:52.947Z
Answers.LastActivityDate: 2020-10-04T20:28:55.493Z
Answers.LastEditDate: 2020-10-04T20:28:55.493Z
Answers.UserDisplayName: Simbarashe Timothy Motsi
Body: <p>I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion. </p>
  
  <p>PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T08:28:48.333Z
Comments.Text: What's the size of your output layer? Also, I guess with overfitting you imply zero training error?? I would suggest adding dropout before the last hidden layer. P.S. writing as an answer, because I am unable to post comments
Comments.UserDisplayName: Karan Dhingra
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-14T16:49:27.557Z
Comments.Text: Can you overfit with no dropout but half the neurons? Can you overfit with double the amount of neurons and 50% dropout?
Comments.UserDisplayName: Martin Thoma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T02:45:46.950Z
Comments.Text: What do you think “overfit” means? I suspect that you’re using the term incorrectly. In particular, overfitting is *bad*.
Comments.UserDisplayName: Stella Biderman
Tags: <convolutional-neural-networks><linear-regression><dropout>
Title: What to do if CNN cannot overfit a training set on adding dropout?
UserDisplayName: user12754
ThreadText: # What to do if CNN cannot overfit a training set on adding dropout?
  
  (Asked by: username_0 on 2018-02-15)
  
  I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion.
  
  PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum.
  
  **Tags:** convolutional-neural-networks, linear-regression, dropout
  
  > **Comments:**
  > **username_2**: Can you overfit with no dropout but half the neurons? Can you overfit with double the amount of neurons and 50% dropout?  
  > **username_3**: What do you think “overfit” means? I suspect that you’re using the term incorrectly. In particular, overfitting is *bad*.  
  > **username_1**: What's the size of your output layer? Also, I guess with overfitting you imply zero training error?? I would suggest adding dropout before the last hidden layer. P.S. writing as an answer, because I am unable to post comments  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2021-02-22. Score: 2
  
  Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question.
  
  I think some of the answers misunderstood the OP's intention.
  
  Over fitting is used as a means to test the complexity of the model - if a model cannot overfit a small dataset then it's likely not able to generalize well.
  
  It's not that OP misunderstood the meaning of over fitting.
  
  For Instance, I think this discussion is relevant: <https://stats.stackexchange.com/questions/492165/what-to-do-when-a-neural-network-cannot-overfit-one-training-sample>
  
  
  ---
  
  ### Answer by username_5 on 2020-09-04. Score: 1
  
  Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered.
  
  Naturally, you do not want this situation. What you want is a high training accuracy and a very low validation loss, which implies a high validation accuracy.
  
  The first task is to ensure that your model gets a high training accuracy. Once that is accomplished, you can work on getting a low validation loss.
  
  If your model is overfitting, there are several ways to mitigate the problem. First, start out with a simple model. If you have a lot of dense layers with a lot of neurons, reduce the hidden dense layers to a minimum. Typical just leave the top dense layer used for final classification. Then see how the model trains. If it trains well, look at the validation loss and see if it is reducing in the later epochs. If the model does not train well, add a dense layer followed by a dropout layer. Use the level of dropout to adjust for overfitting. If it still trains poorly, increase the number of neurons, and train again. If that fails, add another dense hidden layer with fewer neurons than the previous layer followed by another dropout layer.
  
  Another method to combat overfitting is to add regularizers to the dense layers. Documentation for that is [here](https://keras.io/api/layers/regularizers/).
  
  
  ---
  
  ### Answer by username_6 on 2018-05-17. Score: 0
  
  I think that your misuse of the term *over-fitting* made the question vague. In layman terms, over-fitting means that a model fails to generalize to real-world scenarios, but is accurate with the training set.
  
  Using a dropout layer means that the network cuts down on neurons that are used for training, in this case, 50%.
  
  Recommendations for improving training accuracy would be:
  
  * Transfer learning
  * Adding more layers to the network (also shifting number of neurons helps)
  * Adding epochs
  * Changing optimizer (Adam and RMSProp are some of my suggestions)
  * Adding activation layers
  
  > **Comments:**
  > **username_5**: How did you determine your network was over fitting? Did you use a validation set and if you did did the validation error start to increase as you increased the number of epochs? You need to be sure you are over fitting before you start to change things. Can you provide the training loss and validation loss data for each epoch. That will help to tell what is going on.  
  
  ---
  
  