Answers.Body: <p>I identified the origin of this overfitting..</p>
  
  <h2>Origins</h2>
  
  <p>I tried a lot of models, putting more and more dropout, simplifying as much as I could.</p>
  
  <p>No matter what I did, after a few epoch of good learning, invariably my loss function was going up. I tried simpler and simpler models, always the same overfitting behavior.
  What bugged me at that moment is that no matter what kind of model I used, how deep or how complex, always the accuracy was fine, stabilized at some nice level.</p>
  
  <p>So I tried the simplest model I could imagine : Input => Dense with 3 hidden units =>  Output. Finally I got random results, with a 33% accuracy !
  From here, I guilt again my network, layer by layer, to see which one was causing the overfitting.</p>
  
  <p>And it was the <strong>Embedding layer</strong>.</p>
  
  <p>Even with a simple network like Input => Embeddings => Dense with 3 hidden units =>  Output, the model was overfitting.</p>
  
  <h2>How to solve it</h2>
  
  <p>In Keras, simply instantiate the Embeddings layer with <code>trainable=False</code>. After doing this, no more overfit.</p>
  
  <p>In my opinion, this is quite counter-intuitive : I want my embeddings to evolve with the data I show to the network. But look like I can't...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T05:33:16.923Z
Answers.LastActivityDate: 2018-10-19T05:33:16.923Z
Answers.UserDisplayName: Astariul
Answers.Body: <p>I think sometimes it can also help to examine your test and training sets. 
  Fundamentally, your data was produced by an underlying process/system that has certain properties. The system can have many "states" and all the possible states form the state space. If you have really tried things like dropout and regularization, my guess would be that the test set is somehow different from your train set. It is possible that your training set only takes samples from one part of the state space (AKA, your samples might all be similar in the training set and the test set has different samples - imagine you are classifying humans and all of your training samples have a class label of 1 meaning all the training samples have humans in them -> and all your test samples have no humans in them Good luck with that!).  Some questions to ask: </p>
  
  <ol>
  <li><p>Are you combining datasets from different sources? If so: 
     If you have "n" sources of data, you need to make sure that your training set has many samples from each of the "n" sources of data and your test set has samples from each of the "n" sources. </p></li>
  <li><p>Are you shuffling your data enough and randomly putting samples in both the training and test sets? This relates to the human example I gave, make sure your training set has a little bit of everything (different combinations of inputs and/or outputs) and your testing set has a little bit of everything (different combinations of inputs and/or outputs). </p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T23:37:44.197Z
Answers.Text: Thanks for your answer. However I don't think the problem is from the data : I am using the [SNLI Dataset](https://nlp.stanford.edu/projects/snli/), and only this one. This is a good and well balanced dataset.
  
  The test and validation set are already defined by SNLI, and I always shuffle my data.
Answers.UserDisplayName: Astariul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T23:15:50.080Z
Answers.LastActivityDate: 2018-10-15T23:15:50.080Z
Answers.UserDisplayName: Ali_Ayub
Body: <p><em>Following-up my <a href="https://ai.stackexchange.com/questions/8338/interpretation-of-a-good-overfitting-score">question</a> about my over-fitting network</em> </p>
  
  <p>My deep neural network is over-fitting : 
  <a href="https://i.sstatic.net/6xKGF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6xKGF.png" alt="enter image description here"></a></p>
  
  <p>I have tried several things : </p>
  
  <ul>
  <li>Simplify the architecture</li>
  <li>Apply more (and more !) Dropout</li>
  <li>Data augmentation</li>
  </ul>
  
  <p>But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.</p>
  
  <p>I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :
  <a href="https://i.sstatic.net/nh4Bs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nh4Bs.png" alt="enter image description here"></a> </p>
  
  <p>My question is : <strong>This accuracy of 70% is the best my model can reach ?</strong></p>
  
  <p>If yes :</p>
  
  <ul>
  <li>Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?</li>
  <li>My only option to improve the accuracy is then to change my model, right ?</li>
  </ul>
  
  <p>If no :</p>
  
  <ul>
  <li>What are my options to improve this accuracy ?</li>
  </ul>
  
  <p><em>I'v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can't exceed this limit, even though it seems easy to my network to reach it (short convergence time)</em></p>
  
  <h2>Edit</h2>
  
  <p>Here is the Confusion matrix :</p>
  
  <p><a href="https://i.sstatic.net/S2Hro.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/S2Hro.png" alt="enter image description here"></a></p>
  
  <p>I don't think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : <a href="https://nlp.stanford.edu/projects/snli/" rel="nofollow noreferrer">SNLI Dataset</a> </p>
  
  <p>And here is the learning curve :</p>
  
  <p><a href="https://i.sstatic.net/MjFPu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MjFPu.png" alt="enter image description here"></a></p>
  
  <p><em>Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma</em></p>
  
  <p>It's really ugly one. I guess there is some problem here.
  Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don't really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T06:40:48.467Z
Comments.Text: Did you have a look at chapter 2.5 of https://arxiv.org/abs/1707.09725 ? Please post the results of those analysis techniques, especially the learning curve and the confusion matrix.
Comments.UserDisplayName: Martin Thoma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T07:08:13.597Z
Comments.Text: I did read it, but I didn't apply it since I didn't understand all. I will try to apply it and come back here with the results.
Comments.UserDisplayName: Astariul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T07:22:20.603Z
Comments.Text: If something is unclear, let me know.
Comments.UserDisplayName: Martin Thoma
Tags: <neural-networks><machine-learning><deep-learning><overfitting>
Title: How to improve testing accuracy when training accuracy is high?
UserDisplayName: Astariul
ThreadText: # How to improve testing accuracy when training accuracy is high?
  
  (Asked by: username_0 on 2018-10-15)
  
  *Following-up my [question](https://ai.stackexchange.com/questions/8338/interpretation-of-a-good-overfitting-score) about my over-fitting network*
  
  My deep neural network is over-fitting :
  [![enter image description here](https://i.sstatic.net/6xKGF.png)](https://i.sstatic.net/6xKGF.png)
  
  I have tried several things :
  
  * Simplify the architecture
  * Apply more (and more !) Dropout
  * Data augmentation
  
  But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.
  
  I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :
  [![enter image description here](https://i.sstatic.net/nh4Bs.png)](https://i.sstatic.net/nh4Bs.png)
  
  My question is : **This accuracy of 70% is the best my model can reach ?**
  
  If yes :
  
  * Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?
  * My only option to improve the accuracy is then to change my model, right ?
  
  If no :
  
  * What are my options to improve this accuracy ?
  
  *I'v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can't exceed this limit, even though it seems easy to my network to reach it (short convergence time)*
  
  ## Edit
  
  Here is the Confusion matrix :
  
  [![enter image description here](https://i.sstatic.net/S2Hro.png)](https://i.sstatic.net/S2Hro.png)
  
  I don't think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : [SNLI Dataset](https://nlp.stanford.edu/projects/snli/)
  
  And here is the learning curve :
  
  [![enter image description here](https://i.sstatic.net/MjFPu.png)](https://i.sstatic.net/MjFPu.png)
  
  *Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma*
  
  It's really ugly one. I guess there is some problem here.
  Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don't really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.
  
  **Tags:** neural-networks, machine-learning, deep-learning, overfitting
  > **Comments:**
  > **username_1**: Did you have a look at chapter 2.5 of https://arxiv.org/abs/1707.09725 ? Please post the results of those analysis techniques, especially the learning curve and the confusion matrix.  
  > **username_0**: I did read it, but I didn't apply it since I didn't understand all. I will try to apply it and come back here with the results.  
  > **username_1**: If something is unclear, let me know.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-10-19. Score: 2 | Accepted answer
  
  I identified the origin of this overfitting..
  
  ## Origins
  
  I tried a lot of models, putting more and more dropout, simplifying as much as I could.
  
  No matter what I did, after a few epoch of good learning, invariably my loss function was going up. I tried simpler and simpler models, always the same overfitting behavior.
  What bugged me at that moment is that no matter what kind of model I used, how deep or how complex, always the accuracy was fine, stabilized at some nice level.
  
  So I tried the simplest model I could imagine : Input => Dense with 3 hidden units => Output. Finally I got random results, with a 33% accuracy !
  From here, I guilt again my network, layer by layer, to see which one was causing the overfitting.
  
  And it was the **Embedding layer**.
  
  Even with a simple network like Input => Embeddings => Dense with 3 hidden units => Output, the model was overfitting.
  
  ## How to solve it
  
  In Keras, simply instantiate the Embeddings layer with `trainable=False`. After doing this, no more overfit.
  
  In my opinion, this is quite counter-intuitive : I want my embeddings to evolve with the data I show to the network. But look like I can't...
  
  
  ---
  
  ### Answer by username_2 on 2018-10-15. Score: 0
  
  I think sometimes it can also help to examine your test and training sets.
  Fundamentally, your data was produced by an underlying process/system that has certain properties. The system can have many "states" and all the possible states form the state space. If you have really tried things like dropout and regularization, my guess would be that the test set is somehow different from your train set. It is possible that your training set only takes samples from one part of the state space (AKA, your samples might all be similar in the training set and the test set has different samples - imagine you are classifying humans and all of your training samples have a class label of 1 meaning all the training samples have humans in them -> and all your test samples have no humans in them Good luck with that!). Some questions to ask:
  
  1. Are you combining datasets from different sources? If so:
     If you have "n" sources of data, you need to make sure that your training set has many samples from each of the "n" sources of data and your test set has samples from each of the "n" sources.
  2. Are you shuffling your data enough and randomly putting samples in both the training and test sets? This relates to the human example I gave, make sure your training set has a little bit of everything (different combinations of inputs and/or outputs) and your testing set has a little bit of everything (different combinations of inputs and/or outputs).
  
  > **Comments:**
  > **username_0**: Thanks for your answer. However I don't think the problem is from the data : I am using the [SNLI Dataset](https://nlp.stanford.edu/projects/snli/), and only this one. This is a good and well balanced dataset.  The test and validation set are already defined by SNLI, and I always shuffle my data.  
  
  ---
  
  