Answers.Body: <p>I will just add to all the good answers already here. </p>
  
  <p>Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly). </p>
  
  <p>Other reasons could be: </p>
  
  <ul>
  <li>High dropout rate or excessive data augmentation could be one of the reason. This can cause the training accuracy to appear low whist in reality the model is in fact learning. How so, you ask? Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. Recall that every score you are seeing on the training set is being calculated on a "different" set of weights, ie a different network. So the accuracy you are seeing is the accuracy of all those networks your network tried before it settled on the one you are now testing on the test set. The reasoning here being that the other networks were just very "bad" compared to the final one that you are now using to test on the test set. If this is your case then, you are just witnessing neural network dropout and data augmentation doing exactly what they are meant to do, i.e avoid dropout. </li>
  <li>Another possible reason for this could be that your test set is different and simpler in comparison to the training set. That is to say: your data split could be such that you have a simpler test set than a train set and hence your network seems to be doing far much better on the test set than it did on the training set. If this is the case then good job, you have build yourself a very good model, no need to stress, but be careful because with such "bad" data splitting,  future you might not be as lucky :).</li>
  <li>Make sure you are doing the correct pre-processing.</li>
  <li>Your test set might be very small as such, the high accuracy you are seeing on it is not indicative of the real accuracy you would see if you had a larger test set. If this is the case then, you may want to collect more test data and test your model again. </li>
  <li>Another reason could be than your training set is small and the high training error you are seeing there is not "real". This bring up the question of how big your training set it? </li>
  <li>To quote <a href="https://keras.io/getting-started/faq/#why-is-the-training-loss-much-higher-than-the-testing-loss" rel="nofollow noreferrer">the Keras documentation</a>:
  
  <blockquote>
    <p>the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.</p>
  </blockquote></li>
  <li>Lastly you may find this <a href="https://github.com/fchollet/keras/issues/1761" rel="nofollow noreferrer">github postenter link description here</a> useful.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-01T13:30:12.533Z
Answers.LastActivityDate: 2017-11-01T14:03:50.223Z
Answers.LastEditDate: 2017-11-01T14:03:50.223Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>Also, it really depends how uniform your test data is distributed as well as its size. If you have relevantly so small test dataset compared to training one, it may be an expected output. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-30T03:08:04.113Z
Answers.LastActivityDate: 2017-10-30T03:08:04.113Z
Answers.UserDisplayName: Deniz Beker
Answers.Body: <p>Beyond dropout (as @demento already explained) if you are training with data augmentation this is an expected behavior.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-29T20:16:03.163Z
Answers.LastActivityDate: 2017-10-29T20:16:03.163Z
Answers.UserDisplayName: xboard
Answers.Body: <p>You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher.</p>
  
  <p>That's normal behavior if you work with dropout.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-29T12:08:51.977Z
Answers.LastActivityDate: 2017-10-29T12:08:51.977Z
Answers.UserDisplayName: Demento
Body: <p>I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay.</p>
  <p>My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy is better than test accuracy.</p>
  <p>So I wonder how this is happening.</p>
  <p><a href="https://i.sstatic.net/MgVBt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MgVBt.png" alt="my grph" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-24T00:30:13.530Z
Comments.Text: This is an old question, so you probably don't remember what I want you to clarify. Basically, in the title, you use "error", but in the body of the question you use "accuracy". From the plot, it seems that you were plotting the error, so you may want to change "accuracy" to "error" in the body of the post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-29T16:43:41.547Z
Comments.Text: well, ...this is not a bad thing, it means you are not overfiting. Out of curiosity, what is your dropout rate?
Comments.UserDisplayName: Tshilidzi Mudau
Tags: <convolutional-neural-networks><training><accuracy><dropout><testing>
Title: Why is my test error lower than the training error?
UserDisplayName: 이희준
ThreadText: # Why is my test error lower than the training error?
  
  (Asked by: username_0 on 2017-10-29)
  
  I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay.
  
  My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy is better than test accuracy.
  
  So I wonder how this is happening.
  
  [![my grph](https://i.sstatic.net/MgVBt.png)](https://i.sstatic.net/MgVBt.png)
  
  **Tags:** convolutional-neural-networks, training, accuracy, dropout, testing
  > **Comments:**
  > **username_2**: well, ...this is not a bad thing, it means you are not overfiting. Out of curiosity, what is your dropout rate?  
  > **username_1**: This is an old question, so you probably don't remember what I want you to clarify. Basically, in the title, you use "error", but in the body of the question you use "accuracy". From the plot, it seems that you were plotting the error, so you may want to change "accuracy" to "error" in the body of the post.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2017-10-29. Score: 5 | Accepted answer
  
  You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher.
  
  That's normal behavior if you work with dropout.
  
  
  ---
  
  ### Answer by username_2 on 2017-11-01. Score: 4
  
  I will just add to all the good answers already here.
  
  Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly).
  
  Other reasons could be:
  
  * High dropout rate or excessive data augmentation could be one of the reason. This can cause the training accuracy to appear low whist in reality the model is in fact learning. How so, you ask? Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. Recall that every score you are seeing on the training set is being calculated on a "different" set of weights, ie a different network. So the accuracy you are seeing is the accuracy of all those networks your network tried before it settled on the one you are now testing on the test set. The reasoning here being that the other networks were just very "bad" compared to the final one that you are now using to test on the test set. If this is your case then, you are just witnessing neural network dropout and data augmentation doing exactly what they are meant to do, i.e avoid dropout.
  * Another possible reason for this could be that your test set is different and simpler in comparison to the training set. That is to say: your data split could be such that you have a simpler test set than a train set and hence your network seems to be doing far much better on the test set than it did on the training set. If this is the case then good job, you have build yourself a very good model, no need to stress, but be careful because with such "bad" data splitting, future you might not be as lucky :).
  * Make sure you are doing the correct pre-processing.
  * Your test set might be very small as such, the high accuracy you are seeing on it is not indicative of the real accuracy you would see if you had a larger test set. If this is the case then, you may want to collect more test data and test your model again.
  * Another reason could be than your training set is small and the high training error you are seeing there is not "real". This bring up the question of how big your training set it?
  * To quote [the Keras documentation](https://keras.io/getting-started/faq/#why-is-the-training-loss-much-higher-than-the-testing-loss):
    > the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.
  * Lastly you may find this [github postenter link description here](https://github.com/fchollet/keras/issues/1761) useful.
  
  
  ---
  
  ### Answer by username_3 on 2017-10-30. Score: 0
  
  Also, it really depends how uniform your test data is distributed as well as its size. If you have relevantly so small test dataset compared to training one, it may be an expected output.
  
  
  ---
  
  ### Answer by username_4 on 2017-10-29. Score: 0
  
  Beyond dropout (as @demento already explained) if you are training with data augmentation this is an expected behavior.
  
  
  ---
  
  