Answers.Body: <p>It is a bit rare that the validation and test  accuracy exceed the training accuracy. One thing that could cause this is the selection of the validation and test data. Was the data for these two sets selected randomly or did you do the selection yourself? It is generally better to have these sets selected randomly from  the overall data  set. That way the probability distribution in these sets will closely match the distribution of the training set. Normally the training accuracy is higher (especially if you run enough epochs which I see you did) because there is always some degree of over fitting which reduces validation and test accuracy. The only other thing I can think of is the effect of Dropout layers. If you had dropout layers in your model and the drop out ratio was high that could cause this accuracy disparity. When the training accuracy is calculated it is done with drop being active. This can lower the training accuracy to some degree. However when evaluating validation accuracy and test accuracy drop out is NOT active so the model is actually more accurate. This increase in accuracy might be enough to overcome the decrease due to over fitting. Especially possible in this case since the accuracy differences appear to be quite small.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-06T11:45:36.457Z
Answers.Text: So, it *Is* due to the drop layers being disabled, as I suspected
Answers.UserDisplayName: Tobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-06T07:06:23.983Z
Answers.LastActivityDate: 2020-03-06T07:06:23.983Z
Answers.UserDisplayName: Gerry P
Body: <p><a href="https://i.sstatic.net/UFg5b.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UFg5b.png" alt="enter image description here"></a></p>
  
  <p>Is this due to my dropout layers being disabled during evaluation?</p>
  
  <p>I'm classifying the CIFAR-10 dataset with a CNN using the Keras library.</p>
  
  <p>There are 50000 samples in the training set; I'm using a 20% validation split for my training data (10000:40000). I have 10000 instances in the test set.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T14:06:21.630Z
Comments.Text: Please, provide the size of your datasets, batch size, the specific architecture (`model.summary()`) the loss function and which accuracy metric are you falling. The validation and test accuracies are only slightly greater than the training accuracy. This can happen (e.g. due to the fact that the validation or test examples come from a distribution where the model performs actually better), although that usually doesn't happen. How many examples do you use for validation and testing?
Comments.UserDisplayName: nbro
Tags: <neural-networks><tensorflow><keras><dropout>
Title: Why is my validation/test accuracy higher than my training accuracy
UserDisplayName: Tobi
ThreadText: # Why is my validation/test accuracy higher than my training accuracy
  
  (Asked by: username_0 on 2020-03-03)
  
  [![enter image description here](https://i.sstatic.net/UFg5b.png)](https://i.sstatic.net/UFg5b.png)
  
  Is this due to my dropout layers being disabled during evaluation?
  
  I'm classifying the CIFAR-10 dataset with a CNN using the Keras library.
  
  There are 50000 samples in the training set; I'm using a 20% validation split for my training data (10000:40000). I have 10000 instances in the test set.
  
  **Tags:** neural-networks, tensorflow, keras, dropout
  
  > **Comments:**
  > **username_1**: Please, provide the size of your datasets, batch size, the specific architecture (`model.summary()`) the loss function and which accuracy metric are you falling. The validation and test accuracies are only slightly greater than the training accuracy. This can happen (e.g. due to the fact that the validation or test examples come from a distribution where the model performs actually better), although that usually doesn't happen. How many examples do you use for validation and testing?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-03-06. Score: 0
  
  It is a bit rare that the validation and test accuracy exceed the training accuracy. One thing that could cause this is the selection of the validation and test data. Was the data for these two sets selected randomly or did you do the selection yourself? It is generally better to have these sets selected randomly from the overall data set. That way the probability distribution in these sets will closely match the distribution of the training set. Normally the training accuracy is higher (especially if you run enough epochs which I see you did) because there is always some degree of over fitting which reduces validation and test accuracy. The only other thing I can think of is the effect of Dropout layers. If you had dropout layers in your model and the drop out ratio was high that could cause this accuracy disparity. When the training accuracy is calculated it is done with drop being active. This can lower the training accuracy to some degree. However when evaluating validation accuracy and test accuracy drop out is NOT active so the model is actually more accurate. This increase in accuracy might be enough to overcome the decrease due to over fitting. Especially possible in this case since the accuracy differences appear to be quite small.
  
  > **Comments:**
  > **username_0**: So, it *Is* due to the drop layers being disabled, as I suspected  
  
  ---
  
  