Answers.Body: <p>Your test set images are more clean than your training set images, because you applied more types of noise to the training set images, so the classifier performs better on the cleaner images in the test set.</p>
  <p>I don't see why this is surprising.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T21:11:35.443Z
Answers.Text: Yeah , sorry if I forgot to specify that: if I apply the same trasformation to images (or no trasformation at all) I got the same result, a bigger accuracy in the test set than on the training set, I'm going to make an edit to the question
Answers.UserDisplayName: Loris Simonetti
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T21:02:10.287Z
Answers.LastActivityDate: 2023-03-10T21:02:10.287Z
Answers.UserDisplayName: Criticize SE actions means ban
Answers.Body: <p>Getting higher accuracy in test set than in training set means that your test set is easier to work on than training set. Your graph shows results for only one epoch. In the 1st epoch, your train accuracy/loss will include the learning phase when loss was high whereas by the time evaluation happens, the model has already learned. Is the behaviour same in subsequent epochs?</p>
  <p>As you mentioned that you get similar results even with same or no augmentation, ignoring the learning phase issue, it is possible that your training set has more samples for which the network doesn't perform well, dragging the performance down. You can test this using something like k-fold cross validation which tests your network on multiple train-test pairs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T18:14:17.353Z
Answers.LastActivityDate: 2023-03-11T18:14:17.353Z
Answers.UserDisplayName: SajanGohil
Body: <p>Hi I'm trying to train an ANN model to classify images containing these characters: 0,1,2,3,4,T,X,S
  eg.
  <a href="https://i.sstatic.net/KnDvB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KnDvB.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/oAryk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oAryk.png" alt="enter image description here" /></a>
  etc...</p>
  <p>so something like the classification of records of the MNIST dataset but using my generated images (not handwritten).</p>
  <p>I generate ~10000 records that have to be divided in the training set and in the test set:</p>
  <pre><code>n = 10000
  X_train, y_train = generate_typed(typed_alphabet_resized.astype('uint8'), int(n*0.8))
  X_test, y_test = generate_typed_test(typed_alphabet_resized.astype('uint8'), int(n*0.3))
  </code></pre>
  <p><code>generate_typed</code> and <code>generate_typed_test</code> are pretty much the same function, the sole difference is the augmentation applied to images:
  for generate_typed_test the only augmentation applied is scale(1,3) while for generate_typed:</p>
  <pre><code>augmentation_seq = iaa.Sequential([
      iaa.Sometimes(0.3, iaa.GaussianBlur(sigma=(0, .1))), 
      iaa.Sometimes(0.3, iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.2)), 
      iaa.Affine(scale=(1, 3)), 
  ], random_order=True)
  </code></pre>
  <p>I did that to make the model more generalized as possibile.
  But really the kind of images the model is supposed to work on in the real world might just (sometimes) be zoomed in or be slightly less centered.
  However even if I use these two different functions, the images I got are very very similar.</p>
  <p>Now, the problem is: I got a bigger accuracy on the test set than on the training set, and I really can't understand why and how to fix it.
  It seems like that the accuracy of my model doesn't matter, if my model has around 80% of accuracy or 95% of accuracy, I still got an 0.9997 on test set (bigger than the accuracy during the training).
  Note: Even if I apply the same type of trasformations on the train and test set, or if I apply no trasformation to both sets nothing changes.</p>
  <p>I'm not saying the model doesn't work, but it's really wrong quite often.
  I don't know if this means overfitting or not, but I don't think so because loss &amp; accuracy doesn't intersect each other:</p>
  <p><a href="https://i.sstatic.net/w82jt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/w82jt.png" alt="enter image description here" /></a></p>
  <p>Model:</p>
  <pre><code>model3 = Sequential([
    Flatten(input_shape=(28,28,1)), # &quot;appiattisco&quot; l'immagine
    Dense(20, activation='relu'), # creo uno strato denso la cui funzione di attivazione è relu
    Dropout(0.5), # Spengo a caso dei neuroni (per evitare overfitting)
    Dense(num_classes, activation='softmax') # strato denso di 10 neuroni con funzione di attivazione softmax (perchè è un classificatore)
  ])
  
  model3.compile(
      loss = 'sparse_categorical_crossentropy',
      metrics=['accuracy'],
      optimizer=Adam(lr=0.01)
  )
  
  
  X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.1, random_state=42)
  model3.fit(X_train, y_train, epochs=5, batch_size = 5, validation_data=(X_val, y_val), callbacks=[early_stop])
  
  _________________________________________________________________
   Layer (type)                Output Shape              Param #   
  =================================================================
   flatten_34 (Flatten)        (None, 784)               0         
                                                                   
   dense_71 (Dense)            (None, 20)                15700     
                                                                   
   dropout_37 (Dropout)        (None, 20)                0         
                                                                   
   dense_72 (Dense)            (None, 105)               2205      
                                                                   
  =================================================================
  Total params: 17,905
  Trainable params: 17,905
  Non-trainable params: 0
  </code></pre>
  
Tags: <neural-networks><deep-learning><classification><image-recognition><image-generation>
Title: Higher accuracy in the test set than in the training set
UserDisplayName: Loris Simonetti
ThreadText: # Higher accuracy in the test set than in the training set
  
  (Asked by: username_0 on 2023-03-10)
  
  Hi I'm trying to train an ANN model to classify images containing these characters: 0,1,2,3,4,T,X,S
  eg.
  [![enter image description here](https://i.sstatic.net/KnDvB.png)](https://i.sstatic.net/KnDvB.png)
  [![enter image description here](https://i.sstatic.net/oAryk.png)](https://i.sstatic.net/oAryk.png)
  etc...
  
  so something like the classification of records of the MNIST dataset but using my generated images (not handwritten).
  
  I generate ~10000 records that have to be divided in the training set and in the test set:
  
  ```
  n = 10000
  X_train, y_train = generate_typed(typed_alphabet_resized.astype('uint8'), int(n*0.8))
  X_test, y_test = generate_typed_test(typed_alphabet_resized.astype('uint8'), int(n*0.3))
  
  ```
  
  `generate_typed` and `generate_typed_test` are pretty much the same function, the sole difference is the augmentation applied to images:
  for generate\_typed\_test the only augmentation applied is scale(1,3) while for generate\_typed:
  
  ```
  augmentation_seq = iaa.Sequential([
      iaa.Sometimes(0.3, iaa.GaussianBlur(sigma=(0, .1))), 
      iaa.Sometimes(0.3, iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.2)), 
      iaa.Affine(scale=(1, 3)), 
  ], random_order=True)
  
  ```
  
  I did that to make the model more generalized as possibile.
  But really the kind of images the model is supposed to work on in the real world might just (sometimes) be zoomed in or be slightly less centered.
  However even if I use these two different functions, the images I got are very very similar.
  
  Now, the problem is: I got a bigger accuracy on the test set than on the training set, and I really can't understand why and how to fix it.
  It seems like that the accuracy of my model doesn't matter, if my model has around 80% of accuracy or 95% of accuracy, I still got an 0.9997 on test set (bigger than the accuracy during the training).
  Note: Even if I apply the same type of trasformations on the train and test set, or if I apply no trasformation to both sets nothing changes.
  
  I'm not saying the model doesn't work, but it's really wrong quite often.
  I don't know if this means overfitting or not, but I don't think so because loss & accuracy doesn't intersect each other:
  
  [![enter image description here](https://i.sstatic.net/w82jt.png)](https://i.sstatic.net/w82jt.png)
  
  Model:
  
  ```
  model3 = Sequential([
    Flatten(input_shape=(28,28,1)), # "appiattisco" l'immagine
    Dense(20, activation='relu'), # creo uno strato denso la cui funzione di attivazione è relu
    Dropout(0.5), # Spengo a caso dei neuroni (per evitare overfitting)
    Dense(num_classes, activation='softmax') # strato denso di 10 neuroni con funzione di attivazione softmax (perchè è un classificatore)
  ])
  
  model3.compile(
      loss = 'sparse_categorical_crossentropy',
      metrics=['accuracy'],
      optimizer=Adam(lr=0.01)
  )
  
  X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.1, random_state=42)
  model3.fit(X_train, y_train, epochs=5, batch_size = 5, validation_data=(X_val, y_val), callbacks=[early_stop])
  
  _________________________________________________________________
   Layer (type)                Output Shape              Param #   
  =================================================================
   flatten_34 (Flatten)        (None, 784)               0         
                                                                   
   dense_71 (Dense)            (None, 20)                15700     
                                                                   
   dropout_37 (Dropout)        (None, 20)                0         
                                                                   
   dense_72 (Dense)            (None, 105)               2205      
                                                                   
  =================================================================
  Total params: 17,905
  Trainable params: 17,905
  Non-trainable params: 0
  
  ```
  
  **Tags:** neural-networks, deep-learning, classification, image-recognition, image-generation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-03-10. Score: 0
  
  Your test set images are more clean than your training set images, because you applied more types of noise to the training set images, so the classifier performs better on the cleaner images in the test set.
  
  I don't see why this is surprising.
  
  > **Comments:**
  > **username_0**: Yeah , sorry if I forgot to specify that: if I apply the same trasformation to images (or no trasformation at all) I got the same result, a bigger accuracy in the test set than on the training set, I'm going to make an edit to the question  
  
  ---
  
  ### Answer by username_2 on 2023-03-11. Score: 0
  
  Getting higher accuracy in test set than in training set means that your test set is easier to work on than training set. Your graph shows results for only one epoch. In the 1st epoch, your train accuracy/loss will include the learning phase when loss was high whereas by the time evaluation happens, the model has already learned. Is the behaviour same in subsequent epochs?
  
  As you mentioned that you get similar results even with same or no augmentation, ignoring the learning phase issue, it is possible that your training set has more samples for which the network doesn't perform well, dragging the performance down. You can test this using something like k-fold cross validation which tests your network on multiple train-test pairs.
  
  
  ---
  
  