Body: <p>My CNN tensorflow model reports 100% validation accuracy within 2 epochs. But it incorrectly predicts on single new images. (It is multiclass problem. I have 3 classes). How to resolve this? Can you please help me understand these epoch results?</p>
  <p>I have 1,000 images per class that are representative of my testing data. How can validation accuracy reach 1.00 in just the first epoch when I have a dataset of 3,000 images in total, equal amount per class? (I would expect this to start at around 33% percent -- 1/ 3 classes.)</p>
  <p>I understand overfitting can be a problem. I've added a dropout layer to try to solve this potential problem. From this question<a href="https://ai.stackexchange.com/questions/5318/what-to-do-if-cnn-cannot-overfit-a-training-set-on-adding-dropout/23425">What to do if CNN cannot overfit a training set on adding dropout?</a> I learned that a &quot;model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered.&quot; I don't believe my model is overfitting based on this description. (My model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy.)</p>
  <p>My model:</p>
  <pre><code>def model():
    model_input = tf.keras.layers.Input(shape=(h, w, 3)) 
    x = tf.keras.layers.Rescaling(rescale_factor)(model_input) 
    x = tf.keras.layers.Conv2D(16, 3, activation='relu',padding='same')(x)
    x = tf.keras.layers.Dropout(.5)(x)
    x = tf.keras.layers.MaxPooling2D()(x) 
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dense(128, activation='relu')(x)
    outputs = tf.keras.layers.Dense(num_classes, activation = 'softmax')(x)
  </code></pre>
  <p>Epoch results:</p>
  <pre><code>Epoch 1/10
  /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: UserWarning: &quot;`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?&quot;
    return dispatch_target(*args, **kwargs)
  27/27 [==============================] - 13s 124ms/step - loss: 1.0004 - accuracy: 0.5953 - val_loss: 0.5053 - val_accuracy: 0.8920
  Epoch 2/10
  27/27 [==============================] - 1s 46ms/step - loss: 0.1368 - accuracy: 0.9825 - val_loss: 0.0126 - val_accuracy: 1.0000
  Epoch 3/10
  27/27 [==============================] - 1s 42ms/step - loss: 0.0020 - accuracy: 1.0000 - val_loss: 5.9116e-04 - val_accuracy: 1.0000
  Epoch 4/10
  27/27 [==============================] - 1s 42ms/step - loss: 3.0633e-04 - accuracy: 1.0000 - val_loss: 3.5376e-04 - val_accuracy: 1.0000
  Epoch 5/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.7445e-04 - accuracy: 1.0000 - val_loss: 2.2319e-04 - val_accuracy: 1.0000
  Epoch 6/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.2910e-04 - accuracy: 1.0000 - val_loss: 1.8078e-04 - val_accuracy: 1.0000
  Epoch 7/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.0425e-04 - accuracy: 1.0000 - val_loss: 1.4247e-04 - val_accuracy: 1.0000
  Epoch 8/10
  27/27 [==============================] - 1s 42ms/step - loss: 8.6284e-05 - accuracy: 1.0000 - val_loss: 1.2057e-04 - val_accuracy: 1.0000
  Epoch 9/10
  27/27 [==============================] - 1s 42ms/step - loss: 7.0085e-05 - accuracy: 1.0000 - val_loss: 9.3485e-05 - val_accuracy: 1.0000
  Epoch 10/10
  27/27 [==============================] - 1s 42ms/step - loss: 5.4979e-05 - accuracy: 1.0000 - val_loss: 8.5952e-05 - val_accuracy: 1.0000
  </code></pre>
  <p>Model.fit and model.compile:</p>
  <pre><code>model = model()
  
  model = tf.keras.Model(model_input, outputs)
    
   model.compile(optimizer='adam',
                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
                metrics=['accuracy'])
    
  hist = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=10
  )
  </code></pre>
  <p>Code to predict new image:</p>
  <pre><code>def makePrediction(image):
    from IPython.display import display
    from PIL import Image
    from tensorflow.keras.preprocessing import image_dataset_from_directory 
    img = keras.preprocessing.image.load_img(
    image, target_size=(h, q)
    )
    img_array = keras.preprocessing.image.img_to_array(img)
    img_array = tf.expand_dims(img_array, 0) #Create a batch
   
    predicts = model.predict(img_array)
    p = class_names[np.argmax(predicts)]
    return p
  </code></pre>
  <p>Going to the &quot;data&quot; directory and using the folders to create a dataset. Each folder is a class label:</p>
  <pre><code>from keras.preprocessing import image
  directory_data = &quot;data&quot;
  tf.keras.utils.image_dataset_from_directory(
      directory_testData, labels='inferred', label_mode='int',
      class_names=None, color_mode='rgb', batch_size=32, image_size=(256,
      256), shuffle=True, seed=123, validation_split=0.2, subset=&quot;validation&quot;,
      interpolation='bilinear', follow_links=False,
      crop_to_aspect_ratio=False
  )
   
  tf.keras.utils.image_dataset_from_directory(directory_testData, labels='inferred')
  </code></pre>
  <p>Creating dataset and splitting it:</p>
  <p>Train_ds code: (Output: Found 1605 files belonging to 3 classes.
  Using 1284 files for training.)</p>
  <pre><code>train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    directory_data = &quot;data&quot;,
    validation_split=0.2,
    subset=&quot;training&quot;,
    seed=123,
    image_size=(h, w),
    batch_size=batch_size)
  </code></pre>
  <p>Val_ds code: (Output: Found 1605 files belonging to 3 classes.
  Using 321 files for validation.)</p>
  <pre><code>val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  directory_data = &quot;data&quot;,
    validation_split=0.2,
    subset=&quot;validation&quot;,
    seed=123,
    image_size=(h, w),
    batch_size=batch_size)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T03:38:51.900Z
Comments.Text: it can be data leakage. Check whether train and val images are same or not.
Comments.UserDisplayName: Prakash Dahal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T03:59:47.500Z
Comments.Text: @PrakashDahal Thank you for your help! I tried my best to check this. I don't think they are the same, and I believe I have split correctly. I would appreciate your help to ensure that data is not leaking, as I am new to ML :)
Comments.UserDisplayName: user4561
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T04:03:33.903Z
Comments.Text: I also checked my batch size to ensure it is reasonable. (If all the data is being passed to the model in 1 or 2 epochs then it may make sense for val accuracy to be high within first 2 epochs) My batch size is 32, so the model will not see all data in the first 2 epochs.
Comments.UserDisplayName: user4561
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T10:10:20.713Z
Comments.Text: The batch size is used to control number of items used per weight update step, it does not affect the size of the epoch - an epoch is by definition a loop through all the training data. In your case you would expect around 40 weight updates per epoch.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T14:02:21.743Z
Comments.Text: @NeilSlater Thank you for explaining! I would appreciate any additional pointers for things to consider as I try to understand why val acciuracy is 1.00 in the first epoch and why my model then incorrectly classifies all new images.
Comments.UserDisplayName: user4561
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-28T14:47:16.390Z
Comments.Text: Sorry I cannot see anything wrong. Looks like something basic has gone wrong though, and I suggest you check your assumptions about training and validation data by inspecting some of it mid-process.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-29T19:15:14.313Z
Comments.Text: @NeilSlater Than you for your advice. Any particular techniques you recommend me trying to inspect mid-process? It has been quite challenging to understand where I'm going wrong here :)
Comments.UserDisplayName: user4561
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-29T19:20:06.480Z
Comments.Text: Any debug tool with breakpoints, or simple logging, or you could try tensorboard https://www.tensorflow.org/tensorboard
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-29T22:08:38.450Z
Comments.Text: @NeilSlater Thank you. Can you please check my understanding of overfitting? I don't believe my model is overfitting because my model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy. Does training continue to occur (weights are updated) after validation accuracy is reported as 100%?
Comments.UserDisplayName: user4561
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-30T08:24:12.393Z
Comments.Text: I agree that the issue does not look like overfitting, although the result does match what you might see if you were overfitting with a data leak in the validation set, which is why Prakash mentioned it.
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><deep-learning><tensorflow><keras>
Title: Why is val accuracy 100% within 2 epochs and incorrectly predicting new images? (1,000 images per class when training)
UserDisplayName: user4561
ThreadText: # Why is val accuracy 100% within 2 epochs and incorrectly predicting new images? (1,000 images per class when training)
  
  (Asked by: username_0 on 2021-11-28)
  
  My CNN tensorflow model reports 100% validation accuracy within 2 epochs. But it incorrectly predicts on single new images. (It is multiclass problem. I have 3 classes). How to resolve this? Can you please help me understand these epoch results?
  
  I have 1,000 images per class that are representative of my testing data. How can validation accuracy reach 1.00 in just the first epoch when I have a dataset of 3,000 images in total, equal amount per class? (I would expect this to start at around 33% percent -- 1/ 3 classes.)
  
  I understand overfitting can be a problem. I've added a dropout layer to try to solve this potential problem. From this question[What to do if CNN cannot overfit a training set on adding dropout?](https://ai.stackexchange.com/questions/5318/what-to-do-if-cnn-cannot-overfit-a-training-set-on-adding-dropout/23425) I learned that a "model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered." I don't believe my model is overfitting based on this description. (My model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy.)
  
  My model:
  
  ```
  def model():
    model_input = tf.keras.layers.Input(shape=(h, w, 3)) 
    x = tf.keras.layers.Rescaling(rescale_factor)(model_input) 
    x = tf.keras.layers.Conv2D(16, 3, activation='relu',padding='same')(x)
    x = tf.keras.layers.Dropout(.5)(x)
    x = tf.keras.layers.MaxPooling2D()(x) 
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dense(128, activation='relu')(x)
    outputs = tf.keras.layers.Dense(num_classes, activation = 'softmax')(x)
  
  ```
  
  Epoch results:
  
  ```
  Epoch 1/10
  /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: UserWarning: "`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?"
    return dispatch_target(*args, **kwargs)
  27/27 [==============================] - 13s 124ms/step - loss: 1.0004 - accuracy: 0.5953 - val_loss: 0.5053 - val_accuracy: 0.8920
  Epoch 2/10
  27/27 [==============================] - 1s 46ms/step - loss: 0.1368 - accuracy: 0.9825 - val_loss: 0.0126 - val_accuracy: 1.0000
  Epoch 3/10
  27/27 [==============================] - 1s 42ms/step - loss: 0.0020 - accuracy: 1.0000 - val_loss: 5.9116e-04 - val_accuracy: 1.0000
  Epoch 4/10
  27/27 [==============================] - 1s 42ms/step - loss: 3.0633e-04 - accuracy: 1.0000 - val_loss: 3.5376e-04 - val_accuracy: 1.0000
  Epoch 5/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.7445e-04 - accuracy: 1.0000 - val_loss: 2.2319e-04 - val_accuracy: 1.0000
  Epoch 6/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.2910e-04 - accuracy: 1.0000 - val_loss: 1.8078e-04 - val_accuracy: 1.0000
  Epoch 7/10
  27/27 [==============================] - 1s 42ms/step - loss: 1.0425e-04 - accuracy: 1.0000 - val_loss: 1.4247e-04 - val_accuracy: 1.0000
  Epoch 8/10
  27/27 [==============================] - 1s 42ms/step - loss: 8.6284e-05 - accuracy: 1.0000 - val_loss: 1.2057e-04 - val_accuracy: 1.0000
  Epoch 9/10
  27/27 [==============================] - 1s 42ms/step - loss: 7.0085e-05 - accuracy: 1.0000 - val_loss: 9.3485e-05 - val_accuracy: 1.0000
  Epoch 10/10
  27/27 [==============================] - 1s 42ms/step - loss: 5.4979e-05 - accuracy: 1.0000 - val_loss: 8.5952e-05 - val_accuracy: 1.0000
  
  ```
  
  Model.fit and model.compile:
  
  ```
  model = model()
  
  model = tf.keras.Model(model_input, outputs)
    
   model.compile(optimizer='adam',
                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
                metrics=['accuracy'])
    
  hist = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=10
  )
  
  ```
  
  Code to predict new image:
  
  ```
  def makePrediction(image):
    from IPython.display import display
    from PIL import Image
    from tensorflow.keras.preprocessing import image_dataset_from_directory 
    img = keras.preprocessing.image.load_img(
    image, target_size=(h, q)
    )
    img_array = keras.preprocessing.image.img_to_array(img)
    img_array = tf.expand_dims(img_array, 0) #Create a batch
   
    predicts = model.predict(img_array)
    p = class_names[np.argmax(predicts)]
    return p
  
  ```
  
  Going to the "data" directory and using the folders to create a dataset. Each folder is a class label:
  
  ```
  from keras.preprocessing import image
  directory_data = "data"
  tf.keras.utils.image_dataset_from_directory(
      directory_testData, labels='inferred', label_mode='int',
      class_names=None, color_mode='rgb', batch_size=32, image_size=(256,
      256), shuffle=True, seed=123, validation_split=0.2, subset="validation",
      interpolation='bilinear', follow_links=False,
      crop_to_aspect_ratio=False
  )
   
  tf.keras.utils.image_dataset_from_directory(directory_testData, labels='inferred')
  
  ```
  
  Creating dataset and splitting it:
  
  Train\_ds code: (Output: Found 1605 files belonging to 3 classes.
  Using 1284 files for training.)
  
  ```
  train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    directory_data = "data",
    validation_split=0.2,
    subset="training",
    seed=123,
    image_size=(h, w),
    batch_size=batch_size)
  
  ```
  
  Val\_ds code: (Output: Found 1605 files belonging to 3 classes.
  Using 321 files for validation.)
  
  ```
  val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  directory_data = "data",
    validation_split=0.2,
    subset="validation",
    seed=123,
    image_size=(h, w),
    batch_size=batch_size)
  
  ```
  
  **Tags:** machine-learning, deep-learning, tensorflow, keras
  
  > **Comments:**
  > **username_1**: it can be data leakage. Check whether train and val images are same or not.  
  > **username_0**: @username_1 Thank you for your help! I tried my best to check this. I don't think they are the same, and I believe I have split correctly. I would appreciate your help to ensure that data is not leaking, as I am new to ML :)  
  > **username_0**: I also checked my batch size to ensure it is reasonable. (If all the data is being passed to the model in 1 or 2 epochs then it may make sense for val accuracy to be high within first 2 epochs) My batch size is 32, so the model will not see all data in the first 2 epochs.  
  > **username_2**: The batch size is used to control number of items used per weight update step, it does not affect the size of the epoch - an epoch is by definition a loop through all the training data. In your case you would expect around 40 weight updates per epoch.  
  > **username_0**: @username_2 Thank you for explaining! I would appreciate any additional pointers for things to consider as I try to understand why val acciuracy is 1.00 in the first epoch and why my model then incorrectly classifies all new images.  
  > **username_2**: Sorry I cannot see anything wrong. Looks like something basic has gone wrong though, and I suggest you check your assumptions about training and validation data by inspecting some of it mid-process.  
  > **username_0**: @username_2 Than you for your advice. Any particular techniques you recommend me trying to inspect mid-process? It has been quite challenging to understand where I'm going wrong here :)  
  > **username_2**: Any debug tool with breakpoints, or simple logging, or you could try tensorboard https://www.tensorflow.org/tensorboard  
  > **username_0**: @username_2 Thank you. Can you please check my understanding of overfitting? I don't believe my model is overfitting because my model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy. Does training continue to occur (weights are updated) after validation accuracy is reported as 100%?  
  > **username_2**: I agree that the issue does not look like overfitting, although the result does match what you might see if you were overfitting with a data leak in the validation set, which is why username_1 mentioned it.  
  
  ---
  
  ## 0 answers
  
  