title: Tensorflow accuracy from model.predict does not match final epoch val_accuracy of model.fit
body: <p>I am trying to match the accuracy of a model.predict call to the final val_accuracy of model.fit().  I am using tf dataset.</p>
  <pre><code>val_ds = tf.keras.utils.image_dataset_from_directory(
      'my_path',
      validation_split=0.2,
      subset=&quot;validation&quot;,
      seed=38,
      image_size=(SIZE,SIZE),
  )
  </code></pre>
  <p>The dataset setup for train_ds is similar.  I prefetch both...</p>
  <pre><code>train_ds = train_ds.prefetch(buffer_size=AUTOTUNE)
  val_ds = val_ds.prefetch(buffer_size=AUTOTUNE)
  </code></pre>
  <p>Than I get the labels for the val_ds so I can use them later</p>
  <pre><code>true_categories = tf.concat([y for x, y in val_ds], axis=0)
  </code></pre>
  <p>My model</p>
  <pre><code>inputs = tf.keras.Input(shape=(SIZE, SIZE, 3))
  # ... some other layers
  outputs = tf.keras.layers.Dense( len(CLASS_NAMES), activation = tf.keras.activations.softmax)(intermediate)
  model = tf.keras.Model(inputs, outputs)
  </code></pre>
  <p>Compiles fine</p>
  <pre><code>model.compile(
    optimizer = 'adam', 
    loss=tf.keras.losses.SparseCategoricalCrossentropy(), 
    metrics = ['accuracy'])
  </code></pre>
  <p>Seems to fit fine</p>
  <pre><code>history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=10, 
    class_weight=class_weights) #i do weight the classes due to imbalance
  </code></pre>
  <p>The last epoch output</p>
  <blockquote>
  <p>Epoch 10: val_accuracy did not improve from 0.92291
  176/176 [==============================] - 191s 1s/step - loss: 0.9876 - accuracy: 0.7318 - val_loss: 0.4650 - val_accuracy: 0.8580</p>
  </blockquote>
  <p>Now I want to verify the val_accuracy == 0.8580 when I run model.predict()</p>
  <pre><code>predictions = model.predict(val_ds, verbose=2 ) 
  flattened_predictions =  predictions.argmax(axis=1)
  accuracy = metrics.accuracy_score(true_categories, flattened_predictions)
  print (&quot;Accuracy = &quot;, accuracy)
  </code></pre>
  <blockquote>
  <p>Accuracy =  0.7980014275517487</p>
  </blockquote>
  <p>I would have expected that to equal the last val accuracy, which was 0.8580, but it is off.  My val_ds uses a seed so I should be getting the images in the same order when I shuffle, right?  Getting ground truth labels is a pain using datasets, but I think (???) my method is correct.</p>
  <p>I only have two classes and when I look at my predictions variable it looks like I am getting probabilities as I would expect, so I think I set up, compiled and fit my model correctly for sparse categorical cross entropy using softmax on my final layer output.</p>
  <pre><code>predictions[:3] #show the first 3 predictions, the values sum to 1.0 as expected
  </code></pre>
  <blockquote>
  <p>array([[0.42447385, 0.5755262 ],
  [0.2162129 , 0.7837871 ],
  [0.31917858, 0.6808214 ]], dtype=float32)</p>
  </blockquote>
  <p>What am I missing?</p>