Answers.Body: <p>After spending many hours, I found the issue was due to the shuffle function. I was using the below function to shuffle, batch and prefetch the dataset.</p>
  <pre><code>def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                              shuffle_buffer_size=None,
                              drop_remainder=False, 
                              interleave_num_pcall=None):
  
      if shuffle_buffer_size is None:
          raise ValueError(&quot;shuffle_buffer_size can't be None&quot;)
      def shuffle_fn(ds):
           return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
      dataset = dataset.apply(shuffle_fn)
      dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
      dataset = dataset.prefetch(buffer_size=prefetch_size)
      return dataset
  </code></pre>
  <p>Part of the function that causes the problem</p>
  <pre><code>def shuffle_fn(ds):
      return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
  dataset = dataset.apply(shuffle_fn)
  </code></pre>
  <p>I removed the shuffle part and metrics are back as per the expectation.
  Function after removing the shuffle part</p>
  <pre><code>def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                             drop_remainder=False, 
                            interleave_num_pcall=None):
      dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
      dataset = dataset.prefetch(buffer_size=prefetch_size)
      return dataset
  </code></pre>
  <p>Results after removing the shuffle part
  <a href="https://i.sstatic.net/E5hmU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E5hmU.png" alt="enter image description here" /></a></p>
  <p>I am still not able to understand why shuffling causes this error. Shuffling was the best practice to follow before training your data. Although, I have already shuffled training data during data read time.</p>
  <p>I have also posted the same issue at <a href="https://stackoverflow.com/questions/73005511/metrics-using-batches-v-s-metrics-using-full-dataset/73013819#73013819">SO</a> thinking that it might be related to some coding error.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-17T17:18:15.203Z
Answers.LastActivityDate: 2022-07-17T17:18:15.203Z
Answers.UserDisplayName: learner
Body: <p>I am using training an image classification model using the pre-trained mobile network. During training, I am seeing very high values (more than 70%) for Accuracy, Precision, Recall, and F1-score on both the training dataset and validation dataset.
  <a href="https://i.sstatic.net/yKwYZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yKwYZ.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/GxA4n.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GxA4n.png" alt="enter image description here" /></a>
  For me, this is an indication that my model is learning fine.</p>
  <p><strong>But when I checked these metrics on an Unbatched training and Unbatched Validation these metrics are very low. Even are then on 1%.</strong></p>
  <p><em>Unbatched dataset means I am not taking calculating these metrics over batches and taking the average of metrics to calculate the final metrics which is what Tensorflow/Keras does during model training.</em></p>
  <p><a href="https://i.sstatic.net/GYSAv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GYSAv.png" alt="enter image description here" /></a></p>
  <p>I am unable to find out what is causing this Behaviour. Please help me understand what is causing this difference and how to ensure that results are consistent on both, i.e. a minor difference is acceptable.</p>
  <p>Code that I used for evaluating metrics</p>
  <pre><code>def test_model(model, data, CLASSES, label_one_hot=True, average=&quot;micro&quot;, 
                  threshold_analysis=False, thres_analysis_start_point=0.0, 
                  thres_analysis_end_point=0.95, thres_step=0.05, classwise_analysis=False,
                  produce_confusion_matrix=False):
      images_ds = data.map(lambda image, label: image)
      labels_ds = data.map(lambda image, label: label).unbatch()
      NUM_VALIDATION_IMAGES = count_data_items(tf_records_filenames=data)
      cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch
      if label_one_hot is True:
          cm_correct_labels = np.argmax(cm_correct_labels, axis=-1)
      cm_probabilities = model.predict(images_ds)
      cm_predictions = np.argmax(cm_probabilities, axis=-1)
      
      warnings.filterwarnings('ignore')
  
      overall_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      overall_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      overall_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      # cmat = (cmat.T / cmat.sum(axis=1)).T # normalized
      # print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))
      overall_test_results = {'overall_f1_score': overall_score, 'overall_precision':overall_precision, 'overall_recall':overall_recall}
  
      if classwise_analysis is True:
          
          label_index_dict = get_index_label_from_tf_record(dataset=data)
          label_index_dict = {k:v for k, v in sorted(list(label_index_dict.items()))}
          label_index_df = pd.DataFrame(label_index_dict, index=[0]).T.reset_index().rename(columns={'index':'class_ind', 0:'class_names'})
          # Class wise precision, recall and f1_score
          classwise_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
          classwise_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
          classwise_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
  
          ind_class_count_df = class_ind_counter_from_tfrecord(data)
          ind_class_count_df = ind_class_count_df.merge(label_index_df, how='left', left_on='class_names', right_on='class_names')
  
          classwise_test_results = {'classwise_f1_score':classwise_score, 'classwise_precision':classwise_precision,
                          'classwise_recall':classwise_recall, 'class_names':CLASSES}
          classwise_test_results_df = pd.DataFrame(classwise_test_results)
      
          if produce_confusion_matrix is True:
              cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
              return overall_test_results, classwise_test_results, cmat
          return overall_test_results, classwise_test_results
          
      if produce_confusion_matrix is True:
          cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
          return overall_test_results, cmat
      warnings.filterwarnings('always')
      return overall_test_results
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-15T15:44:40.073Z
Comments.Text: are you sure this is not a coding issue? Have you tried feeding dummy predictions and labels all equal to check that in both cases you indeed get 100%? When this happen to me all the time I find some silent bug in my evaluation functions, like a wrong indent that cause me to write to tensorboard only a fraction of the predictions (which are then divided by the total amount of images which could explain the super low scores)
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-15T18:40:08.400Z
Comments.Text: @EdoardoGuerriero I have checked this code looks fine to me. Adding code for your reference. Even I used this code with some previous iteration and that time it was working fine
Comments.UserDisplayName: learner
Tags: <computer-vision><tensorflow><metric><confusion-matrix><mobile-net-v2>
Title: Metrics using batches v/s metrics using full dataset
UserDisplayName: learner
ThreadText: # Metrics using batches v/s metrics using full dataset
  
  (Asked by: username_0 on 2022-07-15)
  
  I am using training an image classification model using the pre-trained mobile network. During training, I am seeing very high values (more than 70%) for Accuracy, Precision, Recall, and F1-score on both the training dataset and validation dataset.
  [![enter image description here](https://i.sstatic.net/yKwYZ.png)](https://i.sstatic.net/yKwYZ.png)
  [![enter image description here](https://i.sstatic.net/GxA4n.png)](https://i.sstatic.net/GxA4n.png)
  For me, this is an indication that my model is learning fine.
  
  **But when I checked these metrics on an Unbatched training and Unbatched Validation these metrics are very low. Even are then on 1%.**
  
  *Unbatched dataset means I am not taking calculating these metrics over batches and taking the average of metrics to calculate the final metrics which is what Tensorflow/Keras does during model training.*
  
  [![enter image description here](https://i.sstatic.net/GYSAv.png)](https://i.sstatic.net/GYSAv.png)
  
  I am unable to find out what is causing this Behaviour. Please help me understand what is causing this difference and how to ensure that results are consistent on both, i.e. a minor difference is acceptable.
  
  Code that I used for evaluating metrics
  
  ```
  def test_model(model, data, CLASSES, label_one_hot=True, average="micro", 
                  threshold_analysis=False, thres_analysis_start_point=0.0, 
                  thres_analysis_end_point=0.95, thres_step=0.05, classwise_analysis=False,
                  produce_confusion_matrix=False):
      images_ds = data.map(lambda image, label: image)
      labels_ds = data.map(lambda image, label: label).unbatch()
      NUM_VALIDATION_IMAGES = count_data_items(tf_records_filenames=data)
      cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch
      if label_one_hot is True:
          cm_correct_labels = np.argmax(cm_correct_labels, axis=-1)
      cm_probabilities = model.predict(images_ds)
      cm_predictions = np.argmax(cm_probabilities, axis=-1)
      
      warnings.filterwarnings('ignore')
  
      overall_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      overall_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      overall_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
      # cmat = (cmat.T / cmat.sum(axis=1)).T # normalized
      # print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))
      overall_test_results = {'overall_f1_score': overall_score, 'overall_precision':overall_precision, 'overall_recall':overall_recall}
  
      if classwise_analysis is True:
          
          label_index_dict = get_index_label_from_tf_record(dataset=data)
          label_index_dict = {k:v for k, v in sorted(list(label_index_dict.items()))}
          label_index_df = pd.DataFrame(label_index_dict, index=[0]).T.reset_index().rename(columns={'index':'class_ind', 0:'class_names'})
          # Class wise precision, recall and f1_score
          classwise_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
          classwise_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
          classwise_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
  
          ind_class_count_df = class_ind_counter_from_tfrecord(data)
          ind_class_count_df = ind_class_count_df.merge(label_index_df, how='left', left_on='class_names', right_on='class_names')
  
          classwise_test_results = {'classwise_f1_score':classwise_score, 'classwise_precision':classwise_precision,
                          'classwise_recall':classwise_recall, 'class_names':CLASSES}
          classwise_test_results_df = pd.DataFrame(classwise_test_results)
      
          if produce_confusion_matrix is True:
              cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
              return overall_test_results, classwise_test_results, cmat
          return overall_test_results, classwise_test_results
          
      if produce_confusion_matrix is True:
          cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
          return overall_test_results, cmat
      warnings.filterwarnings('always')
      return overall_test_results
  
  ```
  
  **Tags:** computer-vision, tensorflow, metric, confusion-matrix, mobile-net-v2
  
  > **Comments:**
  > **username_1**: are you sure this is not a coding issue? Have you tried feeding dummy predictions and labels all equal to check that in both cases you indeed get 100%? When this happen to me all the time I find some silent bug in my evaluation functions, like a wrong indent that cause me to write to tensorboard only a fraction of the predictions (which are then divided by the total amount of images which could explain the super low scores)  
  > **username_0**: @username_1 I have checked this code looks fine to me. Adding code for your reference. Even I used this code with some previous iteration and that time it was working fine  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-07-17. Score: 0 | Accepted answer
  
  After spending many hours, I found the issue was due to the shuffle function. I was using the below function to shuffle, batch and prefetch the dataset.
  
  ```
  def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                              shuffle_buffer_size=None,
                              drop_remainder=False, 
                              interleave_num_pcall=None):
  
      if shuffle_buffer_size is None:
          raise ValueError("shuffle_buffer_size can't be None")
      def shuffle_fn(ds):
           return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
      dataset = dataset.apply(shuffle_fn)
      dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
      dataset = dataset.prefetch(buffer_size=prefetch_size)
      return dataset
  
  ```
  
  Part of the function that causes the problem
  
  ```
  def shuffle_fn(ds):
      return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
  dataset = dataset.apply(shuffle_fn)
  
  ```
  
  I removed the shuffle part and metrics are back as per the expectation.
  Function after removing the shuffle part
  
  ```
  def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                             drop_remainder=False, 
                            interleave_num_pcall=None):
      dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
      dataset = dataset.prefetch(buffer_size=prefetch_size)
      return dataset
  
  ```
  
  Results after removing the shuffle part
  [![enter image description here](https://i.sstatic.net/E5hmU.png)](https://i.sstatic.net/E5hmU.png)
  
  I am still not able to understand why shuffling causes this error. Shuffling was the best practice to follow before training your data. Although, I have already shuffled training data during data read time.
  
  I have also posted the same issue at [SO](https://stackoverflow.com/questions/73005511/metrics-using-batches-v-s-metrics-using-full-dataset/73013819#73013819) thinking that it might be related to some coding error.
  
  
  ---
  
  