<p>After spending many hours, I found the issue was due to the shuffle function. I was using the below function to shuffle, batch and prefetch the dataset.</p>
<pre><code>def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                            shuffle_buffer_size=None,
                            drop_remainder=False, 
                            interleave_num_pcall=None):

    if shuffle_buffer_size is None:
        raise ValueError(&quot;shuffle_buffer_size can't be None&quot;)
    def shuffle_fn(ds):
         return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
    dataset = dataset.apply(shuffle_fn)
    dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
    dataset = dataset.prefetch(buffer_size=prefetch_size)
    return dataset
</code></pre>
<p>Part of the function that causes the problem</p>
<pre><code>def shuffle_fn(ds):
    return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
dataset = dataset.apply(shuffle_fn)
</code></pre>
<p>I removed the shuffle part and metrics are back as per the expectation.
Function after removing the shuffle part</p>
<pre><code>def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                           drop_remainder=False, 
                          interleave_num_pcall=None):
    dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
    dataset = dataset.prefetch(buffer_size=prefetch_size)
    return dataset
</code></pre>
<p>Results after removing the shuffle part
<a href="https://i.sstatic.net/E5hmU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E5hmU.png" alt="enter image description here" /></a></p>
<p>I am still not able to understand why shuffling causes this error. Shuffling was the best practice to follow before training your data. Although, I have already shuffled training data during data read time.</p>
<p>I have also posted the same issue at <a href="https://stackoverflow.com/questions/73005511/metrics-using-batches-v-s-metrics-using-full-dataset/73013819#73013819">SO</a> thinking that it might be related to some coding error.</p>

CC BY-SA 4.0
2022-07-17T17:18:15.203Z
2022-07-17T17:18:15.203Z
learner
<p>I am using training an image classification model using the pre-trained mobile network. During training, I am seeing very high values (more than 70%) for Accuracy, Precision, Recall, and F1-score on both the training dataset and validation dataset.
<a href="https://i.sstatic.net/yKwYZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yKwYZ.png" alt="enter image description here" /></a>
<a href="https://i.sstatic.net/GxA4n.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GxA4n.png" alt="enter image description here" /></a>
For me, this is an indication that my model is learning fine.</p>
<p><strong>But when I checked these metrics on an Unbatched training and Unbatched Validation these metrics are very low. Even are then on 1%.</strong></p>
<p><em>Unbatched dataset means I am not taking calculating these metrics over batches and taking the average of metrics to calculate the final metrics which is what Tensorflow/Keras does during model training.</em></p>
<p><a href="https://i.sstatic.net/GYSAv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GYSAv.png" alt="enter image description here" /></a></p>
<p>I am unable to find out what is causing this Behaviour. Please help me understand what is causing this difference and how to ensure that results are consistent on both, i.e. a minor difference is acceptable.</p>
<p>Code that I used for evaluating metrics</p>
<pre><code>def test_model(model, data, CLASSES, label_one_hot=True, average=&quot;micro&quot;, 
                threshold_analysis=False, thres_analysis_start_point=0.0, 
                thres_analysis_end_point=0.95, thres_step=0.05, classwise_analysis=False,
                produce_confusion_matrix=False):
    images_ds = data.map(lambda image, label: image)
    labels_ds = data.map(lambda image, label: label).unbatch()
    NUM_VALIDATION_IMAGES = count_data_items(tf_records_filenames=data)
    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch
    if label_one_hot is True:
        cm_correct_labels = np.argmax(cm_correct_labels, axis=-1)
    cm_probabilities = model.predict(images_ds)
    cm_predictions = np.argmax(cm_probabilities, axis=-1)
    
    warnings.filterwarnings('ignore')

    overall_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    overall_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    overall_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    # cmat = (cmat.T / cmat.sum(axis=1)).T # normalized
    # print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))
    overall_test_results = {'overall_f1_score': overall_score, 'overall_precision':overall_precision, 'overall_recall':overall_recall}

    if classwise_analysis is True:
        
        label_index_dict = get_index_label_from_tf_record(dataset=data)
        label_index_dict = {k:v for k, v in sorted(list(label_index_dict.items()))}
        label_index_df = pd.DataFrame(label_index_dict, index=[0]).T.reset_index().rename(columns={'index':'class_ind', 0:'class_names'})
        # Class wise precision, recall and f1_score
        classwise_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
        classwise_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
        classwise_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)

        ind_class_count_df = class_ind_counter_from_tfrecord(data)
        ind_class_count_df = ind_class_count_df.merge(label_index_df, how='left', left_on='class_names', right_on='class_names')

        classwise_test_results = {'classwise_f1_score':classwise_score, 'classwise_precision':classwise_precision,
                        'classwise_recall':classwise_recall, 'class_names':CLASSES}
        classwise_test_results_df = pd.DataFrame(classwise_test_results)
    
        if produce_confusion_matrix is True:
            cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
            return overall_test_results, classwise_test_results, cmat
        return overall_test_results, classwise_test_results
        
    if produce_confusion_matrix is True:
        cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
        return overall_test_results, cmat
    warnings.filterwarnings('always')
    return overall_test_results
</code></pre>

CC BY-SA 4.0
2022-07-15T15:44:40.073Z
are you sure this is not a coding issue? Have you tried feeding dummy predictions and labels all equal to check that in both cases you indeed get 100%? When this happen to me all the time I find some silent bug in my evaluation functions, like a wrong indent that cause me to write to tensorboard only a fraction of the predictions (which are then divided by the total amount of images which could explain the super low scores)
Edoardo Guerriero
CC BY-SA 4.0
2022-07-15T18:40:08.400Z
@EdoardoGuerriero I have checked this code looks fine to me. Adding code for your reference. Even I used this code with some previous iteration and that time it was working fine
learner
<computer-vision><tensorflow><metric><confusion-matrix><mobile-net-v2>
Metrics using batches v/s metrics using full dataset
learner
# Metrics using batches v/s metrics using full dataset

(Asked by: username_0 on 2022-07-15)

I am using training an image classification model using the pre-trained mobile network. During training, I am seeing very high values (more than 70%) for Accuracy, Precision, Recall, and F1-score on both the training dataset and validation dataset.


For me, this is an indication that my model is learning fine.
But when I checked these metrics on an Unbatched training and Unbatched Validation these metrics are very low. Even are then on 1%.
Unbatched dataset means I am not taking calculating these metrics over batches and taking the average of metrics to calculate the final metrics which is what Tensorflow/Keras does during model training.

I am unable to find out what is causing this Behaviour. Please help me understand what is causing this difference and how to ensure that results are consistent on both, i.e. a minor difference is acceptable.
Code that I used for evaluating metrics
def test_model(model, data, CLASSES, label_one_hot=True, average="micro", 
                threshold_analysis=False, thres_analysis_start_point=0.0, 
                thres_analysis_end_point=0.95, thres_step=0.05, classwise_analysis=False,
                produce_confusion_matrix=False):
    images_ds = data.map(lambda image, label: image)
    labels_ds = data.map(lambda image, label: label).unbatch()
    NUM_VALIDATION_IMAGES = count_data_items(tf_records_filenames=data)
    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch
    if label_one_hot is True:
        cm_correct_labels = np.argmax(cm_correct_labels, axis=-1)
    cm_probabilities = model.predict(images_ds)
    cm_predictions = np.argmax(cm_probabilities, axis=-1)
    
    warnings.filterwarnings('ignore')

    overall_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    overall_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    overall_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=average)
    # cmat = (cmat.T / cmat.sum(axis=1)).T # normalized
    # print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))
    overall_test_results = {'overall_f1_score': overall_score, 'overall_precision':overall_precision, 'overall_recall':overall_recall}

    if classwise_analysis is True:
        
        label_index_dict = get_index_label_from_tf_record(dataset=data)
        label_index_dict = {k:v for k, v in sorted(list(label_index_dict.items()))}
        label_index_df = pd.DataFrame(label_index_dict, index=[0]).T.reset_index().rename(columns={'index':'class_ind', 0:'class_names'})
        # Class wise precision, recall and f1_score
        classwise_score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
        classwise_precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)
        classwise_recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average=None)

        ind_class_count_df = class_ind_counter_from_tfrecord(data)
        ind_class_count_df = ind_class_count_df.merge(label_index_df, how='left', left_on='class_names', right_on='class_names')

        classwise_test_results = {'classwise_f1_score':classwise_score, 'classwise_precision':classwise_precision,
                        'classwise_recall':classwise_recall, 'class_names':CLASSES}
        classwise_test_results_df = pd.DataFrame(classwise_test_results)
    
        if produce_confusion_matrix is True:
            cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
            return overall_test_results, classwise_test_results, cmat
        return overall_test_results, classwise_test_results
        
    if produce_confusion_matrix is True:
        cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))
        return overall_test_results, cmat
    warnings.filterwarnings('always')
    return overall_test_results

**Tags:** computer-vision, tensorflow, metric, confusion-matrix, mobile-net-v2

> **Comments:**
> **username_1**: are you sure this is not a coding issue? Have you tried feeding dummy predictions and labels all equal to check that in both cases you indeed get 100%? When this happen to me all the time I find some silent bug in my evaluation functions, like a wrong indent that cause me to write to tensorboard only a fraction of the predictions (which are then divided by the total amount of images which could explain the super low scores)  
> **username_0**: @EdoardoGuerriero I have checked this code looks fine to me. Adding code for your reference. Even I used this code with some previous iteration and that time it was working fine  

---

## 1 answers

### Answer by username_0 on 2022-07-17. Score: 0 | Accepted answer

After spending many hours, I found the issue was due to the shuffle function. I was using the below function to shuffle, batch and prefetch the dataset.
def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                            shuffle_buffer_size=None,
                            drop_remainder=False, 
                            interleave_num_pcall=None):

    if shuffle_buffer_size is None:
        raise ValueError("shuffle_buffer_size can't be None")
    def shuffle_fn(ds):
         return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
    dataset = dataset.apply(shuffle_fn)
    dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
    dataset = dataset.prefetch(buffer_size=prefetch_size)
    return dataset

Part of the function that causes the problem
def shuffle_fn(ds):
    return ds.shuffle(buffer_size=shuffle_buffer_size, seed=108)
dataset = dataset.apply(shuffle_fn)

I removed the shuffle part and metrics are back as per the expectation.
Function after removing the shuffle part
def shuffle_batch_prefetch(dataset, prefetch_size=1, batch_size=16, 
                           drop_remainder=False, 
                          interleave_num_pcall=None):
    dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
    dataset = dataset.prefetch(buffer_size=prefetch_size)
    return dataset

Results after removing the shuffle part

I am still not able to understand why shuffling causes this error. Shuffling was the best practice to follow before training your data. Although, I have already shuffled training data during data read time.
I have also posted the same issue at SO thinking that it might be related to some coding error.


---

