Answers.Body: <p><strong>1.</strong> Yes, accuracy can be lower than the F score for a set of predictions and targets. You can easily test it by generating a fixed array of predictions and a varying array of labels (or vice versa). You'll get a graph like in the image below.
  Notice that you could obtain a horizontally mirrored version of the same graph by inverting the labels (or predictions).</p>
  <p>To understand why you need to keep in mid that the F score is a skewed metric,because in its calculation we ignore an entire class, i.e. the true negatives.</p>
  <p><span class="math-container">$$F1= \frac{2 TP}{2TP + FP + FN} $$</span></p>
  <p>So if I have an unbalanced dataset with only 10 instances of class A and 90 of class B I'll get an accuracy of 90% by predicting only B, with an f score of 0%, but if I invert the classes (i.e. predict only A) I'll get an accuracy of 10% and an F score of 18%.</p>
  <p><strong>2.</strong> The safer answer is always: check all metrics. Every metric tells you a different story about the model performance, so it's always a good practice to not rely only on one. But it might depends also on your specific use case. Why do you need to classify between low and high (whatever low and high refer to)? Do you care about a specific class? Then the recall is the most interesting metric to keep an eye on. In general, your problem is similar to probability thresholding when calculating the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic" rel="nofollow noreferrer">ROC-AUC score</a> so you might want to check it out and see if it's applicable also to dataset splitting (a bit impractical if a single training session of your model is long though), if it's applicable, it's surely more reliable than the F score.</p>
  <p><a href="https://i.sstatic.net/Czqb6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Czqb6.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-03T14:51:30.000Z
Answers.LastActivityDate: 2022-02-03T14:51:30.000Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm working on a classification machine learning problem with two classes: high and low, which are derived from another numerical column x. Previously, if x&gt;100, the sample is considered <code>high</code>, otherwise, it is considered <code>low</code>. I used a 1D CNN model.</p>
  <p>I wanted to test if changing the threshold would impact on the model performance. So I increased the threshold to <code>1000</code>(ie. if <code>x&gt;1000</code>, the sample is considered <code>high</code>, otherwise, it is considered <code>low</code>.
  ).</p>
  <p>What I did:</p>
  <ol>
  <li>Increased the threshold from <code>100</code> to <code>1000</code>, thus the number of samples that fall into the <code>low</code> category increased and that of the <code>high</code> category decreased. The data thus become imbalanced with the following ratio:</li>
  </ol>
  <pre><code>Ratio of low : high
  
  (with 100 threshold)       43% : 57%
  
  (with 1000 threshold)       8% : 92%
  </code></pre>
  <ol start="2">
  <li>Noticing that the imbalanced dataset might lead to imbalanced training data, I applied <code>undersampling</code> to the training data to make sure that there are equal number of <code>low</code> and <code>high</code> samples.</li>
  <li>At 100 threshold, the model achieved an accuracy level of around 80%. At 1000 threshold, the model achieved an accuracy level of around 69%, <strong>which is significantly lower than that of the 100 threshold.</strong></li>
  <li>I then saved each model and generated a prediction using each model. I evaluated both sets of results with the <code>Precision</code>, <code>Recall</code> and <code>F1 Score</code>.</li>
  </ol>
  <pre><code>
                         Precision: 0.38
  (with 100 threshold)   Recall: 0.86
                         F1: 0.527
  
  
                         Precision: 0.8307692307692308 
  (with 1000 threshold)  Recall: 0.6206896551724138 
                         F1: 0.7105263157894737
  </code></pre>
  <p>Based on accuracy level, it seems 100 threshold achieved a better performance than that of 1000 threshold. However, according to <code>F1 Score</code>, <strong>1000 threshold seems to achieve better score 0.71 (as compared to 0.527 of 100 threshold).</strong></p>
  <p>My question are:</p>
  <ol>
  <li>Is it possible for a model to have lower accuracy level but higher <code>F1 Score</code>? How can this make sense in light of confusion-matrix measures and model accuracy level?</li>
  <li>Which measures should I use to determine using which threshold is better?</li>
  </ol>
  <hr />
  <p>Edit:</p>
  <p>Since the entire code is too long, I will show some snippets of it below.</p>
  <p>To answer @MASTER OF CODE's question, I believe the test data is balanced and tested in the validation stage at the end of each epoch as I called <code>model.fit(x_train,y_train, validation)</code> in Keras which used the <a href="https://www.tensorflow.org/guide/keras/train_and_evaluate" rel="nofollow noreferrer">built-in API for training and evaluation</a>(please cmiiw).</p>
  <p><strong>For Undersampling:</strong></p>
  <p>I used <code>np.random.choice()</code> to randomly select an equal amount of majority samples as the minority group, then concatenated the minority and majority samples into one dataframe named <code>under_sample</code>, before fitting a <code>1D CNN</code> model on the dataframe, see below:</p>
  <pre><code>np.random.seed(42)
  ############ minority_class_len
  minority_class_len = len(df[df['Label'] ==1])
  
  
  ############ majority_class_indices
  majority_class_indices = df[df['Label'] ==0].index
  
  
  ############ random_majority_indices
  random_majority_indices = np.random.choice(majority_class_indices,
                                             minority_class_len,
                                             replace = False)
  
  ############ minority_class_indices
  minority_class_indices = df[df['Label'] ==1].index
  
  
  ############ concatenate positive and negative sample indices
  under_sample_indices = np.concatenate([minority_class_indices, random_majority_indices])
  
  
  ############ select samples by indices
  under_sample = df.loc[under_sample_indices]
  <span class="math-container">```</span>
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-03T10:02:42.677Z
Comments.Text: Is Your testing set also balanced? Also could You show some code to recreate this behaviour?
Comments.UserDisplayName: MASTER OF CODE
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-03T13:09:35.453Z
Comments.Text: Hi @MASTEROFCODE, thanks for your reply, please see question edit. Unfortunately I'm unable to upload the entire code but I added some snippets, hopefully it makes it a bit clear.
Comments.UserDisplayName: nilsinelabore
Tags: <convolutional-neural-networks><performance><accuracy><confusion-matrix>
Title: Confusion Matrix Measures vs Accuracy level in Neural Network Model
UserDisplayName: nilsinelabore
ThreadText: # Confusion Matrix Measures vs Accuracy level in Neural Network Model
  
  (Asked by: username_0 on 2022-02-03)
  
  I'm working on a classification machine learning problem with two classes: high and low, which are derived from another numerical column x. Previously, if x>100, the sample is considered `high`, otherwise, it is considered `low`. I used a 1D CNN model.
  
  I wanted to test if changing the threshold would impact on the model performance. So I increased the threshold to `1000`(ie. if `x>1000`, the sample is considered `high`, otherwise, it is considered `low`.
  ).
  
  What I did:
  
  1. Increased the threshold from `100` to `1000`, thus the number of samples that fall into the `low` category increased and that of the `high` category decreased. The data thus become imbalanced with the following ratio:
  
  ```
  Ratio of low : high
  
  (with 100 threshold)       43% : 57%
  
  (with 1000 threshold)       8% : 92%
  
  ```
  
  2. Noticing that the imbalanced dataset might lead to imbalanced training data, I applied `undersampling` to the training data to make sure that there are equal number of `low` and `high` samples.
  3. At 100 threshold, the model achieved an accuracy level of around 80%. At 1000 threshold, the model achieved an accuracy level of around 69%, **which is significantly lower than that of the 100 threshold.**
  4. I then saved each model and generated a prediction using each model. I evaluated both sets of results with the `Precision`, `Recall` and `F1 Score`.
  
  ```
  
                         Precision: 0.38
  (with 100 threshold)   Recall: 0.86
                         F1: 0.527
  
                         Precision: 0.8307692307692308 
  (with 1000 threshold)  Recall: 0.6206896551724138 
                         F1: 0.7105263157894737
  
  ```
  
  Based on accuracy level, it seems 100 threshold achieved a better performance than that of 1000 threshold. However, according to `F1 Score`, **1000 threshold seems to achieve better score 0.71 (as compared to 0.527 of 100 threshold).**
  
  My question are:
  
  1. Is it possible for a model to have lower accuracy level but higher `F1 Score`? How can this make sense in light of confusion-matrix measures and model accuracy level?
  2. Which measures should I use to determine using which threshold is better?
  
  ---
  
  Edit:
  
  Since the entire code is too long, I will show some snippets of it below.
  
  To answer @username_1's question, I believe the test data is balanced and tested in the validation stage at the end of each epoch as I called `model.fit(x_train,y_train, validation)` in Keras which used the [built-in API for training and evaluation](https://www.tensorflow.org/guide/keras/train_and_evaluate)(please cmiiw).
  
  **For Undersampling:**
  
  I used `np.random.choice()` to randomly select an equal amount of majority samples as the minority group, then concatenated the minority and majority samples into one dataframe named `under_sample`, before fitting a `1D CNN` model on the dataframe, see below:
  
  ```
  np.random.seed(42)
  ############ minority_class_len
  minority_class_len = len(df[df['Label'] ==1])
  
  ############ majority_class_indices
  majority_class_indices = df[df['Label'] ==0].index
  
  ############ random_majority_indices
  random_majority_indices = np.random.choice(majority_class_indices,
                                             minority_class_len,
                                             replace = False)
  
  ############ minority_class_indices
  minority_class_indices = df[df['Label'] ==1].index
  
  ############ concatenate positive and negative sample indices
  under_sample_indices = np.concatenate([minority_class_indices, random_majority_indices])
  
  ############ select samples by indices
  under_sample = df.loc[under_sample_indices]
  ```
  
  ```
  
  **Tags:** convolutional-neural-networks, performance, accuracy, confusion-matrix
  
  > **Comments:**
  > **username_1**: Is Your testing set also balanced? Also could You show some code to recreate this behaviour?  
  > **username_0**: Hi @username_1, thanks for your reply, please see question edit. Unfortunately I'm unable to upload the entire code but I added some snippets, hopefully it makes it a bit clear.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-03. Score: 0 | Accepted answer
  
  **1.** Yes, accuracy can be lower than the F score for a set of predictions and targets. You can easily test it by generating a fixed array of predictions and a varying array of labels (or vice versa). You'll get a graph like in the image below.
  Notice that you could obtain a horizontally mirrored version of the same graph by inverting the labels (or predictions).
  
  To understand why you need to keep in mid that the F score is a skewed metric,because in its calculation we ignore an entire class, i.e. the true negatives.
  
  $$F1= \frac{2 TP}{2TP + FP + FN} $$
  
  So if I have an unbalanced dataset with only 10 instances of class A and 90 of class B I'll get an accuracy of 90% by predicting only B, with an f score of 0%, but if I invert the classes (i.e. predict only A) I'll get an accuracy of 10% and an F score of 18%.
  
  **2.** The safer answer is always: check all metrics. Every metric tells you a different story about the model performance, so it's always a good practice to not rely only on one. But it might depends also on your specific use case. Why do you need to classify between low and high (whatever low and high refer to)? Do you care about a specific class? Then the recall is the most interesting metric to keep an eye on. In general, your problem is similar to probability thresholding when calculating the [ROC-AUC score](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) so you might want to check it out and see if it's applicable also to dataset splitting (a bit impractical if a single training session of your model is long though), if it's applicable, it's surely more reliable than the F score.
  
  [![enter image description here](https://i.sstatic.net/Czqb6.png)](https://i.sstatic.net/Czqb6.png)
  
  
  ---
  
  