Answers.Body: <p>Consider use proper scoring metrics that account for prediction confidence as you specified such as the <a href="https://en.wikipedia.org/wiki/Brier_score" rel="nofollow noreferrer">Brier score</a> which is based on squared error and thus is less sensitive to extremely rare or extremely common classes.</p>
  <blockquote>
  <p>The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities. The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1).</p>
  </blockquote>
  <p>So you can train your classification model using your existing weighted proper scoring cross-entropy accounting for class imbalance or using a well calibrated metric such as Brier score as your loss function. Of course there's nothing wrong to try your own customary metric and I believe you should increase penalty for guessing the &quot;uncertain&quot; class, but bear in mind that it may not be proper in the sense of calibration.</p>
  <p>Then you can use a validation set to empirically find the cutoffs that minimize the uncertain class while maintaining high accuracy for class 0 and class 1. Simple calibration technique such as temperature scaling is discussed in this recent <a href="https://ai.stackexchange.com/questions/46473/is-there-a-better-way-to-do-this-type-of-optimization/46474#46474">answer</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-14T22:15:40.073Z
Answers.Text: Thank you for the answer and confirming my intuition! Special thanks for your comment on temperature scaling -- I hadn't considered it before, but in the context of my problem is actually extremely valuable.
Answers.UserDisplayName: StatisticallyImprobable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-14T00:17:45.423Z
Answers.LastActivityDate: 2024-08-14T00:23:52.103Z
Answers.LastEditDate: 2024-08-14T00:23:52.103Z
Answers.UserDisplayName: cinch
Body: <p>So I have a binary classification problem, however I don't want to predict on places where the model isn't very confident. So let's say for a probability output between 0.4 &amp; 0.7, I return no output. Above 0.7 I return class 1, below 0.4 I return class 0. I obviously want high performance while minimizing this uncertain class.</p>
  <p>My question is how do I pick the cutoffs &amp; the model? I am training using a weighted cross entropy function due to class imbalance which only complicates things. There's two questions here, how to pick the model and then how to pick the cutoffs:</p>
  <p>For picking the model:
  is this test loss itself the best way to determine the best model?</p>
  <p>Should I train the model from the onset to be a 3 class classifier with a lowered penalty for guessing this &quot;uncertain&quot; class?</p>
  <p>For picking cutoffs:
  Should I design my own scoring rule that somehow quantifies error and penalizes middle probabilities when picking cutoffs?</p>
  
Tags: <binary-classification>
Title: Treating binary classification as 3 class problem with two cutoffs -- what metric to use?
UserDisplayName: StatisticallyImprobable
ThreadText: # Treating binary classification as 3 class problem with two cutoffs -- what metric to use?
  
  (Asked by: username_0 on 2024-08-12)
  
  So I have a binary classification problem, however I don't want to predict on places where the model isn't very confident. So let's say for a probability output between 0.4 & 0.7, I return no output. Above 0.7 I return class 1, below 0.4 I return class 0. I obviously want high performance while minimizing this uncertain class.
  
  My question is how do I pick the cutoffs & the model? I am training using a weighted cross entropy function due to class imbalance which only complicates things. There's two questions here, how to pick the model and then how to pick the cutoffs:
  
  For picking the model:
  is this test loss itself the best way to determine the best model?
  
  Should I train the model from the onset to be a 3 class classifier with a lowered penalty for guessing this "uncertain" class?
  
  For picking cutoffs:
  Should I design my own scoring rule that somehow quantifies error and penalizes middle probabilities when picking cutoffs?
  
  **Tags:** binary-classification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-14. Score: 0 | Accepted answer
  
  Consider use proper scoring metrics that account for prediction confidence as you specified such as the [Brier score](https://en.wikipedia.org/wiki/Brier_score) which is based on squared error and thus is less sensitive to extremely rare or extremely common classes.
  
  > The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities. The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1).
  
  So you can train your classification model using your existing weighted proper scoring cross-entropy accounting for class imbalance or using a well calibrated metric such as Brier score as your loss function. Of course there's nothing wrong to try your own customary metric and I believe you should increase penalty for guessing the "uncertain" class, but bear in mind that it may not be proper in the sense of calibration.
  
  Then you can use a validation set to empirically find the cutoffs that minimize the uncertain class while maintaining high accuracy for class 0 and class 1. Simple calibration technique such as temperature scaling is discussed in this recent [answer](https://ai.stackexchange.com/questions/46473/is-there-a-better-way-to-do-this-type-of-optimization/46474#46474).
  
  > **Comments:**
  > **username_0**: Thank you for the answer and confirming my intuition! Special thanks for your comment on temperature scaling -- I hadn't considered it before, but in the context of my problem is actually extremely valuable.  
  
  ---
  
  