Answers.Body: <p>Even you trained your classifier using CE loss, in practice the usual overfitting, imbalanced training data and distribution shift may result in poorly calibrated predictions as discussed in a recent <a href="https://ai.stackexchange.com/questions/46373/dealing-with-likely-class-imbalance-for-unlabeled-test-data/">post</a> here. For modern neural network classifiers Guo et al showed empirically that a simple temperature scaling as a single parameter variant of Platt scaling produces the lowest expected calibration error (ECE) on a variety of ANN classification problems in their 2017 paper <em>On Calibration of Modern Neural Networks</em>.</p>
  <blockquote>
  <p>Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration... Modern neural networks exhibit a strange phenomenon: probabilistic error and miscalibration worsen even as classification error is reduced...  Nevertheless, simple techniques can effectively remedy the miscalibration phenomenon in neural networks. Temperature scaling is the simplest, fastest, and most straightforward of the methods, and surprisingly is often the most effective.</p>
  </blockquote>
  <p>In summary calibrations usually do not affect the identity of the most probable class label, so these methods may have little impact on classification accuracy. However, they do improve calibration performance in terms of forecasting, for example, the effect of relatively high temperature (&gt;1) in temperature scaling is to make the trained model's predictive distribution less peaky which you'd better employ to adjust your thresholds <em>after</em> you trained your model. Your class weights essentially acting as cost-structure should be only used in your reweighted CE loss during training. Of course you can also move calibrated ratios to loss as additional class weights in some distribution shift calibration cases as discussed in the linked post which your problem hopefully doesn't face.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-09T05:03:10.427Z
Answers.LastActivityDate: 2024-08-09T05:03:10.427Z
Answers.UserDisplayName: cinch
Body: <p>I have an image classification task that uses an object detection model as its basis. For each image, I get a vector of confidences (one value for each class), and I take the class with the highest confidence as the actual prediction for the image. I also have a weight for each class - all confidences of a class will be multiplied by a weight; this is used to improve accuracy when confidences for certain classes are abnormally low or high (as a result of the nature of the model). In addition, I also have a vector of &quot;thresholds&quot; (one for each class). These reduce to 0 any confidences that are below the threshold. Again, this is used to tune and improve accuracy.</p>
  <p>I am currently working on a method to automatically tune these &quot;class weights&quot;, using a ground truth labeled dataset. I try to adjust these class weights/thresholds such that my confidences (after weighting and thresholding) are calibrated so that the highest confidences for each image are aligned with the ground truth label as much as possible. I am currently using a gradient descent method based on cross-entropy (similar logic with normal classification model training) to tune these weights, but I am stuck on what to do with the thresholds. Anyone has any suggestions for technical ideas/algorithms that fit my task?</p>
  
Tags: <machine-learning><computer-vision><optimization>
Title: Is there a better way to do this type of optimization?
UserDisplayName: Hummus Akemi
ThreadText: # Is there a better way to do this type of optimization?
  
  (Asked by: username_0 on 2024-08-09)
  
  I have an image classification task that uses an object detection model as its basis. For each image, I get a vector of confidences (one value for each class), and I take the class with the highest confidence as the actual prediction for the image. I also have a weight for each class - all confidences of a class will be multiplied by a weight; this is used to improve accuracy when confidences for certain classes are abnormally low or high (as a result of the nature of the model). In addition, I also have a vector of "thresholds" (one for each class). These reduce to 0 any confidences that are below the threshold. Again, this is used to tune and improve accuracy.
  
  I am currently working on a method to automatically tune these "class weights", using a ground truth labeled dataset. I try to adjust these class weights/thresholds such that my confidences (after weighting and thresholding) are calibrated so that the highest confidences for each image are aligned with the ground truth label as much as possible. I am currently using a gradient descent method based on cross-entropy (similar logic with normal classification model training) to tune these weights, but I am stuck on what to do with the thresholds. Anyone has any suggestions for technical ideas/algorithms that fit my task?
  
  **Tags:** machine-learning, computer-vision, optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-09. Score: 0
  
  Even you trained your classifier using CE loss, in practice the usual overfitting, imbalanced training data and distribution shift may result in poorly calibrated predictions as discussed in a recent [post](https://ai.stackexchange.com/questions/46373/dealing-with-likely-class-imbalance-for-unlabeled-test-data/) here. For modern neural network classifiers Guo et al showed empirically that a simple temperature scaling as a single parameter variant of Platt scaling produces the lowest expected calibration error (ECE) on a variety of ANN classification problems in their 2017 paper *On Calibration of Modern Neural Networks*.
  
  > Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration... Modern neural networks exhibit a strange phenomenon: probabilistic error and miscalibration worsen even as classification error is reduced... Nevertheless, simple techniques can effectively remedy the miscalibration phenomenon in neural networks. Temperature scaling is the simplest, fastest, and most straightforward of the methods, and surprisingly is often the most effective.
  
  In summary calibrations usually do not affect the identity of the most probable class label, so these methods may have little impact on classification accuracy. However, they do improve calibration performance in terms of forecasting, for example, the effect of relatively high temperature (>1) in temperature scaling is to make the trained model's predictive distribution less peaky which you'd better employ to adjust your thresholds *after* you trained your model. Your class weights essentially acting as cost-structure should be only used in your reweighted CE loss during training. Of course you can also move calibrated ratios to loss as additional class weights in some distribution shift calibration cases as discussed in the linked post which your problem hopefully doesn't face.
  
  
  ---
  
  