Is "probability calibration" intended to improve the performance of a statistical model? I was watching this video over here: https://www.youtube.com/watch?v=AunotauS5yI
This video brought up an interesting point that I never knew had a specific term for (i.e. probability calibration).
If I have understood this correctly: in the case of a supervised binary classification problem, the goal of probability calibration is to attempt to make sure that when the statistical model predicts a new observation with a higher probabilities - on average, higher prediction probabilities should more likely to result in correct predictions (e.g. see @ 6:04 in the video)
Question: It seems that probability calibration models the prediction probabilities of the trained model vs the true labels. Is this more for diagnostic purposes? Or can doing this actually improve the statistical model?
Has anyone ever attempted to do this before? I found a way to do this using the R programming language:
https://rdrr.io/cran/rfUtilities/man/probability.calibration.html

No, you did not understand it correctly. Calibration will not change the ordering of your predictions, so relative a higher predictions before calibration is still a higher after calibration relative to other predictions, but the actual values of these predictions will be changed. So the goal is not to get higher probabilities but to get more correct probabilities. This is explained at 1:00 in the video. This will improve the model based on measures affected by the calibration, such as brier score or log score, but it will not affect measures that are not, such as AUC. This is not only for diagnostic purposes but because we often care about probabilistic predictions since probabilistic predictions are often more useful than just categorical predictions. For example, would you rather get a prediction that you will not get a heart attack tomorrow or that the chance of you getting a heart attack tomorrow is 49%?
Has anyone ever attempted to do this before? Yes, people do this all the time. That's why you are being taught this. For example, the Platt's scaling paper has 6500+ citations
