Improving Predictor Reliability with Selective Recalibration
<|reference_start|>Improving Predictor Reliability with Selective Recalibration: A reliable deep learning system should be able to accurately express its confidence with respect to its predictions, a quality known as calibration. One of the most effective ways to produce reliable confidence estimates with a pre-trained model is by applying a post-hoc recalibration method. Popular recalibration methods like temperature scaling are typically fit on a small amount of data and work in the model's output space, as opposed to the more expressive feature embedding space, and thus usually have only one or a handful of parameters. However, the target distribution to which they are applied is often complex and difficult to fit well with such a function. To this end we propose \textit{selective recalibration}, where a selection model learns to reject some user-chosen proportion of the data in order to allow the recalibrator to focus on regions of the input space that can be well-captured by such a model. We provide theoretical analysis to motivate our algorithm, and test our method through comprehensive experiments on difficult medical imaging and zero-shot classification tasks. Our results show that selective recalibration consistently leads to significantly lower calibration error than a wide range of selection and recalibration baselines.<|reference_end|>
@article{zollo2024improving,
    title={Improving Predictor Reliability with Selective Recalibration},
    author={Thomas P. Zollo, Zhun Deng, Jake C. Snell, Toniann Pitassi, Richard
  Zemel},
    journal={arXiv preprint arXiv:2410.05407},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05407},
    primaryClass={cs.LG cs.AI}
}