Overcoming Common Flaws in the Evaluation of Selective Classification Systems
<|reference_start|>Overcoming Common Flaws in the Evaluation of Selective Classification Systems: Selective Classification, wherein models can reject low-confidence predictions, promises reliable translation of machine-learning based classification systems to real-world scenarios such as clinical diagnostics. While current evaluation of these systems typically assumes fixed working points based on pre-defined rejection thresholds, methodological progress requires benchmarking the general performance of systems akin to the $\mathrm{AUROC}$ in standard classification. In this work, we define 5 requirements for multi-threshold metrics in selective classification regarding task alignment, interpretability, and flexibility, and show how current approaches fail to meet them. We propose the Area under the Generalized Risk Coverage curve ($\mathrm{AUGRC}$), which meets all requirements and can be directly interpreted as the average risk of undetected failures. We empirically demonstrate the relevance of $\mathrm{AUGRC}$ on a comprehensive benchmark spanning 6 data sets and 13 confidence scoring functions. We find that the proposed metric substantially changes metric rankings on 5 out of the 6 data sets.<|reference_end|>
@article{traub2024overcoming,
    title={Overcoming Common Flaws in the Evaluation of Selective Classification
  Systems},
    author={Jeremias Traub, Till J. Bungert, Carsten T. L"uth, Michael
  Baumgartner, Klaus H. Maier-Hein, Lena Maier-Hein, Paul F Jaeger},
    journal={arXiv preprint arXiv:2407.01032},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01032},
    primaryClass={cs.LG cs.CV stat.ME}
}