When is Multicalibration Post-Processing Necessary?
<|reference_start|>When is Multicalibration Post-Processing Necessary?: Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates. Multicalibration is a related notion -- originating in algorithmic fairness -- which requires predictors to be simultaneously calibrated over a potentially complex and overlapping collection of protected subpopulations (such as groups defined by ethnicity, race, or income). We conduct the first comprehensive study evaluating the usefulness of multicalibration post-processing across a broad set of tabular, image, and language datasets for models spanning from simple decision trees to 90 million parameter fine-tuned LLMs. Our findings can be summarized as follows: (1) models which are calibrated out of the box tend to be relatively multicalibrated without any additional post-processing; (2) multicalibration post-processing can help inherently uncalibrated models; and (3) traditional calibration measures may sometimes provide multicalibration implicitly. More generally, we also distill many independent observations which may be useful for practical and effective applications of multicalibration post-processing in real-world contexts.<|reference_end|>
@article{hansen2024when,
    title={When is Multicalibration Post-Processing Necessary?},
    author={Dutch Hansen, Siddartha Devic, Preetum Nakkiran, Vatsal Sharan},
    journal={arXiv preprint arXiv:2406.06487},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06487},
    primaryClass={cs.LG}
}