Measuring Stochastic Data Complexity with Boltzmann Influence Functions
<|reference_start|>Measuring Stochastic Data Complexity with Boltzmann Influence Functions: Estimating the uncertainty of a model's prediction on a test point is a crucial part of ensuring reliability and calibration under distribution shifts. A minimum description length approach to this problem uses the predictive normalized maximum likelihood (pNML) distribution, which considers every possible label for a data point, and decreases confidence in a prediction if other labels are also consistent with the model and training data. In this work we propose IF-COMP, a scalable and efficient approximation of the pNML distribution that linearizes the model with a temperature-scaled Boltzmann influence function. IF-COMP can be used to produce well-calibrated predictions on test points as well as measure complexity in both labelled and unlabelled settings. We experimentally validate IF-COMP on uncertainty calibration, mislabel detection, and OOD detection tasks, where it consistently matches or beats strong baseline methods.<|reference_end|>
@article{ng2024measuring,
    title={Measuring Stochastic Data Complexity with Boltzmann Influence Functions},
    author={Nathan Ng and Roger Grosse and Marzyeh Ghassemi},
    journal={arXiv preprint arXiv:2406.02745},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02745},
    primaryClass={cs.LG}
}