Pearls from Pebbles: Improved Confidence Functions for Auto-labeling
<|reference_start|>Pearls from Pebbles: Improved Confidence Functions for Auto-labeling: Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual labeling. A prominent variant, threshold-based auto-labeling (TBAL), works by finding a threshold on a model's confidence scores above which it can accurately label unlabeled data points. However, many models are known to produce overconfident scores, leading to poor TBAL performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, such methods still fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the \emph{optimal} TBAL confidence function. We develop a tractable version of the framework to obtain \texttt{Colander} (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of our method \texttt{Colander} and compare it against methods designed for calibration. \texttt{Colander} achieves up to 60\% improvements on coverage over the baselines while maintaining auto-labeling error below $5\%$ and using the same amount of labeled data as the baselines.<|reference_end|>
@article{vishwakarma2024pearls,
    title={Pearls from Pebbles: Improved Confidence Functions for Auto-labeling},
    author={Harit Vishwakarma, Reid (Yi) Chen, Sui Jiet Tay, Satya Sai Srinath
  Namburi, Frederic Sala, Ramya Korlakai Vinayak},
    journal={arXiv preprint arXiv:2404.16188},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16188},
    primaryClass={cs.LG cs.AI stat.ML}
}