Bayesian Prediction-Powered Inference
<|reference_start|>Bayesian Prediction-Powered Inference: Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. Specifically, PPI methods provide tighter confidence intervals by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate, but potentially biased, automatic system. We propose a framework for PPI based on Bayesian inference that allows researchers to develop new task-appropriate PPI methods easily. Exploiting the ease with which we can design new metrics, we propose improved PPI methods for several importantcases, such as autoraters that give discrete responses (e.g., prompted LLM ``judges'') and autoraters with scores that have a non-linear relationship to human scores.<|reference_end|>
@article{hofer2024bayesian,
    title={Bayesian Prediction-Powered Inference},
    author={R. Alex Hofer and Joshua Maynez and Bhuwan Dhingra and Adam Fisch and
  Amir Globerson and William W. Cohen},
    journal={arXiv preprint arXiv:2405.06034},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.06034},
    primaryClass={cs.LG}
}