Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt
<|reference_start|>Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt: We formulate the problem of constructing multiple simultaneously valid confidence intervals (CIs) as estimating a high probability upper bound on the maximum error for a class/set of estimate-estimand-error tuples, and refer to this as the error estimation problem. For a single such tuple, data-driven confidence intervals can often be used to bound the error in our estimate. However, for a class of estimate-estimand-error tuples, nontrivial high probability upper bounds on the maximum error often require class complexity as input -- limiting the practicality of such methods and often resulting in loose bounds. Rather than deriving theoretical class complexity-based bounds, we propose a completely data-driven approach to estimate an upper bound on the maximum error. The simple and general nature of our solution to this fundamental challenge lends itself to several applications including: multiple CI construction, multiple hypothesis testing, estimating excess risk bounds (a fundamental measure of uncertainty in machine learning) for any training/fine-tuning algorithm, and enabling the development of a contextual bandit pipeline that can leverage any reward model estimation procedure as input (without additional mathematical analysis).<|reference_end|>
@article{krishnamurthy2024data-driven,
    title={Data-driven Error Estimation: Upper Bounding Multiple Errors with No
  Technical Debt},
    author={Sanath Kumar Krishnamurthy, Susan Athey, Emma Brunskill},
    journal={arXiv preprint arXiv:2405.04636},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04636},
    primaryClass={cs.LG stat.ML}
}