Just rotate it! Uncertainty estimation in closed-source models via multiple queries
<|reference_start|>Just rotate it! Uncertainty estimation in closed-source models via multiple queries: We propose a simple and effective method to estimate the uncertainty of closed-source deep neural network image classification models. Given a base image, our method creates multiple transformed versions and uses them to query the top-1 prediction of the closed-source model. We demonstrate significant improvements in the calibration of uncertainty estimates compared to the naive baseline of assigning 100\% confidence to all predictions. While we initially explore Gaussian perturbations, our empirical findings indicate that natural transformations, such as rotations and elastic deformations, yield even better-calibrated predictions. Furthermore, through empirical results and a straightforward theoretical analysis, we elucidate the reasons behind the superior performance of natural transformations over Gaussian noise. Leveraging these insights, we propose a transfer learning approach that further improves our calibration results.<|reference_end|>
@article{pitas2024just,
    title={Just rotate it! Uncertainty estimation in closed-source models via
  multiple queries},
    author={Konstantinos Pitas, Julyan Arbel},
    journal={arXiv preprint arXiv:2405.13864},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13864},
    primaryClass={cs.CV cs.AI}
}