A Rate-Distortion View of Uncertainty Quantification
<|reference_start|>A Rate-Distortion View of Uncertainty Quantification: In supervised learning, understanding an input's proximity to the training data can help a model decide whether it has sufficient evidence for reaching a reliable prediction. While powerful probabilistic models such as Gaussian Processes naturally have this property, deep neural networks often lack it. In this paper, we introduce Distance Aware Bottleneck (DAB), i.e., a new method for enriching deep neural networks with this property. Building on prior information bottleneck approaches, our method learns a codebook that stores a compressed representation of all inputs seen during training. The distance of a new example from this codebook can serve as an uncertainty estimate for the example. The resulting model is simple to train and provides deterministic uncertainty estimates by a single forward pass. Finally, our method achieves better out-of-distribution (OOD) detection and misclassification prediction than prior methods, including expensive ensemble methods, deep kernel Gaussian Processes, and approaches based on the standard information bottleneck.<|reference_end|>
@article{apostolopoulou2024a,
    title={A Rate-Distortion View of Uncertainty Quantification},
    author={Ifigeneia Apostolopoulou, Benjamin Eysenbach, Frank Nielsen, Artur
  Dubrawski},
    journal={International Conference on Machine Learning, 2024},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10775},
    primaryClass={cs.LG cs.AI stat.ML}
}