On Measuring Calibration of Discrete Probabilistic Neural Networks
<|reference_start|>On Measuring Calibration of Discrete Probabilistic Neural Networks: As machine learning systems become increasingly integrated into real-world applications, accurately representing uncertainty is crucial for enhancing their safety, robustness, and reliability. Training neural networks to fit high-dimensional probability distributions via maximum likelihood has become an effective method for uncertainty quantification. However, such models often exhibit poor calibration, leading to overconfident predictions. Traditional metrics like Expected Calibration Error (ECE) and Negative Log Likelihood (NLL) have limitations, including biases and parametric assumptions. This paper proposes a new approach using conditional kernel mean embeddings to measure calibration discrepancies without these biases and assumptions. Preliminary experiments on synthetic data demonstrate the method's potential, with future work planned for more complex applications.<|reference_end|>
@article{young2024beyond,
    title={Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors
  via Conditional Congruence},
    author={Spencer Young, Cole Edgren, Riley Sinema, Andrew Hall, Nathan Dong,
  Porter Jenkins},
    journal={arXiv preprint arXiv:2405.12412},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.12412},
    primaryClass={cs.LG stat.ML}
}