Full-ECE: A Metric For Token-level Calibration on Large Language Models
<|reference_start|>Full-ECE: A Metric For Token-level Calibration on Large Language Models: Deep Neural Networks (DNNs) excel in various domains but face challenges in providing accurate uncertainty estimates, which are crucial for high-stakes applications. Large Language Models (LLMs) have recently emerged as powerful tools, demonstrating exceptional performance in language tasks. However, traditional calibration metrics such as Expected Calibration Error (ECE) and classwise-ECE (cw-ECE) are inadequate for LLMs due to their vast vocabularies, data complexity, and distributional focus. To address this, we propose a novel calibration concept called full calibration and introduce its corresponding metric, Full-ECE. Full-ECE evaluates the entire predicted probability distribution, offering a more accurate and robust measure of calibration for LLMs.<|reference_end|>
@article{liu2024full-ece:,
    title={Full-ECE: A Metric For Token-level Calibration on Large Language Models},
    author={Han Liu, Yupeng Zhang, Bingning Wang, Weipeng Chen, Xiaolin Hu},
    journal={arXiv preprint arXiv:2406.11345},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11345},
    primaryClass={cs.CL cs.AI}
}