Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs
<|reference_start|>Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs: In this work, we introduce the Learnable Response Scoring Function (LARS) for Uncertainty Estimation (UE) in generative Large Language Models (LLMs). Current scoring functions for probability-based UE, such as length-normalized scoring and semantic contribution-based weighting, are designed to solve specific aspects of the problem but exhibit limitations, including the inability to handle biased probabilities and under-performance in low-resource languages like Turkish. To address these issues, we propose LARS, a scoring function that leverages supervised data to capture complex dependencies between tokens and probabilities, thereby producing more reliable and calibrated response scores in computing the uncertainty of generations. Our extensive experiments across multiple datasets show that LARS substantially outperforms existing scoring functions considering various probability-based UE methods.<|reference_end|>
@article{yaldiz2024do,
    title={Do Not Design, Learn: A Trainable Scoring Function for Uncertainty
  Estimation in Generative LLMs},
    author={Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang
  Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, Salman Avestimehr},
    journal={arXiv preprint arXiv:2406.11278},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11278},
    primaryClass={cs.CL}
}