title: Revisiting Uncertainty Quantification Evaluation in Language Models:
    Spurious Interactions with Response Length Bias Results
summary: Uncertainty Quantification (UQ) in Language Models (LMs) is crucial for
  improving their safety and reliability. Evaluations often use performance
  metrics like AUROC to assess how well UQ methods (e.g., negative sequence
  probabilities) correlate with task correctness functions (e.g., ROUGE-L). In
  this paper, we show that commonly used correctness functions bias UQ
  evaluations by inflating the performance of certain UQ methods. We evaluate 7
  correctness functions -- from lexical-based and embedding-based metrics to
  LLM-as-a-judge approaches -- across 4 datasets x 4 models x 6 UQ methods. Our
  analysis reveals that length biases in the errors of these correctness
  functions distort UQ assessments by interacting with length biases in UQ
  methods. We identify LLM-as-a-judge approaches as among the least length-biased
  choices and hence a potential solution to mitigate these biases.
paper_authors._id: 6806a3464020c8351ddd6b90
paper_authors.name: Andrea Santilli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:51:54.306Z
paper_authors._id: 5e8ef1f14957053f606489e6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635502086699-5e8ef1f14957053f606489e6.jpeg
paper_authors.fullname: Andrea Santilli
paper_authors.type: user
paper_authors.user: teelinsan
paper_authors._id: 6806a3464020c8351ddd6b91
paper_authors.name: Adam Golinski
paper_authors._id: 6806a3464020c8351ddd6b92
paper_authors.name: Michael Kirchhof
paper_authors._id: 6806a3464020c8351ddd6b93
paper_authors.name: Federico Danieli
paper_authors._id: 6806a3464020c8351ddd6b94
paper_authors.name: Arno Blaas
paper_authors._id: 6806a3464020c8351ddd6b95
paper_authors.name: Miao Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T10:03:45.808Z
paper_authors._id: 6530cf34e7535baecd9620a7
paper_authors.avatarUrl: /avatars/e6058a932d88e42b4957734f653cbcfd.svg
paper_authors.fullname: Miao Xiong
paper_authors.type: user
paper_authors.user: happymio
paper_authors._id: 6806a3464020c8351ddd6b96
paper_authors.name: Luca Zappella
paper_authors._id: 6806a3464020c8351ddd6b97
paper_authors.name: Sinead Williamson
paper_title: Revisiting Uncertainty Quantification Evaluation in Language Models:
    Spurious Interactions with Response Length Bias Results
paper_summary: Uncertainty Quantification (UQ) in Language Models (LMs) is crucial for
  improving their safety and reliability. Evaluations often use performance
  metrics like AUROC to assess how well UQ methods (e.g., negative sequence
  probabilities) correlate with task correctness functions (e.g., ROUGE-L). In
  this paper, we show that commonly used correctness functions bias UQ
  evaluations by inflating the performance of certain UQ methods. We evaluate 7
  correctness functions -- from lexical-based and embedding-based metrics to
  LLM-as-a-judge approaches -- across 4 datasets x 4 models x 6 UQ methods. Our
  analysis reveals that length biases in the errors of these correctness
  functions distort UQ assessments by interacting with length biases in UQ
  methods. We identify LLM-as-a-judge approaches as among the least length-biased
  choices and hence a potential solution to mitigate these biases.
paper_ai_summary: Evaluations of Uncertainty Quantification (UQ) in language models are biased by correctness functions, affecting UQ methods' performance, with LLM-as-a-judge approaches identified as less biased.
paper_ai_keywords: Uncertainty Quantification (UQ) Language Models (LMs) AUROC negative sequence probabilities ROUGE-L lexical-based metrics embedding-based metrics length biases LLM-as-a-judge approaches