title: Investigating Human-Aligned Large Language Model Uncertainty
summary: Recent work has sought to quantify large language model uncertainty to
  facilitate model control and modulate user trust. Previous works focus on
  measures of uncertainty that are theoretically grounded or reflect the average
  overt behavior of the model. In this work, we investigate a variety of
  uncertainty measures, in order to identify measures that correlate with human
  group-level uncertainty. We find that Bayesian measures and a variation on
  entropy measures, top-k entropy, tend to agree with human behavior as a
  function of model size. We find that some strong measures decrease in
  human-similarity with model size, but, by multiple linear regression, we find
  that combining multiple uncertainty measures provide comparable human-alignment
  with reduced size-dependency.
paper_authors._id: 67d8f044f8b0e148f60cef0d
paper_authors.name: Kyle Moore
paper_authors._id: 67d8f044f8b0e148f60cef0e
paper_authors.name: Jesse Roberts
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-18T04:02:14.260Z
paper_authors._id: 63c19eb3a0ffa3857eae2efa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c19eb3a0ffa3857eae2efa/pB-CbPa1QFTImiHanNCxU.jpeg
paper_authors.fullname: Jesse Roberts
paper_authors.type: user
paper_authors.user: JesseTNRoberts
paper_authors._id: 67d8f044f8b0e148f60cef0f
paper_authors.name: Daryl Watson
paper_authors._id: 67d8f044f8b0e148f60cef10
paper_authors.name: Pamela Wisniewski
paper_title: Investigating Human-Aligned Large Language Model Uncertainty
paper_summary: Recent work has sought to quantify large language model uncertainty to
  facilitate model control and modulate user trust. Previous works focus on
  measures of uncertainty that are theoretically grounded or reflect the average
  overt behavior of the model. In this work, we investigate a variety of
  uncertainty measures, in order to identify measures that correlate with human
  group-level uncertainty. We find that Bayesian measures and a variation on
  entropy measures, top-k entropy, tend to agree with human behavior as a
  function of model size. We find that some strong measures decrease in
  human-similarity with model size, but, by multiple linear regression, we find
  that combining multiple uncertainty measures provide comparable human-alignment
  with reduced size-dependency.
paper_ai_summary: Various uncertainty measures, particularly Bayesian and top-k entropy, are found to correlate with human uncertainty, and combining multiple measures can mitigate size-dependent human alignment issues.
paper_ai_keywords: Bayesian measures entropy measures top-k entropy multiple linear regression human-alignment size-dependency