title: Explaining Sources of Uncertainty in Automated Fact-Checking
summary: Understanding sources of a model's uncertainty regarding its predictions is
  crucial for effective human-AI collaboration. Prior work proposes using
  numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
  uncertainty that arises from conflicting evidence, leaving users unable to
  resolve disagreements or rely on the output. We introduce CLUE
  (Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
  first framework to generate natural language explanations of model uncertainty
  by (i) identifying relationships between spans of text that expose
  claim-evidence or inter-evidence conflicts and agreements that drive the
  model's predictive uncertainty in an unsupervised way, and (ii) generating
  explanations via prompting and attention steering that verbalize these critical
  interactions. Across three language models and two fact-checking datasets, we
  show that CLUE produces explanations that are more faithful to the model's
  uncertainty and more consistent with fact-checking decisions than prompting for
  uncertainty explanations without span-interaction guidance. Human evaluators
  judge our explanations to be more helpful, more informative, less redundant,
  and more logically consistent with the input than this baseline. CLUE requires
  no fine-tuning or architectural changes, making it plug-and-play for any
  white-box language model. By explicitly linking uncertainty to evidence
  conflicts, it offers practical support for fact-checking and generalises
  readily to other tasks that require reasoning over complex information.
paper_authors._id: 68370a318cd8af122276627d
paper_authors.name: Jingyi Sun
paper_authors._id: 68370a318cd8af122276627e
paper_authors.name: Greta Warren
paper_authors._id: 68370a318cd8af122276627f
paper_authors.name: Irina Shklovski
paper_authors._id: 68370a318cd8af1222766280
paper_authors.name: Isabelle Augenstein
paper_title: Explaining Sources of Uncertainty in Automated Fact-Checking
paper_summary: Understanding sources of a model's uncertainty regarding its predictions is
  crucial for effective human-AI collaboration. Prior work proposes using
  numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
  uncertainty that arises from conflicting evidence, leaving users unable to
  resolve disagreements or rely on the output. We introduce CLUE
  (Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
  first framework to generate natural language explanations of model uncertainty
  by (i) identifying relationships between spans of text that expose
  claim-evidence or inter-evidence conflicts and agreements that drive the
  model's predictive uncertainty in an unsupervised way, and (ii) generating
  explanations via prompting and attention steering that verbalize these critical
  interactions. Across three language models and two fact-checking datasets, we
  show that CLUE produces explanations that are more faithful to the model's
  uncertainty and more consistent with fact-checking decisions than prompting for
  uncertainty explanations without span-interaction guidance. Human evaluators
  judge our explanations to be more helpful, more informative, less redundant,
  and more logically consistent with the input than this baseline. CLUE requires
  no fine-tuning or architectural changes, making it plug-and-play for any
  white-box language model. By explicitly linking uncertainty to evidence
  conflicts, it offers practical support for fact-checking and generalises
  readily to other tasks that require reasoning over complex information.
paper_ai_summary: CLUE generates natural language explanations for a language model's uncertainty by identifying and explaining conflicts and agreements in text spans, enhancing the clarity and helpfulness of explanations in tasks like fact-checking.
paper_ai_keywords: knowledge graphs span-interaction guidance claim-evidence inter-evidence conflicts agreements predictive uncertainty prompting attention steering fact-checking datasets white-box language models