title: HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
summary: Despite their impressive ability to generate high-quality and fluent text,
  generative large language models (LLMs) also produce hallucinations: statements
  that are misaligned with established world knowledge or provided input context.
  However, measuring hallucination can be challenging, as having humans verify
  model generations on-the-fly is both expensive and time-consuming. In this
  work, we release HALoGEN, a comprehensive hallucination benchmark consisting
  of: (1) 10,923 prompts for generative models spanning nine domains including
  programming, scientific attribution, and summarization, and (2) automatic
  high-precision verifiers for each use case that decompose LLM generations into
  atomic units, and verify each unit against a high-quality knowledge source. We
  use this framework to evaluate ~150,000 generations from 14 language models,
  finding that even the best-performing models are riddled with hallucinations
  (sometimes up to 86% of generated atomic facts depending on the domain). We
  further define a novel error classification for LLM hallucinations based on
  whether they likely stem from incorrect recollection of training data (Type A
  errors), or incorrect knowledge in training data (Type B errors), or are
  fabrication (Type C errors). We hope our framework provides a foundation to
  enable the principled study of why generative models hallucinate, and advances
  the development of trustworthy large language models.
paper_authors._id: 678731bbe24feaaa753768b5
paper_authors.name: Abhilasha Ravichander
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T17:13:38.972Z
paper_authors._id: 6349886c429608888c42319a
paper_authors.avatarUrl: /avatars/f84b5fe8b76172878274754e3399d6ec.svg
paper_authors.fullname: Abhilasha Ravichander
paper_authors.type: user
paper_authors.user: lasha-nlp
paper_authors._id: 678731bbe24feaaa753768b6
paper_authors.name: Shrusti Ghela
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:06:55.911Z
paper_authors._id: 65020c3506a2c356286c1c47
paper_authors.avatarUrl: /avatars/d445d361430bd205d858c2e9c78127bf.svg
paper_authors.fullname: Shrusti Ghela
paper_authors.type: user
paper_authors.user: shrustighela
paper_authors._id: 678731bbe24feaaa753768b7
paper_authors.name: David Wadden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:06:50.718Z
paper_authors._id: 5ff4e2a1463be69ae4bd42bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
paper_authors.fullname: David Wadden
paper_authors.type: user
paper_authors.user: dwadden
paper_authors._id: 678731bbe24feaaa753768b8
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:06:44.995Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
paper_summary: Despite their impressive ability to generate high-quality and fluent text,
  generative large language models (LLMs) also produce hallucinations: statements
  that are misaligned with established world knowledge or provided input context.
  However, measuring hallucination can be challenging, as having humans verify
  model generations on-the-fly is both expensive and time-consuming. In this
  work, we release HALoGEN, a comprehensive hallucination benchmark consisting
  of: (1) 10,923 prompts for generative models spanning nine domains including
  programming, scientific attribution, and summarization, and (2) automatic
  high-precision verifiers for each use case that decompose LLM generations into
  atomic units, and verify each unit against a high-quality knowledge source. We
  use this framework to evaluate ~150,000 generations from 14 language models,
  finding that even the best-performing models are riddled with hallucinations
  (sometimes up to 86% of generated atomic facts depending on the domain). We
  further define a novel error classification for LLM hallucinations based on
  whether they likely stem from incorrect recollection of training data (Type A
  errors), or incorrect knowledge in training data (Type B errors), or are
  fabrication (Type C errors). We hope our framework provides a foundation to
  enable the principled study of why generative models hallucinate, and advances
  the development of trustworthy large language models.
paper_ai_summary: HALoGEN is a benchmark for identifying and categorizing hallucinations in generative language models by verifying atomic units against knowledge sources.
paper_ai_keywords: generative large language models (LLMs) hallucinations HALoGEN automatic verifiers atomic units knowledge source Type A errors Type B errors Type C errors