title: How Much Do LLMs Hallucinate across Languages? On Multilingual
    Estimation of LLM Hallucination in the Wild
summary: In the age of misinformation, hallucination -- the tendency of Large Language
  Models (LLMs) to generate non-factual or unfaithful responses -- represents the
  main risk for their global utility. Despite LLMs becoming increasingly
  multilingual, the vast majority of research on detecting and quantifying LLM
  hallucination are (a) English-centric and (b) focus on machine translation (MT)
  and summarization, tasks that are less common ``in the wild'' than open
  information seeking. In contrast, we aim to quantify the extent of LLM
  hallucination across languages in knowledge-intensive long-form question
  answering. To this end, we train a multilingual hallucination detection model
  and conduct a large-scale study across 30 languages and 6 open-source LLM
  families. We start from an English hallucination detection dataset and rely on
  MT to generate (noisy) training data in other languages. We also manually
  annotate gold data for five high-resource languages; we then demonstrate, for
  these languages, that the estimates of hallucination rates are similar between
  silver (LLM-generated) and gold test sets, validating the use of silver data
  for estimating hallucination rates for other languages. For the final rates
  estimation, we build a knowledge-intensive QA dataset for 30 languages with
  LLM-generated prompts and Wikipedia articles as references. We find that, while
  LLMs generate longer responses with more hallucinated tokens for
  higher-resource languages, there is no correlation between length-normalized
  hallucination rates of languages and their digital representation. Further, we
  find that smaller LLMs exhibit larger hallucination rates than larger models.
paper_authors._id: 67b597146e53744c2a39e335
paper_authors.name: Saad Obaid ul Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:53:18.182Z
paper_authors._id: 6182910a68444be3259d8b67
paper_authors.avatarUrl: /avatars/b47b0609f61b1192a1337fd7c9f8a75b.svg
paper_authors.fullname: Saad Obaid ul Islam
paper_authors.type: user
paper_authors.user: saadob12
paper_authors._id: 67b597146e53744c2a39e336
paper_authors.name: Anne Lauscher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:05:49.990Z
paper_authors._id: 626c02e7703f3b27dd590896
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
paper_authors.fullname: Anne Lauscher
paper_authors.type: user
paper_authors.user: anlausch
paper_authors._id: 67b597146e53744c2a39e337
paper_authors.name: Goran Glavaš
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:05:56.804Z
paper_authors._id: 6335af67a09fc16c7e7b4879
paper_authors.avatarUrl: /avatars/047fad65ceb33203f97064d6a92ecdc1.svg
paper_authors.fullname: Goran Glavaš
paper_authors.type: user
paper_authors.user: gg42554
paper_title: How Much Do LLMs Hallucinate across Languages? On Multilingual
    Estimation of LLM Hallucination in the Wild
paper_summary: In the age of misinformation, hallucination -- the tendency of Large Language
  Models (LLMs) to generate non-factual or unfaithful responses -- represents the
  main risk for their global utility. Despite LLMs becoming increasingly
  multilingual, the vast majority of research on detecting and quantifying LLM
  hallucination are (a) English-centric and (b) focus on machine translation (MT)
  and summarization, tasks that are less common ``in the wild'' than open
  information seeking. In contrast, we aim to quantify the extent of LLM
  hallucination across languages in knowledge-intensive long-form question
  answering. To this end, we train a multilingual hallucination detection model
  and conduct a large-scale study across 30 languages and 6 open-source LLM
  families. We start from an English hallucination detection dataset and rely on
  MT to generate (noisy) training data in other languages. We also manually
  annotate gold data for five high-resource languages; we then demonstrate, for
  these languages, that the estimates of hallucination rates are similar between
  silver (LLM-generated) and gold test sets, validating the use of silver data
  for estimating hallucination rates for other languages. For the final rates
  estimation, we build a knowledge-intensive QA dataset for 30 languages with
  LLM-generated prompts and Wikipedia articles as references. We find that, while
  LLMs generate longer responses with more hallucinated tokens for
  higher-resource languages, there is no correlation between length-normalized
  hallucination rates of languages and their digital representation. Further, we
  find that smaller LLMs exhibit larger hallucination rates than larger models.
paper_ai_keywords: Large Language Models (LLMs) hallucination machine translation (MT) summarization knowledge-intensive long-form question answering hallucination detection model multilingual hallucination detection model English-centric open information seeking manual annotation gold data silver data knowledge-intensive QA dataset LLM-generated prompts Wikipedia articles hallucinated tokens length-normalized hallucination rates