How Much Do LLMs Hallucinate across Languages? On Multilingual
  Estimation of LLM Hallucination in the Wild
In the age of misinformation, hallucination -- the tendency of Large Language
Models (LLMs) to generate non-factual or unfaithful responses -- represents the
main risk for their global utility. Despite LLMs becoming increasingly
multilingual, the vast majority of research on detecting and quantifying LLM
hallucination are (a) English-centric and (b) focus on machine translation (MT)
and summarization, tasks that are less common ``in the wild'' than open
information seeking. In contrast, we aim to quantify the extent of LLM
hallucination across languages in knowledge-intensive long-form question
answering. To this end, we train a multilingual hallucination detection model
and conduct a large-scale study across 30 languages and 6 open-source LLM
families. We start from an English hallucination detection dataset and rely on
MT to generate (noisy) training data in other languages. We also manually
annotate gold data for five high-resource languages; we then demonstrate, for
these languages, that the estimates of hallucination rates are similar between
silver (LLM-generated) and gold test sets, validating the use of silver data
for estimating hallucination rates for other languages. For the final rates
estimation, we build a knowledge-intensive QA dataset for 30 languages with
LLM-generated prompts and Wikipedia articles as references. We find that, while
LLMs generate longer responses with more hallucinated tokens for
higher-resource languages, there is no correlation between length-normalized
hallucination rates of languages and their digital representation. Further, we
find that smaller LLMs exhibit larger hallucination rates than larger models.
67b597146e53744c2a39e335
Saad Obaid ul Islam
claimed_verified
2025-02-20T15:53:18.182Z
6182910a68444be3259d8b67
/avatars/b47b0609f61b1192a1337fd7c9f8a75b.svg
Saad Obaid ul Islam
user
saadob12
67b597146e53744c2a39e336
Anne Lauscher
admin_assigned
2025-02-21T15:05:49.990Z
626c02e7703f3b27dd590896
https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
Anne Lauscher
user
anlausch
67b597146e53744c2a39e337
Goran Glavaš
admin_assigned
2025-02-21T15:05:56.804Z
6335af67a09fc16c7e7b4879
/avatars/047fad65ceb33203f97064d6a92ecdc1.svg
Goran Glavaš
user
gg42554
How Much Do LLMs Hallucinate across Languages? On Multilingual
  Estimation of LLM Hallucination in the Wild
In the age of misinformation, hallucination -- the tendency of Large Language
Models (LLMs) to generate non-factual or unfaithful responses -- represents the
main risk for their global utility. Despite LLMs becoming increasingly
multilingual, the vast majority of research on detecting and quantifying LLM
hallucination are (a) English-centric and (b) focus on machine translation (MT)
and summarization, tasks that are less common ``in the wild'' than open
information seeking. In contrast, we aim to quantify the extent of LLM
hallucination across languages in knowledge-intensive long-form question
answering. To this end, we train a multilingual hallucination detection model
and conduct a large-scale study across 30 languages and 6 open-source LLM
families. We start from an English hallucination detection dataset and rely on
MT to generate (noisy) training data in other languages. We also manually
annotate gold data for five high-resource languages; we then demonstrate, for
these languages, that the estimates of hallucination rates are similar between
silver (LLM-generated) and gold test sets, validating the use of silver data
for estimating hallucination rates for other languages. For the final rates
estimation, we build a knowledge-intensive QA dataset for 30 languages with
LLM-generated prompts and Wikipedia articles as references. We find that, while
LLMs generate longer responses with more hallucinated tokens for
higher-resource languages, there is no correlation between length-normalized
hallucination rates of languages and their digital representation. Further, we
find that smaller LLMs exhibit larger hallucination rates than larger models.
This study quantifies and compares hallucination rates in multilingual large language models across knowledge-intensive long-form question answering tasks.
Large Language Models hallucination machine translation summarization multilingual hallucination detection knowledge-intensive QA hallucination rates LLM-generated prompts Wikipedia articles