MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation
  of LLM Hallucinations
Large Language Models (LLMs) have inherent limitations of faithfulness and
factuality, commonly referred to as hallucinations. Several benchmarks have
been developed that provide a test bed for factuality evaluation within the
context of English-centric datasets, while relying on supplementary informative
context like web links or text passages but ignoring the available structured
factual resources. To this end, Knowledge Graphs (KGs) have been identified as
a useful aid for hallucination mitigation, as they provide a structured way to
represent the facts about entities and their relations with minimal linguistic
overhead. We bridge the lack of KG paths and multilinguality for factual
language modeling within the existing hallucination evaluation benchmarks and
propose a KG-based multilingual, multihop benchmark called MultiHal
framed for generative text evaluation. As part of our data collection pipeline,
we mined 140k KG-paths from open-domain KGs, from which we pruned noisy
KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation
shows an absolute scale increase by approximately 0.12 to 0.36 points for the
semantic similarity score in KG-RAG over vanilla QA across multiple languages
and multiple models, demonstrating the potential of KG integration. We
anticipate MultiHal will foster future research towards several graph-based
hallucination mitigation and fact-checking tasks.
682de7de6bc533fbfe13061a
Ernests Lavrinovics
claimed_verified
2025-05-22T08:47:22.712Z
638f0f9ab0525fa370479467
/avatars/0ee239721747bf0b5d976bfceb393707.svg
Ernests Lavrinovics
user
ernlavr
682de7de6bc533fbfe13061b
Russa Biswas
682de7de6bc533fbfe13061c
Katja Hose
682de7de6bc533fbfe13061d
Johannes Bjerva
MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation
  of LLM Hallucinations
Large Language Models (LLMs) have inherent limitations of faithfulness and
factuality, commonly referred to as hallucinations. Several benchmarks have
been developed that provide a test bed for factuality evaluation within the
context of English-centric datasets, while relying on supplementary informative
context like web links or text passages but ignoring the available structured
factual resources. To this end, Knowledge Graphs (KGs) have been identified as
a useful aid for hallucination mitigation, as they provide a structured way to
represent the facts about entities and their relations with minimal linguistic
overhead. We bridge the lack of KG paths and multilinguality for factual
language modeling within the existing hallucination evaluation benchmarks and
propose a KG-based multilingual, multihop benchmark called MultiHal
framed for generative text evaluation. As part of our data collection pipeline,
we mined 140k KG-paths from open-domain KGs, from which we pruned noisy
KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation
shows an absolute scale increase by approximately 0.12 to 0.36 points for the
semantic similarity score in KG-RAG over vanilla QA across multiple languages
and multiple models, demonstrating the potential of KG integration. We
anticipate MultiHal will foster future research towards several graph-based
hallucination mitigation and fact-checking tasks.
A multilingual, multihop benchmark using knowledge graphs for evaluating and mitigating hallucinations in large language models.
Large Language Models hallucinations Knowledge Graphs multilingual multihop benchmark MultiHal semantic similarity score KG-RAG knowledge graph paths graph-based hallucination mitigation fact-checking tasks