title: MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation
    of LLM Hallucinations
summary: Large Language Models (LLMs) have inherent limitations of faithfulness and
  factuality, commonly referred to as hallucinations. Several benchmarks have
  been developed that provide a test bed for factuality evaluation within the
  context of English-centric datasets, while relying on supplementary informative
  context like web links or text passages but ignoring the available structured
  factual resources. To this end, Knowledge Graphs (KGs) have been identified as
  a useful aid for hallucination mitigation, as they provide a structured way to
  represent the facts about entities and their relations with minimal linguistic
  overhead. We bridge the lack of KG paths and multilinguality for factual
  language modeling within the existing hallucination evaluation benchmarks and
  propose a KG-based multilingual, multihop benchmark called MultiHal
  framed for generative text evaluation. As part of our data collection pipeline,
  we mined 140k KG-paths from open-domain KGs, from which we pruned noisy
  KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation
  shows an absolute scale increase by approximately 0.12 to 0.36 points for the
  semantic similarity score in KG-RAG over vanilla QA across multiple languages
  and multiple models, demonstrating the potential of KG integration. We
  anticipate MultiHal will foster future research towards several graph-based
  hallucination mitigation and fact-checking tasks.
paper_authors._id: 682de7de6bc533fbfe13061a
paper_authors.name: Ernests Lavrinovics
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T08:47:22.712Z
paper_authors._id: 638f0f9ab0525fa370479467
paper_authors.avatarUrl: /avatars/0ee239721747bf0b5d976bfceb393707.svg
paper_authors.fullname: Ernests Lavrinovics
paper_authors.type: user
paper_authors.user: ernlavr
paper_authors._id: 682de7de6bc533fbfe13061b
paper_authors.name: Russa Biswas
paper_authors._id: 682de7de6bc533fbfe13061c
paper_authors.name: Katja Hose
paper_authors._id: 682de7de6bc533fbfe13061d
paper_authors.name: Johannes Bjerva
paper_title: MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation
    of LLM Hallucinations
paper_summary: Large Language Models (LLMs) have inherent limitations of faithfulness and
  factuality, commonly referred to as hallucinations. Several benchmarks have
  been developed that provide a test bed for factuality evaluation within the
  context of English-centric datasets, while relying on supplementary informative
  context like web links or text passages but ignoring the available structured
  factual resources. To this end, Knowledge Graphs (KGs) have been identified as
  a useful aid for hallucination mitigation, as they provide a structured way to
  represent the facts about entities and their relations with minimal linguistic
  overhead. We bridge the lack of KG paths and multilinguality for factual
  language modeling within the existing hallucination evaluation benchmarks and
  propose a KG-based multilingual, multihop benchmark called MultiHal
  framed for generative text evaluation. As part of our data collection pipeline,
  we mined 140k KG-paths from open-domain KGs, from which we pruned noisy
  KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation
  shows an absolute scale increase by approximately 0.12 to 0.36 points for the
  semantic similarity score in KG-RAG over vanilla QA across multiple languages
  and multiple models, demonstrating the potential of KG integration. We
  anticipate MultiHal will foster future research towards several graph-based
  hallucination mitigation and fact-checking tasks.
paper_ai_summary: A multilingual, multihop benchmark using knowledge graphs for evaluating and mitigating hallucinations in large language models.
paper_ai_keywords: Large Language Models hallucinations Knowledge Graphs multilingual multihop benchmark MultiHal semantic similarity score KG-RAG knowledge graph paths graph-based hallucination mitigation fact-checking tasks