title: Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-OASIS
summary: After the introduction of Large Language Models (LLMs), there have been
  substantial improvements in the performance of Natural Language Generation
  (NLG) tasks, including Text Summarization and Machine Translation. However,
  LLMs still produce outputs containing hallucinations, that is, content not
  grounded in factual information. Therefore, developing methods to assess the
  factuality of LLMs has become urgent.
    Indeed, resources for factuality evaluation have recently emerged. Although
  challenging, these resources face one or more of the following limitations: (i)
  they are tailored to a specific task or domain; (ii) they are limited in size,
  thereby preventing the training of new factuality evaluators; (iii) they are
  designed for simpler verification tasks, such as claim verification.
    To address these issues, we introduce LLM-Oasis, to the best of our knowledge
  the largest resource for training end-to-end factuality evaluators. LLM-Oasis
  is constructed by extracting claims from Wikipedia, falsifying a subset of
  these claims, and generating pairs of factual and unfactual texts. We then rely
  on human annotators to both validate the quality of our dataset and to create a
  gold standard test set for benchmarking factuality evaluation systems.
    Our experiments demonstrate that LLM-Oasis presents a significant challenge
  for state-of-the-art LLMs, with GPT-4o achieving up to 60% accuracy in our
  proposed end-to-end factuality evaluation task, highlighting its potential to
  drive future research in the field.
paper_authors._id: 674db37adefe7edb9f5e0028
paper_authors.name: Alessandro Scirè
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:58.122Z
paper_authors._id: 65ef2286d08d92badac66e34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ef2286d08d92badac66e34/-FtE-wVpqlbZtNLhEOCc_.jpeg
paper_authors.fullname: Alessandro Scirè
paper_authors.type: user
paper_authors.user: alescire94
paper_authors._id: 674db37adefe7edb9f5e0029
paper_authors.name: Andrei Stefan Bejgu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:59.956Z
paper_authors._id: 633ee9704be90e06da1eb774
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633ee9704be90e06da1eb774/xU6gXNHkBub0hAQb9RRsR.jpeg
paper_authors.fullname: Andrei Stefan Bejgu
paper_authors.type: user
paper_authors.user: andreim14
paper_authors._id: 674db37adefe7edb9f5e002a
paper_authors.name: Simone Tedeschi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:11:53.665Z
paper_authors._id: 61b85aa99ba538c73a7dc78b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b85aa99ba538c73a7dc78b/gWxtQAvOYn7cXgE_nAy0p.jpeg
paper_authors.fullname: Simone Tedeschi
paper_authors.type: user
paper_authors.user: sted97
paper_authors._id: 674db37adefe7edb9f5e002b
paper_authors.name: Karim Ghonim
paper_authors._id: 674db37adefe7edb9f5e002c
paper_authors.name: Federico Martelli
paper_authors._id: 674db37adefe7edb9f5e002d
paper_authors.name: Roberto Navigli
paper_title: Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-OASIS
paper_summary: After the introduction of Large Language Models (LLMs), there have been
  substantial improvements in the performance of Natural Language Generation
  (NLG) tasks, including Text Summarization and Machine Translation. However,
  LLMs still produce outputs containing hallucinations, that is, content not
  grounded in factual information. Therefore, developing methods to assess the
  factuality of LLMs has become urgent.
    Indeed, resources for factuality evaluation have recently emerged. Although
  challenging, these resources face one or more of the following limitations: (i)
  they are tailored to a specific task or domain; (ii) they are limited in size,
  thereby preventing the training of new factuality evaluators; (iii) they are
  designed for simpler verification tasks, such as claim verification.
    To address these issues, we introduce LLM-Oasis, to the best of our knowledge
  the largest resource for training end-to-end factuality evaluators. LLM-Oasis
  is constructed by extracting claims from Wikipedia, falsifying a subset of
  these claims, and generating pairs of factual and unfactual texts. We then rely
  on human annotators to both validate the quality of our dataset and to create a
  gold standard test set for benchmarking factuality evaluation systems.
    Our experiments demonstrate that LLM-Oasis presents a significant challenge
  for state-of-the-art LLMs, with GPT-4o achieving up to 60% accuracy in our
  proposed end-to-end factuality evaluation task, highlighting its potential to
  drive future research in the field.
paper_ai_summary: LLM-Oasis, a comprehensive dataset for factuality evaluation, challenges state-of-the-art LLMs by requiring high accuracy in distinguishing factual from unfactual texts.
paper_ai_keywords: LLMs Natural Language Generation Text Summarization Machine Translation hallucinations factuality evaluation LLM-Oasis claims falsification human annotators benchmarking