title: Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented
    Generation
summary: Large Language Models (LLMs) have demonstrated significant performance
  improvements across various cognitive tasks. An emerging application is using
  LLMs to enhance retrieval-augmented generation (RAG) capabilities. These
  systems require LLMs to understand user queries, retrieve relevant information,
  and synthesize coherent and accurate responses. Given the increasing real-world
  deployment of such systems, comprehensive evaluation becomes crucial. To this
  end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set),
  a high-quality evaluation dataset designed to test LLMs' ability to provide
  factual responses, assess retrieval capabilities, and evaluate the reasoning
  required to generate final answers. While previous work has provided datasets
  and benchmarks to evaluate these abilities in isolation, FRAMES offers a
  unified framework that provides a clearer picture of LLM performance in
  end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions
  that require the integration of information from multiple sources. We present
  baseline results demonstrating that even state-of-the-art LLMs struggle with
  this task, achieving 0.40 accuracy with no retrieval. The accuracy is
  significantly improved with our proposed multi-step retrieval pipeline,
  achieving an accuracy of 0.66 (>50% improvement). We hope our work will help
  bridge evaluation gaps and assist in developing more robust and capable RAG
  systems.
paper_authors._id: 66ed96efce6e5db9b396f912
paper_authors.name: Satyapriya Krishna
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T13:48:40.554Z
paper_authors._id: 6186fef1b1085ab638324e7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
paper_authors.fullname: Satya
paper_authors.type: user
paper_authors.user: skrishna
paper_authors._id: 66ed96efce6e5db9b396f913
paper_authors.name: Kalpesh Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T16:30:53.112Z
paper_authors._id: 62b9f1034dde8074b78cc6f2
paper_authors.avatarUrl: /avatars/2bde72c6b8baee2b31f318b3010e483a.svg
paper_authors.fullname: Kalpesh Krishna
paper_authors.type: user
paper_authors.user: kalpeshk2011
paper_authors._id: 66ed96efce6e5db9b396f914
paper_authors.name: Anhad Mohananey
paper_authors._id: 66ed96efce6e5db9b396f915
paper_authors.name: Steven Schwarcz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T16:31:05.703Z
paper_authors._id: 651c3ddd09e70aeed232ff97
paper_authors.avatarUrl: /avatars/aa92d8568ced53c932c79c7441d3b5da.svg
paper_authors.fullname: Steven Schwarcz
paper_authors.type: user
paper_authors.user: OMN
paper_authors._id: 66ed96efce6e5db9b396f916
paper_authors.name: Adam Stambler
paper_authors._id: 66ed96efce6e5db9b396f917
paper_authors.name: Shyam Upadhyay
paper_authors._id: 66ed96efce6e5db9b396f918
paper_authors.name: Manaal Faruqui
paper_title: Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented
    Generation
paper_summary: Large Language Models (LLMs) have demonstrated significant performance
  improvements across various cognitive tasks. An emerging application is using
  LLMs to enhance retrieval-augmented generation (RAG) capabilities. These
  systems require LLMs to understand user queries, retrieve relevant information,
  and synthesize coherent and accurate responses. Given the increasing real-world
  deployment of such systems, comprehensive evaluation becomes crucial. To this
  end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set),
  a high-quality evaluation dataset designed to test LLMs' ability to provide
  factual responses, assess retrieval capabilities, and evaluate the reasoning
  required to generate final answers. While previous work has provided datasets
  and benchmarks to evaluate these abilities in isolation, FRAMES offers a
  unified framework that provides a clearer picture of LLM performance in
  end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions
  that require the integration of information from multiple sources. We present
  baseline results demonstrating that even state-of-the-art LLMs struggle with
  this task, achieving 0.40 accuracy with no retrieval. The accuracy is
  significantly improved with our proposed multi-step retrieval pipeline,
  achieving an accuracy of 0.66 (>50% improvement). We hope our work will help
  bridge evaluation gaps and assist in developing more robust and capable RAG
  systems.
paper_ai_summary: A new evaluation dataset called FRAMES assesses LLMs' factuality, retrieval, and reasoning in retrieval-augmented generation (RAG) tasks, demonstrating the need for improved multi-step retrieval pipelines.
paper_ai_keywords: Large Language Models retrieval-augmented generation RAG FRAMES factuality retrieval reasoning multi-hop questions accuracy multi-step retrieval pipeline