Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented
  Generation
Large Language Models (LLMs) have demonstrated significant performance
improvements across various cognitive tasks. An emerging application is using
LLMs to enhance retrieval-augmented generation (RAG) capabilities. These
systems require LLMs to understand user queries, retrieve relevant information,
and synthesize coherent and accurate responses. Given the increasing real-world
deployment of such systems, comprehensive evaluation becomes crucial. To this
end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set),
a high-quality evaluation dataset designed to test LLMs' ability to provide
factual responses, assess retrieval capabilities, and evaluate the reasoning
required to generate final answers. While previous work has provided datasets
and benchmarks to evaluate these abilities in isolation, FRAMES offers a
unified framework that provides a clearer picture of LLM performance in
end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions
that require the integration of information from multiple sources. We present
baseline results demonstrating that even state-of-the-art LLMs struggle with
this task, achieving 0.40 accuracy with no retrieval. The accuracy is
significantly improved with our proposed multi-step retrieval pipeline,
achieving an accuracy of 0.66 (>50% improvement). We hope our work will help
bridge evaluation gaps and assist in developing more robust and capable RAG
systems.
66ed96efce6e5db9b396f912
Satyapriya Krishna
claimed_verified
2024-09-23T13:48:40.554Z
6186fef1b1085ab638324e7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
Satya
user
skrishna
66ed96efce6e5db9b396f913
Kalpesh Krishna
admin_assigned
2024-09-23T16:30:53.112Z
62b9f1034dde8074b78cc6f2
/avatars/2bde72c6b8baee2b31f318b3010e483a.svg
Kalpesh Krishna
user
kalpeshk2011
66ed96efce6e5db9b396f914
Anhad Mohananey
66ed96efce6e5db9b396f915
Steven Schwarcz
admin_assigned
2024-09-23T16:31:05.703Z
651c3ddd09e70aeed232ff97
/avatars/aa92d8568ced53c932c79c7441d3b5da.svg
Steven Schwarcz
user
OMN
66ed96efce6e5db9b396f916
Adam Stambler
66ed96efce6e5db9b396f917
Shyam Upadhyay
66ed96efce6e5db9b396f918
Manaal Faruqui
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented
  Generation
Large Language Models (LLMs) have demonstrated significant performance
improvements across various cognitive tasks. An emerging application is using
LLMs to enhance retrieval-augmented generation (RAG) capabilities. These
systems require LLMs to understand user queries, retrieve relevant information,
and synthesize coherent and accurate responses. Given the increasing real-world
deployment of such systems, comprehensive evaluation becomes crucial. To this
end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set),
a high-quality evaluation dataset designed to test LLMs' ability to provide
factual responses, assess retrieval capabilities, and evaluate the reasoning
required to generate final answers. While previous work has provided datasets
and benchmarks to evaluate these abilities in isolation, FRAMES offers a
unified framework that provides a clearer picture of LLM performance in
end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions
that require the integration of information from multiple sources. We present
baseline results demonstrating that even state-of-the-art LLMs struggle with
this task, achieving 0.40 accuracy with no retrieval. The accuracy is
significantly improved with our proposed multi-step retrieval pipeline,
achieving an accuracy of 0.66 (>50% improvement). We hope our work will help
bridge evaluation gaps and assist in developing more robust and capable RAG
systems.
A new evaluation dataset called FRAMES assesses LLMs' factuality, retrieval, and reasoning in retrieval-augmented generation (RAG) tasks, demonstrating the need for improved multi-step retrieval pipelines.
Large Language Models retrieval-augmented generation RAG FRAMES factuality retrieval reasoning multi-hop questions accuracy multi-step retrieval pipeline