CRAG -- Comprehensive RAG Benchmark
Retrieval-Augmented Generation (RAG) has recently emerged as a promising
solution to alleviate Large Language Model (LLM)'s deficiency in lack of
knowledge. Existing RAG datasets, however, do not adequately represent the
diverse and dynamic nature of real-world Question Answering (QA) tasks. To
bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual
question answering benchmark of 4,409 question-answer pairs and mock APIs to
simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a
diverse array of questions across five domains and eight question categories,
reflecting varied entity popularity from popular to long-tail, and temporal
dynamisms ranging from years to seconds. Our evaluation on this benchmark
highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve
<=34% accuracy on CRAG, adding RAG in a straightforward manner improves the
accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63%
questions without any hallucination. CRAG also reveals much lower accuracy in
answering questions regarding facts with higher dynamism, lower popularity, or
higher complexity, suggesting future research directions. The CRAG benchmark
laid the groundwork for a KDD Cup 2024 challenge, attracting thousands of
participants and submissions within the first 50 days of the competition. We
commit to maintaining CRAG to serve research communities in advancing RAG
solutions and general QA solutions.
666661ce762f3a6a5515146a
Xiao Yang
claimed_verified
2024-06-11T06:59:18.922Z
633490173fbd80d9992a68a5
/avatars/917ecf03128bdcb45219e65348dee323.svg
Xiao Yang
user
xiao-yang-hf
666661ce762f3a6a5515146b
Kai Sun
666661ce762f3a6a5515146c
Hao Xin
666661ce762f3a6a5515146d
Yushi Sun
666661ce762f3a6a5515146e
Nikita Bhalla
admin_assigned
2024-06-10T10:25:50.280Z
625e0f370bec31f086e3f97a
/avatars/2c624695a2117c0b4fc12ff7ad77df07.svg
Nikita Bhalla
user
nikitabhalla
666661ce762f3a6a5515146f
Xiangsen Chen
666661ce762f3a6a55151470
Sajal Choudhary
claimed_verified
2024-09-27T07:00:03.544Z
666781700209de0f572e339f
/avatars/239a233e1ae016edae9120d75e1ba266.svg
Sajal
user
sajalc
666661ce762f3a6a55151471
Rongze Daniel Gui
666661ce762f3a6a55151472
Ziran Will Jiang
666661ce762f3a6a55151473
Ziyu Jiang
claimed_verified
2024-06-10T17:09:16.584Z
6561e2b2280cd6b710a0e70d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bqHpX8FRVIq8EF7UrwzjP.jpeg
Williams JIANG
user
Williams07
666661ce762f3a6a55151474
Lingkun Kong
admin_assigned
2024-06-10T10:29:28.716Z
644553391cfc9ae6bb3d5b34
/avatars/6bd62c1b8afccba97b44f2f005138f86.svg
lingkun
user
konglingkun
666661ce762f3a6a55151475
Brian Moran
666661ce762f3a6a55151476
Jiaqi Wang
claimed_verified
2024-06-14T07:24:37.997Z
6614df3367a63c45c919c667
/avatars/f63810aae49f1afe29ef5948bf4dd4bb.svg
Jiaqi Wang
user
jackiewang92
666661ce762f3a6a55151477
Yifan Ethan Xu
666661ce762f3a6a55151478
An Yan
666661ce762f3a6a55151479
Chenyu Yang
666661ce762f3a6a5515147a
Eting Yuan
666661ce762f3a6a5515147b
Hanwen Zha
666661ce762f3a6a5515147c
Nan Tang
666661ce762f3a6a5515147d
Lei Chen
666661ce762f3a6a5515147e
Nicolas Scheffer
666661ce762f3a6a5515147f
Yue Liu
claimed_verified
2024-06-10T17:09:18.550Z
66671b27ad9cd05ea74202a5
/avatars/a6fa26c173f3e5b0748e5fb007c3105e.svg
Yue Liu
user
liu-yue
666661ce762f3a6a55151480
Nirav Shah
666661ce762f3a6a55151481
Rakesh Wanga
666661ce762f3a6a55151482
Anuj Kumar
666661ce762f3a6a55151483
Wen-tau Yih
666661ce762f3a6a55151484
Xin Luna Dong
CRAG -- Comprehensive RAG Benchmark
Retrieval-Augmented Generation (RAG) has recently emerged as a promising
solution to alleviate Large Language Model (LLM)'s deficiency in lack of
knowledge. Existing RAG datasets, however, do not adequately represent the
diverse and dynamic nature of real-world Question Answering (QA) tasks. To
bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual
question answering benchmark of 4,409 question-answer pairs and mock APIs to
simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a
diverse array of questions across five domains and eight question categories,
reflecting varied entity popularity from popular to long-tail, and temporal
dynamisms ranging from years to seconds. Our evaluation on this benchmark
highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve
<=34% accuracy on CRAG, adding RAG in a straightforward manner improves the
accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63%
questions without any hallucination. CRAG also reveals much lower accuracy in
answering questions regarding facts with higher dynamism, lower popularity, or
higher complexity, suggesting future research directions. The CRAG benchmark
laid the groundwork for a KDD Cup 2024 challenge, attracting thousands of
participants and submissions within the first 50 days of the competition. We
commit to maintaining CRAG to serve research communities in advancing RAG
solutions and general QA solutions.
The Comprehensive RAG Benchmark (CRAG) evaluates RAG solutions with diverse QA tasks, revealing gaps in LLM accuracy and future research directions.
Retrieval-Augmented Generation RAG Large Language Model LLM Question Answering QA Comprehensive RAG Benchmark CRAG Knowledge Graph KDD Cup