title: CRAG -- Comprehensive RAG Benchmark
summary: Retrieval-Augmented Generation (RAG) has recently emerged as a promising
  solution to alleviate Large Language Model (LLM)'s deficiency in lack of
  knowledge. Existing RAG datasets, however, do not adequately represent the
  diverse and dynamic nature of real-world Question Answering (QA) tasks. To
  bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual
  question answering benchmark of 4,409 question-answer pairs and mock APIs to
  simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a
  diverse array of questions across five domains and eight question categories,
  reflecting varied entity popularity from popular to long-tail, and temporal
  dynamisms ranging from years to seconds. Our evaluation on this benchmark
  highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve
  <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the
  accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63%
  questions without any hallucination. CRAG also reveals much lower accuracy in
  answering questions regarding facts with higher dynamism, lower popularity, or
  higher complexity, suggesting future research directions. The CRAG benchmark
  laid the groundwork for a KDD Cup 2024 challenge, attracting thousands of
  participants and submissions within the first 50 days of the competition. We
  commit to maintaining CRAG to serve research communities in advancing RAG
  solutions and general QA solutions.
paper_authors._id: 666661ce762f3a6a5515146a
paper_authors.name: Xiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T06:59:18.922Z
paper_authors._id: 633490173fbd80d9992a68a5
paper_authors.avatarUrl: /avatars/917ecf03128bdcb45219e65348dee323.svg
paper_authors.fullname: Xiao Yang
paper_authors.type: user
paper_authors.user: xiao-yang-hf
paper_authors._id: 666661ce762f3a6a5515146b
paper_authors.name: Kai Sun
paper_authors._id: 666661ce762f3a6a5515146c
paper_authors.name: Hao Xin
paper_authors._id: 666661ce762f3a6a5515146d
paper_authors.name: Yushi Sun
paper_authors._id: 666661ce762f3a6a5515146e
paper_authors.name: Nikita Bhalla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:25:50.280Z
paper_authors._id: 625e0f370bec31f086e3f97a
paper_authors.avatarUrl: /avatars/2c624695a2117c0b4fc12ff7ad77df07.svg
paper_authors.fullname: Nikita Bhalla
paper_authors.type: user
paper_authors.user: nikitabhalla
paper_authors._id: 666661ce762f3a6a5515146f
paper_authors.name: Xiangsen Chen
paper_authors._id: 666661ce762f3a6a55151470
paper_authors.name: Sajal Choudhary
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T07:00:03.544Z
paper_authors._id: 666781700209de0f572e339f
paper_authors.avatarUrl: /avatars/239a233e1ae016edae9120d75e1ba266.svg
paper_authors.fullname: Sajal
paper_authors.type: user
paper_authors.user: sajalc
paper_authors._id: 666661ce762f3a6a55151471
paper_authors.name: Rongze Daniel Gui
paper_authors._id: 666661ce762f3a6a55151472
paper_authors.name: Ziran Will Jiang
paper_authors._id: 666661ce762f3a6a55151473
paper_authors.name: Ziyu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-10T17:09:16.584Z
paper_authors._id: 6561e2b2280cd6b710a0e70d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bqHpX8FRVIq8EF7UrwzjP.jpeg
paper_authors.fullname: Williams JIANG
paper_authors.type: user
paper_authors.user: Williams07
paper_authors._id: 666661ce762f3a6a55151474
paper_authors.name: Lingkun Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:29:28.716Z
paper_authors._id: 644553391cfc9ae6bb3d5b34
paper_authors.avatarUrl: /avatars/6bd62c1b8afccba97b44f2f005138f86.svg
paper_authors.fullname: lingkun
paper_authors.type: user
paper_authors.user: konglingkun
paper_authors._id: 666661ce762f3a6a55151475
paper_authors.name: Brian Moran
paper_authors._id: 666661ce762f3a6a55151476
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:24:37.997Z
paper_authors._id: 6614df3367a63c45c919c667
paper_authors.avatarUrl: /avatars/f63810aae49f1afe29ef5948bf4dd4bb.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: jackiewang92
paper_authors._id: 666661ce762f3a6a55151477
paper_authors.name: Yifan Ethan Xu
paper_authors._id: 666661ce762f3a6a55151478
paper_authors.name: An Yan
paper_authors._id: 666661ce762f3a6a55151479
paper_authors.name: Chenyu Yang
paper_authors._id: 666661ce762f3a6a5515147a
paper_authors.name: Eting Yuan
paper_authors._id: 666661ce762f3a6a5515147b
paper_authors.name: Hanwen Zha
paper_authors._id: 666661ce762f3a6a5515147c
paper_authors.name: Nan Tang
paper_authors._id: 666661ce762f3a6a5515147d
paper_authors.name: Lei Chen
paper_authors._id: 666661ce762f3a6a5515147e
paper_authors.name: Nicolas Scheffer
paper_authors._id: 666661ce762f3a6a5515147f
paper_authors.name: Yue Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-10T17:09:18.550Z
paper_authors._id: 66671b27ad9cd05ea74202a5
paper_authors.avatarUrl: /avatars/a6fa26c173f3e5b0748e5fb007c3105e.svg
paper_authors.fullname: Yue Liu
paper_authors.type: user
paper_authors.user: liu-yue
paper_authors._id: 666661ce762f3a6a55151480
paper_authors.name: Nirav Shah
paper_authors._id: 666661ce762f3a6a55151481
paper_authors.name: Rakesh Wanga
paper_authors._id: 666661ce762f3a6a55151482
paper_authors.name: Anuj Kumar
paper_authors._id: 666661ce762f3a6a55151483
paper_authors.name: Wen-tau Yih
paper_authors._id: 666661ce762f3a6a55151484
paper_authors.name: Xin Luna Dong
paper_title: CRAG -- Comprehensive RAG Benchmark
paper_summary: Retrieval-Augmented Generation (RAG) has recently emerged as a promising
  solution to alleviate Large Language Model (LLM)'s deficiency in lack of
  knowledge. Existing RAG datasets, however, do not adequately represent the
  diverse and dynamic nature of real-world Question Answering (QA) tasks. To
  bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual
  question answering benchmark of 4,409 question-answer pairs and mock APIs to
  simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a
  diverse array of questions across five domains and eight question categories,
  reflecting varied entity popularity from popular to long-tail, and temporal
  dynamisms ranging from years to seconds. Our evaluation on this benchmark
  highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve
  <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the
  accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63%
  questions without any hallucination. CRAG also reveals much lower accuracy in
  answering questions regarding facts with higher dynamism, lower popularity, or
  higher complexity, suggesting future research directions. The CRAG benchmark
  laid the groundwork for a KDD Cup 2024 challenge, attracting thousands of
  participants and submissions within the first 50 days of the competition. We
  commit to maintaining CRAG to serve research communities in advancing RAG
  solutions and general QA solutions.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) Large Language Model (LLM) Question Answering (QA) Knowledge Graph (KG) Comprehensive RAG Benchmark (CRAG) factual question answering mock APIs entity popularity temporal dynamisms accuracy hallucination KDD Cup 2024