title: RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented
    Generation Systems
summary: Retrieval-Augmented Generation (RAG) enhances recency and factuality in
  answers. However, existing evaluations rarely test how well these systems cope
  with real-world noise, conflicting between internal and external retrieved
  contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
  Evaluation (RARE), a unified framework and large-scale benchmark that jointly
  stress-tests query and document perturbations over dynamic, time-sensitive
  corpora. One of the central features of RARE is a knowledge-graph-driven
  synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
  relations from the customized corpus and generates multi-level question sets
  without manual intervention. Leveraging this pipeline, we construct a dataset
  (RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
  policy documents and 48,322 questions whose distribution evolves as the
  underlying sources change. To quantify resilience, we formalize
  retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
  ability to remain correct or recover when queries, documents, or real-world
  retrieval results are systematically altered. Our results show that RAG systems
  exhibit surprising vulnerability to perturbations, with document robustness
  consistently being the weakest point regardless of generator size or
  architecture. RAG systems consistently show lower robustness on multi-hop
  queries than single-hop queries across all domains.
paper_authors._id: 683f23ffc7f66ac1b46b2e49
paper_authors.name: Yixiao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:01:28.771Z
paper_authors._id: 64992b16406dbbb2c592634a
paper_authors.avatarUrl: /avatars/7d2e8f10f948db1974ca2d73e1834742.svg
paper_authors.fullname: Yixiao Zeng
paper_authors.type: user
paper_authors.user: Rabinovich
paper_authors._id: 683f23ffc7f66ac1b46b2e4a
paper_authors.name: Tianyu Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:01:25.943Z
paper_authors._id: 66cf5e9fda5d6bbe4d4755f3
paper_authors.avatarUrl: /avatars/407698973be8e12ef4fdf3acddd2705f.svg
paper_authors.fullname: Tiya Cao
paper_authors.type: user
paper_authors.user: TiyaC
paper_authors._id: 683f23ffc7f66ac1b46b2e4b
paper_authors.name: Danqing Wang
paper_authors._id: 683f23ffc7f66ac1b46b2e4c
paper_authors.name: Xinran Zhao
paper_authors._id: 683f23ffc7f66ac1b46b2e4d
paper_authors.name: Zimeng Qiu
paper_authors._id: 683f23ffc7f66ac1b46b2e4e
paper_authors.name: Morteza Ziyadi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-03T16:37:16.891Z
paper_authors._id: 6462db3052193f295764f95f
paper_authors.avatarUrl: /avatars/fded932b65d6f7f2773fb2faecde122a.svg
paper_authors.fullname: Morteza Ziyadi
paper_authors.type: user
paper_authors.user: mziyadi
paper_authors._id: 683f23ffc7f66ac1b46b2e4f
paper_authors.name: Tongshuang Wu
paper_authors._id: 683f23ffc7f66ac1b46b2e50
paper_authors.name: Lei Li
paper_title: RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented
    Generation Systems
paper_summary: Retrieval-Augmented Generation (RAG) enhances recency and factuality in
  answers. However, existing evaluations rarely test how well these systems cope
  with real-world noise, conflicting between internal and external retrieved
  contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
  Evaluation (RARE), a unified framework and large-scale benchmark that jointly
  stress-tests query and document perturbations over dynamic, time-sensitive
  corpora. One of the central features of RARE is a knowledge-graph-driven
  synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
  relations from the customized corpus and generates multi-level question sets
  without manual intervention. Leveraging this pipeline, we construct a dataset
  (RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
  policy documents and 48,322 questions whose distribution evolves as the
  underlying sources change. To quantify resilience, we formalize
  retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
  ability to remain correct or recover when queries, documents, or real-world
  retrieval results are systematically altered. Our results show that RAG systems
  exhibit surprising vulnerability to perturbations, with document robustness
  consistently being the weakest point regardless of generator size or
  architecture. RAG systems consistently show lower robustness on multi-hop
  queries than single-hop queries across all domains.
paper_ai_summary: RARE evaluates the robustness of Retrieval-Augmented Generation (RAG) systems against real-world noise, context conflicts, and time-sensitive data with a knowledge-graph-driven benchmark.
paper_ai_keywords: Retrieval-Augmented Generation RAG Retrieval-Aware Robustness Evaluation RARE query perturbations document perturbations dynamic corpora knowledge-graph-driven synthesis pipeline RARE-Get RARE-Set retrieval-conditioned robustness metrics RARE-Met multi-hop queries single-hop queries