title: Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
    Systems for Medical Question Answering
summary: Retrieval-augmented generation (RAG) has emerged as a promising approach to
  enhance the performance of large language models (LLMs) in knowledge-intensive
  tasks such as those from medical domain. However, the sensitive nature of the
  medical domain necessitates a completely accurate and trustworthy system. While
  existing RAG benchmarks primarily focus on the standard retrieve-answer
  setting, they overlook many practical scenarios that measure crucial aspects of
  a reliable medical system. This paper addresses this gap by providing a
  comprehensive evaluation framework for medical question-answering (QA) systems
  in a RAG setting for these situations, including sufficiency, integration, and
  robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
  (MedRGB) that provides various supplementary elements to four medical QA
  datasets for testing LLMs' ability to handle these specific scenarios.
  Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
  commercial LLMs and open-source models across multiple retrieval conditions.
  Our experimental results reveals current models' limited ability to handle
  noise and misinformation in the retrieved documents. We further analyze the
  LLMs' reasoning processes to provides valuable insights and future directions
  for developing RAG systems in this critical medical domain.
paper_authors._id: 673c338be17a2c7a97e8b8c3
paper_authors.name: Nghia Trung Ngo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T14:21:03.206Z
paper_authors._id: 658a8208c4b2004663d82daf
paper_authors.avatarUrl: /avatars/af6bcee06aec82602c2b931f79c008e7.svg
paper_authors.fullname: Nghia Trung Ngo
paper_authors.type: user
paper_authors.user: ntnghia1811
paper_authors._id: 673c338be17a2c7a97e8b8c4
paper_authors.name: Chien Van Nguyen
paper_authors._id: 673c338be17a2c7a97e8b8c5
paper_authors.name: Franck Dernoncourt
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-19T06:43:48.817Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 673c338be17a2c7a97e8b8c6
paper_authors.name: Thien Huu Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T14:21:15.211Z
paper_authors._id: 64804fad8c6a3b8f11f73912
paper_authors.avatarUrl: /avatars/61e37a91d4bba35fda9bf52aadd87745.svg
paper_authors.fullname: Thien Huu Nguyen
paper_authors.type: user
paper_authors.user: anoperson
paper_title: Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
    Systems for Medical Question Answering
paper_summary: Retrieval-augmented generation (RAG) has emerged as a promising approach to
  enhance the performance of large language models (LLMs) in knowledge-intensive
  tasks such as those from medical domain. However, the sensitive nature of the
  medical domain necessitates a completely accurate and trustworthy system. While
  existing RAG benchmarks primarily focus on the standard retrieve-answer
  setting, they overlook many practical scenarios that measure crucial aspects of
  a reliable medical system. This paper addresses this gap by providing a
  comprehensive evaluation framework for medical question-answering (QA) systems
  in a RAG setting for these situations, including sufficiency, integration, and
  robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
  (MedRGB) that provides various supplementary elements to four medical QA
  datasets for testing LLMs' ability to handle these specific scenarios.
  Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
  commercial LLMs and open-source models across multiple retrieval conditions.
  Our experimental results reveals current models' limited ability to handle
  noise and misinformation in the retrieved documents. We further analyze the
  LLMs' reasoning processes to provides valuable insights and future directions
  for developing RAG systems in this critical medical domain.
paper_ai_keywords: Retrieval-augmented generation (RAG) large language models (LLMs) medical domain standard retrieve-answer setting medical question-answering (QA) Systems Medical Retrieval-Augmented Generation Benchmark (MedRGB) medical QA datasets state-of-the-art commercial LLMs open-source models noise misinformation LLMs' reasoning processes