title: Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
summary: Challenges in the automated evaluation of Retrieval-Augmented Generation
  (RAG) Question-Answering (QA) systems include hallucination problems in
  domain-specific knowledge and the lack of gold standard benchmarks for company
  internal tasks. This results in difficulties in evaluating RAG variations, like
  RAG-Fusion (RAGF), in the context of a product QA task at Infineon
  Technologies. To solve these problems, we propose a comprehensive evaluation
  framework, which leverages Large Language Models (LLMs) to generate large
  datasets of synthetic queries based on real user queries and in-domain
  documents, uses LLM-as-a-judge to rate retrieved documents and answers,
  evaluates the quality of answers, and ranks different variants of
  Retrieval-Augmented Generation (RAG) agents with RAGElo's automated Elo-based
  competition. LLM-as-a-judge rating of a random sample of synthetic queries
  shows a moderate, positive correlation with domain expert scoring in relevance,
  accuracy, completeness, and precision. While RAGF outperformed RAG in Elo
  score, a significance analysis against expert annotations also shows that RAGF
  significantly outperforms RAG in completeness, but underperforms in precision.
  In addition, Infineon's RAGF assistant demonstrated slightly higher performance
  in document relevance based on MRR@5 scores. We find that RAGElo positively
  aligns with the preferences of human annotators, though due caution is still
  required. Finally, RAGF's approach leads to more complete answers based on
  expert annotations and better answers overall based on RAGElo's evaluation
  criteria.
paper_authors._id: 66797e920d9dcc40b50f870a
paper_authors.name: Zackary Rackauckas
paper_authors._id: 66797e920d9dcc40b50f870b
paper_authors.name: Arthur Câmara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:16:09.628Z
paper_authors._id: 61e2062e2f94efc39ac05de2
paper_authors.avatarUrl: /avatars/63fe371369ae3c5911b9d4b31fdb3a5f.svg
paper_authors.fullname: Arthur Barbosa Câmara
paper_authors.type: user
paper_authors.user: ArthurCamara
paper_authors._id: 66797e920d9dcc40b50f870c
paper_authors.name: Jakub Zavrel
paper_title: Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
paper_summary: Challenges in the automated evaluation of Retrieval-Augmented Generation
  (RAG) Question-Answering (QA) systems include hallucination problems in
  domain-specific knowledge and the lack of gold standard benchmarks for company
  internal tasks. This results in difficulties in evaluating RAG variations, like
  RAG-Fusion (RAGF), in the context of a product QA task at Infineon
  Technologies. To solve these problems, we propose a comprehensive evaluation
  framework, which leverages Large Language Models (LLMs) to generate large
  datasets of synthetic queries based on real user queries and in-domain
  documents, uses LLM-as-a-judge to rate retrieved documents and answers,
  evaluates the quality of answers, and ranks different variants of
  Retrieval-Augmented Generation (RAG) agents with RAGElo's automated Elo-based
  competition. LLM-as-a-judge rating of a random sample of synthetic queries
  shows a moderate, positive correlation with domain expert scoring in relevance,
  accuracy, completeness, and precision. While RAGF outperformed RAG in Elo
  score, a significance analysis against expert annotations also shows that RAGF
  significantly outperforms RAG in completeness, but underperforms in precision.
  In addition, Infineon's RAGF assistant demonstrated slightly higher performance
  in document relevance based on MRR@5 scores. We find that RAGElo positively
  aligns with the preferences of human annotators, though due caution is still
  required. Finally, RAGF's approach leads to more complete answers based on
  expert annotations and better answers overall based on RAGElo's evaluation
  criteria.
paper_ai_summary: A new evaluation framework using LLMs and an Elo-based system addresses challenges in automating the evaluation of RAG-based QA systems, showing RAG-Fusion outperforms RAG in completeness but not in precision.
paper_ai_keywords: Retrieval-Augmented Generation Q&A systems hallucination gold standard benchmarks RAG-Fusion Large Language Models synthetic queries LLM-as-a-judge Elo-based competition domain expert scoring relevance accuracy completeness precision MRR@5 RAGElo