title: GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question
    Answering
summary: Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use
  Large Language Models (LLMs) alongside private and up-to-date knowledge bases.
  In this work, we address the challenges of using LLM-as-a-Judge when evaluating
  grounded answers generated by RAG systems. To assess the calibration and
  discrimination capabilities of judge models, we identify 7 generator failure
  modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a
  meta-evaluation benchmark of 144 unit tests. This benchmark reveals that
  existing automated RAG evaluation frameworks often overlook important failure
  modes, even when using GPT-4 as a judge.
    To improve on the current design of automated RAG evaluation frameworks, we
  propose a novel pipeline and find that while closed models perform well on
  GroUSE, state-of-the-art open-source judges do not generalize to our proposed
  criteria, despite strong correlation with GPT-4's judgement. Our findings
  suggest that correlation with GPT-4 is an incomplete proxy for the practical
  performance of judge models and should be supplemented with evaluations on unit
  tests for precise failure mode detection.
    We further show that finetuning Llama-3 on GPT-4's reasoning traces
  significantly boosts its evaluation capabilities, improving upon both
  correlation with GPT-4's evaluations and calibration on reference situations.
paper_authors._id: 66e14e7ec81c80a47b064b35
paper_authors.name: Sacha Muller
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T15:01:32.938Z
paper_authors._id: 64b92a67296e311ff66cfe33
paper_authors.avatarUrl: /avatars/762692b77fbb187f42d2f2e8b8baea1b.svg
paper_authors.fullname: Sacha Muller
paper_authors.type: user
paper_authors.user: smuller
paper_authors._id: 66e14e7ec81c80a47b064b36
paper_authors.name: António Loison
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T15:01:31.127Z
paper_authors._id: 6377b63b24d97f9f7ec70064
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6377b63b24d97f9f7ec70064/vBl68PAwwp353MOeOahMH.jpeg
paper_authors.fullname: Antonio Loison
paper_authors.type: user
paper_authors.user: antonioloison
paper_authors._id: 66e14e7ec81c80a47b064b37
paper_authors.name: Bilel Omrani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:10:33.944Z
paper_authors._id: 61fbe1550ec9c25531549928
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650962849424-61fbe1550ec9c25531549928.jpeg
paper_authors.fullname: Bilel Omrani
paper_authors.type: user
paper_authors.user: bilelomrani
paper_authors._id: 66e14e7ec81c80a47b064b38
paper_authors.name: Gautier Viaud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:10:40.568Z
paper_authors._id: 618a65e17304dc918c6602ff
paper_authors.avatarUrl: /avatars/8af1112094169da80c65e24ab71c7e59.svg
paper_authors.fullname: Gautier Viaud
paper_authors.type: user
paper_authors.user: gautierviaud
paper_title: GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question
    Answering
paper_summary: Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use
  Large Language Models (LLMs) alongside private and up-to-date knowledge bases.
  In this work, we address the challenges of using LLM-as-a-Judge when evaluating
  grounded answers generated by RAG systems. To assess the calibration and
  discrimination capabilities of judge models, we identify 7 generator failure
  modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a
  meta-evaluation benchmark of 144 unit tests. This benchmark reveals that
  existing automated RAG evaluation frameworks often overlook important failure
  modes, even when using GPT-4 as a judge.
    To improve on the current design of automated RAG evaluation frameworks, we
  propose a novel pipeline and find that while closed models perform well on
  GroUSE, state-of-the-art open-source judges do not generalize to our proposed
  criteria, despite strong correlation with GPT-4's judgement. Our findings
  suggest that correlation with GPT-4 is an incomplete proxy for the practical
  performance of judge models and should be supplemented with evaluations on unit
  tests for precise failure mode detection.
    We further show that finetuning Llama-3 on GPT-4's reasoning traces
  significantly boosts its evaluation capabilities, improving upon both
  correlation with GPT-4's evaluations and calibration on reference situations.
paper_ai_summary: A retrieved-augmented generation (RAG) evaluation benchmark reveals limitations in existing automated frameworks, suggesting the need for unit test evaluations and fine-tuning LLMs on GPT-4's reasoning.
paper_ai_keywords: Retrieval-Augmented Generation Large Language Models LLM-as-a-Judge ground truth evaluation GroUSE generator failure modes automated evaluation frameworks closed models open-source judges calibration discrimination capabilities fine-tuning Llama-3 reasoning traces