GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question
  Answering
Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use
Large Language Models (LLMs) alongside private and up-to-date knowledge bases.
In this work, we address the challenges of using LLM-as-a-Judge when evaluating
grounded answers generated by RAG systems. To assess the calibration and
discrimination capabilities of judge models, we identify 7 generator failure
modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a
meta-evaluation benchmark of 144 unit tests. This benchmark reveals that
existing automated RAG evaluation frameworks often overlook important failure
modes, even when using GPT-4 as a judge.
  To improve on the current design of automated RAG evaluation frameworks, we
propose a novel pipeline and find that while closed models perform well on
GroUSE, state-of-the-art open-source judges do not generalize to our proposed
criteria, despite strong correlation with GPT-4's judgement. Our findings
suggest that correlation with GPT-4 is an incomplete proxy for the practical
performance of judge models and should be supplemented with evaluations on unit
tests for precise failure mode detection.
  We further show that finetuning Llama-3 on GPT-4's reasoning traces
significantly boosts its evaluation capabilities, improving upon both
correlation with GPT-4's evaluations and calibration on reference situations.
66e14e7ec81c80a47b064b35
Sacha Muller
claimed_verified
2024-09-11T15:01:32.938Z
64b92a67296e311ff66cfe33
/avatars/762692b77fbb187f42d2f2e8b8baea1b.svg
Sacha Muller
user
smuller
66e14e7ec81c80a47b064b36
António Loison
claimed_verified
2024-09-11T15:01:31.127Z
6377b63b24d97f9f7ec70064
https://cdn-avatars.huggingface.co/v1/production/uploads/6377b63b24d97f9f7ec70064/vBl68PAwwp353MOeOahMH.jpeg
Antonio Loison
user
antonioloison
66e14e7ec81c80a47b064b37
Bilel Omrani
admin_assigned
2024-09-12T07:10:33.944Z
61fbe1550ec9c25531549928
https://cdn-avatars.huggingface.co/v1/production/uploads/1650962849424-61fbe1550ec9c25531549928.jpeg
Bilel Omrani
user
bilelomrani
66e14e7ec81c80a47b064b38
Gautier Viaud
admin_assigned
2024-09-12T07:10:40.568Z
618a65e17304dc918c6602ff
/avatars/8af1112094169da80c65e24ab71c7e59.svg
Gautier Viaud
user
gautierviaud
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question
  Answering
Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use
Large Language Models (LLMs) alongside private and up-to-date knowledge bases.
In this work, we address the challenges of using LLM-as-a-Judge when evaluating
grounded answers generated by RAG systems. To assess the calibration and
discrimination capabilities of judge models, we identify 7 generator failure
modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a
meta-evaluation benchmark of 144 unit tests. This benchmark reveals that
existing automated RAG evaluation frameworks often overlook important failure
modes, even when using GPT-4 as a judge.
  To improve on the current design of automated RAG evaluation frameworks, we
propose a novel pipeline and find that while closed models perform well on
GroUSE, state-of-the-art open-source judges do not generalize to our proposed
criteria, despite strong correlation with GPT-4's judgement. Our findings
suggest that correlation with GPT-4 is an incomplete proxy for the practical
performance of judge models and should be supplemented with evaluations on unit
tests for precise failure mode detection.
  We further show that finetuning Llama-3 on GPT-4's reasoning traces
significantly boosts its evaluation capabilities, improving upon both
correlation with GPT-4's evaluations and calibration on reference situations.
A retrieved-augmented generation (RAG) evaluation benchmark reveals limitations in existing automated frameworks, suggesting the need for unit test evaluations and fine-tuning LLMs on GPT-4's reasoning.
Retrieval-Augmented Generation Large Language Models LLM-as-a-Judge ground truth evaluation GroUSE generator failure modes automated evaluation frameworks closed models open-source judges calibration discrimination capabilities fine-tuning Llama-3 reasoning traces