Judging the Judges: Evaluating Alignment and Vulnerabilities in
  LLMs-as-Judges
Offering a promising solution to the scalability challenges associated with
human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an
approach to evaluating large language models (LLMs). However, there are still
many open questions about the strengths and weaknesses of this paradigm, and
what potential biases it may hold. In this paper, we present a comprehensive
study of the performance of various LLMs acting as judges. We leverage TriviaQA
as a benchmark for assessing objective knowledge reasoning of LLMs and evaluate
them alongside human annotations which we found to have a high inter-annotator
agreement. Our study includes 9 judge models and 9 exam taker models -- both
base and instruction-tuned. We assess the judge model's alignment across
different model sizes, families, and judge prompts. Among other results, our
research rediscovers the importance of using Cohen's kappa as a metric of
alignment as opposed to simple percent agreement, showing that judges with high
percent agreement can still assign vastly different scores. We find that both
Llama-3 70B and GPT-4 Turbo have an excellent alignment with humans, but in
terms of ranking exam taker models, they are outperformed by both JudgeLM-7B
and the lexical judge Contains, which have up to 34 points lower human
alignment. Through error analysis and various other studies, including the
effects of instruction length and leniency bias, we hope to provide valuable
lessons for using LLMs as judges in the future.
667801ee0d65792ffd18c41a
Aman Singh Thakur
extracted_confirmed
2024-06-23T13:47:25.956Z
63b491f3103617b0a5af6b4b
https://cdn-avatars.huggingface.co/v1/production/uploads/63b491f3103617b0a5af6b4b/4RqcGDxdF8Ny9cqyVuwYg.jpeg
Aman Singh Thakur
user
singh96aman
667801ee0d65792ffd18c41b
Kartik Choudhary
extracted_confirmed
2024-06-23T22:25:30.094Z
646020c31ba619a54119f8c9
/avatars/43c7bbac7569e3d63017831f71ecd762.svg
Kartik Choudhary
user
kartik727
667801ee0d65792ffd18c41c
Venkat Srinik Ramayapally
extracted_pending
2024-06-23T11:07:28.157Z
65cd743afd5f628b15e6779c
/avatars/5aa58d7cc9da903399210b9f1f54dd68.svg
Venkat Srinik Ramayapally
user
Srinik-1
667801ee0d65792ffd18c41d
Sankaran Vaidyanathan
admin_assigned
2024-06-24T13:27:35.239Z
6632dea587ce9a8935eb8751
/avatars/979a72589441400a9dc25b1008de7def.svg
Sankaran Vaidyanathan
user
sankaranv
667801ee0d65792ffd18c41e
Dieuwke Hupkes
admin_assigned
2024-06-24T13:27:40.612Z
62062bbbc72e929b014f0e49
/avatars/78c49f863eda2eac82b18261e794fb0f.svg
Dieuwke Hupkes
user
dieuwkehupkes
Judging the Judges: Evaluating Alignment and Vulnerabilities in
  LLMs-as-Judges
Offering a promising solution to the scalability challenges associated with
human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an
approach to evaluating large language models (LLMs). However, there are still
many open questions about the strengths and weaknesses of this paradigm, and
what potential biases it may hold. In this paper, we present a comprehensive
study of the performance of various LLMs acting as judges. We leverage TriviaQA
as a benchmark for assessing objective knowledge reasoning of LLMs and evaluate
them alongside human annotations which we found to have a high inter-annotator
agreement. Our study includes 9 judge models and 9 exam taker models -- both
base and instruction-tuned. We assess the judge model's alignment across
different model sizes, families, and judge prompts. Among other results, our
research rediscovers the importance of using Cohen's kappa as a metric of
alignment as opposed to simple percent agreement, showing that judges with high
percent agreement can still assign vastly different scores. We find that both
Llama-3 70B and GPT-4 Turbo have an excellent alignment with humans, but in
terms of ranking exam taker models, they are outperformed by both JudgeLM-7B
and the lexical judge Contains, which have up to 34 points lower human
alignment. Through error analysis and various other studies, including the
effects of instruction length and leniency bias, we hope to provide valuable
lessons for using LLMs as judges in the future.
LLM-as-a-judge method for evaluating LLMs shows promise but varies in performance and alignment with human judges, with smaller models like JudgeLM-7B sometimes outperforming larger ones in ranking tasks.
LLM-as-a-judge large language models (LLMs) TriviaQA Cohen's kappa alignment instruction-tuned JudgeLM lexical judge leniency bias