Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for
  LLM-as-a-Judge
LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become
a widely adopted auto-evaluation method. However, its reliability is
compromised by the CoT reasoning's inability to capture comprehensive and
deeper details, often leading to incomplete outcomes. Existing methods mainly
rely on majority voting or criteria expansion, which is insufficient to address
the limitation in CoT. We propose Crowd-based Comparative Evaluation, which
introduces additional crowd responses to compare with the candidate responses,
thereby exposing deeper and more comprehensive details within the candidate
responses. This process effectively guides LLM-as-a-Judge to provide a more
detailed CoT judgment. Extensive experiments demonstrate that our approach
enhances evaluation reliability, achieving an average accuracy gain of 6.7%
across five benchmarks. Moreover, our method produces higher-quality CoTs that
facilitate judge distillation and exhibit superior performance in rejection
sampling for supervised fine-tuning (SFT), referred to as crowd rejection
sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs
generated by ours are more comprehensive and of higher quality, and evaluation
accuracy improves as inference scales.
67b547ffc9071a3e97139532
Qiyuan Zhang
claimed_verified
2025-02-19T09:01:10.215Z
62a42f22c683d02f5b63320c
/avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg
Qiyuan Zhang
user
DonJoey
67b547ffc9071a3e97139533
Yufei Wang
67b547ffc9071a3e97139534
Yuxin Jiang
claimed_verified
2025-02-19T09:01:08.101Z
63c20105726f62e411fbe882
https://cdn-avatars.huggingface.co/v1/production/uploads/63c20105726f62e411fbe882/2UsU9O2psbDjJzz-sAmGH.jpeg
Yuxin Jiang
user
YuxinJiang
67b547ffc9071a3e97139535
Liangyou Li
67b547ffc9071a3e97139536
Chuhan Wu
67b547ffc9071a3e97139537
Yasheng Wang
67b547ffc9071a3e97139538
Xin Jiang
admin_assigned
2025-02-19T11:54:48.156Z
647415007afa69c3c7a98f1f
https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
Xin Jiang
user
horiz94
67b547ffc9071a3e97139539
Lifeng Shang
admin_assigned
2025-02-19T11:54:33.330Z
655b1360c11dee7f7e7cf794
/avatars/efb4b91e9bb8ab531331c8e4296f754c.svg
lifengshang
user
lifengshang
67b547ffc9071a3e9713953a
Ruiming Tang
admin_assigned
2025-02-19T11:54:25.814Z
6728c3b8d5ceae39aa1d2fdd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VV4-DjhxzEVceNLmsokn2.png
tang ruiming
user
zhangsan5421
67b547ffc9071a3e9713953b
Fuyuan Lyu
admin_assigned
2025-02-19T11:54:14.440Z
65d2bb5c6130ef7be012d235
/avatars/1c1e3bbb2c683a5c9d1f792a2c13fc4a.svg
Fuyuan Lyu
user
silentspring2
67b547ffc9071a3e9713953c
Chen Ma
Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for
  LLM-as-a-Judge
LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become
a widely adopted auto-evaluation method. However, its reliability is
compromised by the CoT reasoning's inability to capture comprehensive and
deeper details, often leading to incomplete outcomes. Existing methods mainly
rely on majority voting or criteria expansion, which is insufficient to address
the limitation in CoT. We propose Crowd-based Comparative Evaluation, which
introduces additional crowd responses to compare with the candidate responses,
thereby exposing deeper and more comprehensive details within the candidate
responses. This process effectively guides LLM-as-a-Judge to provide a more
detailed CoT judgment. Extensive experiments demonstrate that our approach
enhances evaluation reliability, achieving an average accuracy gain of 6.7%
across five benchmarks. Moreover, our method produces higher-quality CoTs that
facilitate judge distillation and exhibit superior performance in rejection
sampling for supervised fine-tuning (SFT), referred to as crowd rejection
sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs
generated by ours are more comprehensive and of higher quality, and evaluation
accuracy improves as inference scales.
Crowd-based Comparative Evaluation enhances the reliability and quality of chain-of-thought judgments generated by LLM-as-a-Judge through additional crowd responses, improving accuracy and facilitating judge distillation and efficient supervised fine-tuning.
chain-of-thought LLM-as-a-Judge auto-evaluation majority voting criteria expansion comparative evaluation candidate responses detailed CoT judgment judge distillation supervised fine-tuning rejection sampling crowd rejection sampling inference scaling