title: Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for
    LLM-as-a-Judge
summary: LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become
  a widely adopted auto-evaluation method. However, its reliability is
  compromised by the CoT reasoning's inability to capture comprehensive and
  deeper details, often leading to incomplete outcomes. Existing methods mainly
  rely on majority voting or criteria expansion, which is insufficient to address
  the limitation in CoT. We propose Crowd-based Comparative Evaluation, which
  introduces additional crowd responses to compare with the candidate responses,
  thereby exposing deeper and more comprehensive details within the candidate
  responses. This process effectively guides LLM-as-a-Judge to provide a more
  detailed CoT judgment. Extensive experiments demonstrate that our approach
  enhances evaluation reliability, achieving an average accuracy gain of 6.7%
  across five benchmarks. Moreover, our method produces higher-quality CoTs that
  facilitate judge distillation and exhibit superior performance in rejection
  sampling for supervised fine-tuning (SFT), referred to as crowd rejection
  sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs
  generated by ours are more comprehensive and of higher quality, and evaluation
  accuracy improves as inference scales.
paper_authors._id: 67b547ffc9071a3e97139532
paper_authors.name: Qiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:10.215Z
paper_authors._id: 62a42f22c683d02f5b63320c
paper_authors.avatarUrl: /avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg
paper_authors.fullname: Qiyuan Zhang
paper_authors.type: user
paper_authors.user: DonJoey
paper_authors._id: 67b547ffc9071a3e97139533
paper_authors.name: Yufei Wang
paper_authors._id: 67b547ffc9071a3e97139534
paper_authors.name: Yuxin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:08.101Z
paper_authors._id: 63c20105726f62e411fbe882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c20105726f62e411fbe882/2UsU9O2psbDjJzz-sAmGH.jpeg
paper_authors.fullname: Yuxin Jiang
paper_authors.type: user
paper_authors.user: YuxinJiang
paper_authors._id: 67b547ffc9071a3e97139535
paper_authors.name: Liangyou Li
paper_authors._id: 67b547ffc9071a3e97139536
paper_authors.name: Chuhan Wu
paper_authors._id: 67b547ffc9071a3e97139537
paper_authors.name: Yasheng Wang
paper_authors._id: 67b547ffc9071a3e97139538
paper_authors.name: Xin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T11:54:48.156Z
paper_authors._id: 647415007afa69c3c7a98f1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
paper_authors.fullname: Xin Jiang
paper_authors.type: user
paper_authors.user: horiz94
paper_authors._id: 67b547ffc9071a3e97139539
paper_authors.name: Lifeng Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T11:54:33.330Z
paper_authors._id: 655b1360c11dee7f7e7cf794
paper_authors.avatarUrl: /avatars/efb4b91e9bb8ab531331c8e4296f754c.svg
paper_authors.fullname: lifengshang
paper_authors.type: user
paper_authors.user: lifengshang
paper_authors._id: 67b547ffc9071a3e9713953a
paper_authors.name: Ruiming Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T11:54:25.814Z
paper_authors._id: 6728c3b8d5ceae39aa1d2fdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VV4-DjhxzEVceNLmsokn2.png
paper_authors.fullname: tang ruiming
paper_authors.type: user
paper_authors.user: zhangsan5421
paper_authors._id: 67b547ffc9071a3e9713953b
paper_authors.name: Fuyuan Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T11:54:14.440Z
paper_authors._id: 65d2bb5c6130ef7be012d235
paper_authors.avatarUrl: /avatars/1c1e3bbb2c683a5c9d1f792a2c13fc4a.svg
paper_authors.fullname: Fuyuan Lyu
paper_authors.type: user
paper_authors.user: silentspring2
paper_authors._id: 67b547ffc9071a3e9713953c
paper_authors.name: Chen Ma
paper_title: Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for
    LLM-as-a-Judge
paper_summary: LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become
  a widely adopted auto-evaluation method. However, its reliability is
  compromised by the CoT reasoning's inability to capture comprehensive and
  deeper details, often leading to incomplete outcomes. Existing methods mainly
  rely on majority voting or criteria expansion, which is insufficient to address
  the limitation in CoT. We propose Crowd-based Comparative Evaluation, which
  introduces additional crowd responses to compare with the candidate responses,
  thereby exposing deeper and more comprehensive details within the candidate
  responses. This process effectively guides LLM-as-a-Judge to provide a more
  detailed CoT judgment. Extensive experiments demonstrate that our approach
  enhances evaluation reliability, achieving an average accuracy gain of 6.7%
  across five benchmarks. Moreover, our method produces higher-quality CoTs that
  facilitate judge distillation and exhibit superior performance in rejection
  sampling for supervised fine-tuning (SFT), referred to as crowd rejection
  sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs
  generated by ours are more comprehensive and of higher quality, and evaluation
  accuracy improves as inference scales.
paper_ai_summary: Crowd-based Comparative Evaluation enhances the reliability and quality of chain-of-thought judgments generated by LLM-as-a-Judge through additional crowd responses, improving accuracy and facilitating judge distillation and efficient supervised fine-tuning.
paper_ai_keywords: chain-of-thought LLM-as-a-Judge auto-evaluation majority voting criteria expansion comparative evaluation candidate responses detailed CoT judgment judge distillation supervised fine-tuning rejection sampling crowd rejection sampling inference scaling