RealCritic: Towards Effectiveness-Driven Evaluation of Language Model
  Critiques
Critiques are important for enhancing the performance of Large Language
Models (LLMs), enabling both self-improvement and constructive feedback for
others by identifying flaws and suggesting improvements. However, evaluating
the critique capabilities of LLMs presents a significant challenge due to the
open-ended nature of the task. In this work, we introduce a new benchmark
designed to assess the critique capabilities of LLMs. Unlike existing
benchmarks, which typically function in an open-loop fashion, our approach
employs a closed-loop methodology that evaluates the quality of corrections
generated from critiques. Moreover, the benchmark incorporates features such as
self-critique, cross-critique, and iterative critique, which are crucial for
distinguishing the abilities of advanced reasoning models from more classical
ones. We implement this benchmark using eight challenging reasoning tasks. We
have several interesting findings. First, despite demonstrating comparable
performance in direct chain-of-thought generation, classical LLMs significantly
lag behind the advanced reasoning-based model o1-mini across all critique
scenarios. Second, in self-critique and iterative critique settings, classical
LLMs may even underperform relative to their baseline capabilities. We hope
that this benchmark will serve as a valuable resource to guide future
advancements. The code and data are available at
https://github.com/tangzhy/RealCritic.
6796fdb7582fda525a7f6434
Zhengyang Tang
admin_assigned
2025-01-27T15:00:51.828Z
64912976b95c3f0a1e6233cb
/avatars/c0615f8c6606073faffb419757d4e667.svg
Zhengyang Tang
user
tangzhy
6796fdb7582fda525a7f6435
Ziniu Li
admin_assigned
2025-01-27T15:01:05.526Z
647c4c901f878439e2fd34d6
/avatars/b4399d210d7239d4662b11a4ee7b527d.svg
Ziniu Li
user
ziniuli
6796fdb7582fda525a7f6436
Zhenyang Xiao
admin_assigned
2025-01-27T15:01:15.578Z
649d87b19f0ffe21ef26d36b
/avatars/2a0b47328a780f24a640933111fe163c.svg
xiaozhenyang
user
yeshoubaizi
6796fdb7582fda525a7f6437
Tian Ding
6796fdb7582fda525a7f6438
Ruoyu Sun
6796fdb7582fda525a7f6439
Benyou Wang
admin_assigned
2025-01-27T15:01:55.582Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
6796fdb7582fda525a7f643a
Dayiheng Liu
admin_assigned
2025-01-27T15:02:01.463Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
6796fdb7582fda525a7f643b
Fei Huang
6796fdb7582fda525a7f643c
Tianyu Liu
6796fdb7582fda525a7f643d
Bowen Yu
admin_assigned
2025-01-27T15:02:16.321Z
6583ab7983a9e1460c67d876
/avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
bowen
user
bowenYu
6796fdb7582fda525a7f643e
Junyang Lin
admin_assigned
2025-01-27T15:02:08.327Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
RealCritic: Towards Effectiveness-Driven Evaluation of Language Model
  Critiques
Critiques are important for enhancing the performance of Large Language
Models (LLMs), enabling both self-improvement and constructive feedback for
others by identifying flaws and suggesting improvements. However, evaluating
the critique capabilities of LLMs presents a significant challenge due to the
open-ended nature of the task. In this work, we introduce a new benchmark
designed to assess the critique capabilities of LLMs. Unlike existing
benchmarks, which typically function in an open-loop fashion, our approach
employs a closed-loop methodology that evaluates the quality of corrections
generated from critiques. Moreover, the benchmark incorporates features such as
self-critique, cross-critique, and iterative critique, which are crucial for
distinguishing the abilities of advanced reasoning models from more classical
ones. We implement this benchmark using eight challenging reasoning tasks. We
have several interesting findings. First, despite demonstrating comparable
performance in direct chain-of-thought generation, classical LLMs significantly
lag behind the advanced reasoning-based model o1-mini across all critique
scenarios. Second, in self-critique and iterative critique settings, classical
LLMs may even underperform relative to their baseline capabilities. We hope
that this benchmark will serve as a valuable resource to guide future
advancements. The code and data are available at
https://github.com/tangzhy/RealCritic.
A new benchmark evaluates LLM critique capabilities using closed-loop methodology with self-critique, cross-critique, and iterative critique, revealing differences between classical and advanced reasoning models.
Large Language Models (LLMs) critique capabilities benchmark closed-loop methodology self-critique cross-critique iterative critique advanced reasoning chain-of-thought generation