title: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model
    Critiques
summary: Critiques are important for enhancing the performance of Large Language
  Models (LLMs), enabling both self-improvement and constructive feedback for
  others by identifying flaws and suggesting improvements. However, evaluating
  the critique capabilities of LLMs presents a significant challenge due to the
  open-ended nature of the task. In this work, we introduce a new benchmark
  designed to assess the critique capabilities of LLMs. Unlike existing
  benchmarks, which typically function in an open-loop fashion, our approach
  employs a closed-loop methodology that evaluates the quality of corrections
  generated from critiques. Moreover, the benchmark incorporates features such as
  self-critique, cross-critique, and iterative critique, which are crucial for
  distinguishing the abilities of advanced reasoning models from more classical
  ones. We implement this benchmark using eight challenging reasoning tasks. We
  have several interesting findings. First, despite demonstrating comparable
  performance in direct chain-of-thought generation, classical LLMs significantly
  lag behind the advanced reasoning-based model o1-mini across all critique
  scenarios. Second, in self-critique and iterative critique settings, classical
  LLMs may even underperform relative to their baseline capabilities. We hope
  that this benchmark will serve as a valuable resource to guide future
  advancements. The code and data are available at
  https://github.com/tangzhy/RealCritic.
paper_authors._id: 6796fdb7582fda525a7f6434
paper_authors.name: Zhengyang Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:00:51.828Z
paper_authors._id: 64912976b95c3f0a1e6233cb
paper_authors.avatarUrl: /avatars/c0615f8c6606073faffb419757d4e667.svg
paper_authors.fullname: Zhengyang Tang
paper_authors.type: user
paper_authors.user: tangzhy
paper_authors._id: 6796fdb7582fda525a7f6435
paper_authors.name: Ziniu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:01:05.526Z
paper_authors._id: 647c4c901f878439e2fd34d6
paper_authors.avatarUrl: /avatars/b4399d210d7239d4662b11a4ee7b527d.svg
paper_authors.fullname: Ziniu Li
paper_authors.type: user
paper_authors.user: ziniuli
paper_authors._id: 6796fdb7582fda525a7f6436
paper_authors.name: Zhenyang Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:01:15.578Z
paper_authors._id: 649d87b19f0ffe21ef26d36b
paper_authors.avatarUrl: /avatars/2a0b47328a780f24a640933111fe163c.svg
paper_authors.fullname: xiaozhenyang
paper_authors.type: user
paper_authors.user: yeshoubaizi
paper_authors._id: 6796fdb7582fda525a7f6437
paper_authors.name: Tian Ding
paper_authors._id: 6796fdb7582fda525a7f6438
paper_authors.name: Ruoyu Sun
paper_authors._id: 6796fdb7582fda525a7f6439
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:01:55.582Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 6796fdb7582fda525a7f643a
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:02:01.463Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 6796fdb7582fda525a7f643b
paper_authors.name: Fei Huang
paper_authors._id: 6796fdb7582fda525a7f643c
paper_authors.name: Tianyu Liu
paper_authors._id: 6796fdb7582fda525a7f643d
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:02:16.321Z
paper_authors._id: 6583ab7983a9e1460c67d876
paper_authors.avatarUrl: /avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
paper_authors.fullname: bowen
paper_authors.type: user
paper_authors.user: bowenYu
paper_authors._id: 6796fdb7582fda525a7f643e
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T15:02:08.327Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model
    Critiques
paper_summary: Critiques are important for enhancing the performance of Large Language
  Models (LLMs), enabling both self-improvement and constructive feedback for
  others by identifying flaws and suggesting improvements. However, evaluating
  the critique capabilities of LLMs presents a significant challenge due to the
  open-ended nature of the task. In this work, we introduce a new benchmark
  designed to assess the critique capabilities of LLMs. Unlike existing
  benchmarks, which typically function in an open-loop fashion, our approach
  employs a closed-loop methodology that evaluates the quality of corrections
  generated from critiques. Moreover, the benchmark incorporates features such as
  self-critique, cross-critique, and iterative critique, which are crucial for
  distinguishing the abilities of advanced reasoning models from more classical
  ones. We implement this benchmark using eight challenging reasoning tasks. We
  have several interesting findings. First, despite demonstrating comparable
  performance in direct chain-of-thought generation, classical LLMs significantly
  lag behind the advanced reasoning-based model o1-mini across all critique
  scenarios. Second, in self-critique and iterative critique settings, classical
  LLMs may even underperform relative to their baseline capabilities. We hope
  that this benchmark will serve as a valuable resource to guide future
  advancements. The code and data are available at
  https://github.com/tangzhy/RealCritic.
paper_ai_keywords: Large Language Models (LLMs) closed-loop methodology self-critique cross-critique iterative critique chain-of-thought generation reasoning tasks advanced reasoning-based model baseline capabilities RealCritic