title: DeepCritic: Deliberate Critique with Large Language Models
summary: As Large Language Models (LLMs) are rapidly evolving, providing accurate
  feedback and scalable oversight on their outputs becomes an urgent and critical
  problem. Leveraging LLMs as critique models to achieve automated supervision is
  a promising solution. In this work, we focus on studying and enhancing the math
  critique ability of LLMs. Current LLM critics provide critiques that are too
  shallow and superficial on each step, leading to low judgment accuracy and
  struggling to offer sufficient feedback for the LLM generator to correct
  mistakes. To tackle this issue, we propose a novel and effective two-stage
  framework to develop LLM critics that are capable of deliberately critiquing on
  each reasoning step of math solutions. In the first stage, we utilize
  Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for
  supervised fine-tuning. Each seed critique consists of deliberate step-wise
  critiques that includes multi-perspective verifications as well as in-depth
  critiques of initial critiques for each reasoning step. Then, we perform
  reinforcement learning on the fine-tuned model with either existing
  human-labeled data from PRM800K or our automatically annotated data obtained
  via Monte Carlo sampling-based correctness estimation, to further incentivize
  its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct
  not only significantly outperforms existing LLM critics (including the
  same-sized DeepSeek-R1-distill models and GPT-4o) on various error
  identification benchmarks, but also more effectively helps the LLM generator
  refine erroneous steps through more detailed feedback.
paper_authors._id: 68142e4a551709da9244e8d1
paper_authors.name: Wenkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:34:18.531Z
paper_authors._id: 64b7df742f5a966b973e25f7
paper_authors.avatarUrl: /avatars/e24e7769188d441317b3b7d10ef8fd60.svg
paper_authors.fullname: Wenkai Yang
paper_authors.type: user
paper_authors.user: Keven16
paper_authors._id: 68142e4a551709da9244e8d2
paper_authors.name: Jingwen Chen
paper_authors._id: 68142e4a551709da9244e8d3
paper_authors.name: Yankai Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:44:11.626Z
paper_authors._id: 657a651e1433ea7d44de6397
paper_authors.avatarUrl: /avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
paper_authors.fullname: Yankai Lin
paper_authors.type: user
paper_authors.user: lyk423
paper_authors._id: 68142e4a551709da9244e8d4
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:44:17.877Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: DeepCritic: Deliberate Critique with Large Language Models
paper_summary: As Large Language Models (LLMs) are rapidly evolving, providing accurate
  feedback and scalable oversight on their outputs becomes an urgent and critical
  problem. Leveraging LLMs as critique models to achieve automated supervision is
  a promising solution. In this work, we focus on studying and enhancing the math
  critique ability of LLMs. Current LLM critics provide critiques that are too
  shallow and superficial on each step, leading to low judgment accuracy and
  struggling to offer sufficient feedback for the LLM generator to correct
  mistakes. To tackle this issue, we propose a novel and effective two-stage
  framework to develop LLM critics that are capable of deliberately critiquing on
  each reasoning step of math solutions. In the first stage, we utilize
  Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for
  supervised fine-tuning. Each seed critique consists of deliberate step-wise
  critiques that includes multi-perspective verifications as well as in-depth
  critiques of initial critiques for each reasoning step. Then, we perform
  reinforcement learning on the fine-tuned model with either existing
  human-labeled data from PRM800K or our automatically annotated data obtained
  via Monte Carlo sampling-based correctness estimation, to further incentivize
  its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct
  not only significantly outperforms existing LLM critics (including the
  same-sized DeepSeek-R1-distill models and GPT-4o) on various error
  identification benchmarks, but also more effectively helps the LLM generator
  refine erroneous steps through more detailed feedback.
paper_ai_summary: A novel two-stage framework using Qwen2.5-72B-Instruct enhances LLMs' math critique ability by generating detailed step-wise critiques and applying reinforcement learning, resulting in better error identification and refinement.
paper_ai_keywords: Large Language Models (LLMs) critique models Qwen2.5-72B-Instruct supervised fine-tuning Monte Carlo sampling reinforcement learning DeepSeek-R1-distill GPT-4o error identification benchmarks