title: Teaching Language Models to Critique via Reinforcement Learning
summary: Teaching large language models (LLMs) to critique and refine their outputs is
  crucial for building systems that can iteratively improve, yet it is
  fundamentally limited by the ability to provide accurate judgments and
  actionable suggestions. In this work, we study LLM critics for code generation
  and propose CTRL, a framework for Critic
  Training via Reinforcement Learning, which
  trains a critic model to generate feedback that maximizes correction
  performance for a fixed generator model without human supervision. Our results
  demonstrate that critics trained with CTRL significantly enhance
  pass rates and mitigate compounding errors across both base and stronger
  generator models. Furthermore, we show that these critic models act as accurate
  generative reward models and enable test-time scaling through iterative
  critique-revision, achieving up to 106.1% relative improvements across
  challenging code generation benchmarks.
paper_authors._id: 67a5a8e595df68b0a167c298
paper_authors.name: Zhihui Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:17:02.682Z
paper_authors._id: 622f103fc78da4c7ebd7c887
paper_authors.avatarUrl: /avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Zhihui
paper_authors._id: 67a5a8e595df68b0a167c299
paper_authors.name: Jie chen
paper_authors._id: 67a5a8e595df68b0a167c29a
paper_authors.name: Liyu Chen
paper_authors._id: 67a5a8e595df68b0a167c29b
paper_authors.name: Weichao Mao
paper_authors._id: 67a5a8e595df68b0a167c29c
paper_authors.name: Jingjing Xu
paper_authors._id: 67a5a8e595df68b0a167c29d
paper_authors.name: Lingpeng Kong
paper_title: Teaching Language Models to Critique via Reinforcement Learning
paper_summary: Teaching large language models (LLMs) to critique and refine their outputs is
  crucial for building systems that can iteratively improve, yet it is
  fundamentally limited by the ability to provide accurate judgments and
  actionable suggestions. In this work, we study LLM critics for code generation
  and propose CTRL, a framework for Critic
  Training via Reinforcement Learning, which
  trains a critic model to generate feedback that maximizes correction
  performance for a fixed generator model without human supervision. Our results
  demonstrate that critics trained with CTRL significantly enhance
  pass rates and mitigate compounding errors across both base and stronger
  generator models. Furthermore, we show that these critic models act as accurate
  generative reward models and enable test-time scaling through iterative
  critique-revision, achieving up to 106.1% relative improvements across
  challenging code generation benchmarks.
paper_ai_keywords: large language models (LLMs) code generation critic model reinforcement learning correction performance generative reward models iterative critique-revision