title: Critique-GRPO: Advancing LLM Reasoning with Natural Language and
    Numerical Feedback
summary: Recent advances in reinforcement learning (RL) with numerical feedback, such
  as scalar rewards, have significantly enhanced the complex reasoning
  capabilities of large language models (LLMs). Despite this success, we identify
  three key challenges encountered by RL with solely numerical feedback:
  performance plateaus, limited effectiveness of self-reflection, and persistent
  failures. We then demonstrate that RL-finetuned models, even after exhibiting
  performance plateaus, can generate correct refinements on persistently failed
  problems by leveraging natural language feedback in the form of critiques.
  Building on this insight, we propose Critique-GRPO, an online RL framework that
  integrates both natural language and numerical feedback for effective policy
  optimization. Critique-GRPO enables LLMs to learn from initial responses and
  critique-guided refinements simultaneously while maintaining exploration.
  Extensive experiments using Qwen2.5-7B-Base and Qwen3-8B-Base show that
  Critique-GRPO consistently outperforms supervised learning-based and RL-based
  fine-tuning approaches across eight challenging mathematical, STEM, and general
  reasoning tasks, improving average pass@1 scores by approximately 4.5% and 5%,
  respectively. Notably, Critique-GRPO surpasses a strong baseline that
  incorporates expert demonstrations within online RL. Further analysis reveals
  two critical insights about policy exploration: (1) higher entropy does not
  always guarantee efficient learning from exploration, and (2) longer responses
  do not necessarily lead to more effective exploration.
paper_authors._id: 684104d9ee7646c073776b2e
paper_authors.name: Xiaoying Zhang
paper_authors._id: 684104d9ee7646c073776b2f
paper_authors.name: Hao Sun
paper_authors._id: 684104d9ee7646c073776b30
paper_authors.name: Yipeng Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-05T02:45:46.851Z
paper_authors._id: 666e91b1623133f1ce35acc5
paper_authors.avatarUrl: /avatars/cc78520e6cfb83817c1d0c1ac867ebdd.svg
paper_authors.fullname: YipengZhang
paper_authors.type: user
paper_authors.user: YipengZhang
paper_authors._id: 684104d9ee7646c073776b31
paper_authors.name: Kaituo Feng
paper_authors._id: 684104d9ee7646c073776b32
paper_authors.name: Chaochao Lu
paper_authors._id: 684104d9ee7646c073776b33
paper_authors.name: Chao Yang
paper_authors._id: 684104d9ee7646c073776b34
paper_authors.name: Helen Meng
paper_title: Critique-GRPO: Advancing LLM Reasoning with Natural Language and
    Numerical Feedback
paper_summary: Recent advances in reinforcement learning (RL) with numerical feedback, such
  as scalar rewards, have significantly enhanced the complex reasoning
  capabilities of large language models (LLMs). Despite this success, we identify
  three key challenges encountered by RL with solely numerical feedback:
  performance plateaus, limited effectiveness of self-reflection, and persistent
  failures. We then demonstrate that RL-finetuned models, even after exhibiting
  performance plateaus, can generate correct refinements on persistently failed
  problems by leveraging natural language feedback in the form of critiques.
  Building on this insight, we propose Critique-GRPO, an online RL framework that
  integrates both natural language and numerical feedback for effective policy
  optimization. Critique-GRPO enables LLMs to learn from initial responses and
  critique-guided refinements simultaneously while maintaining exploration.
  Extensive experiments using Qwen2.5-7B-Base and Qwen3-8B-Base show that
  Critique-GRPO consistently outperforms supervised learning-based and RL-based
  fine-tuning approaches across eight challenging mathematical, STEM, and general
  reasoning tasks, improving average pass@1 scores by approximately 4.5% and 5%,
  respectively. Notably, Critique-GRPO surpasses a strong baseline that
  incorporates expert demonstrations within online RL. Further analysis reveals
  two critical insights about policy exploration: (1) higher entropy does not
  always guarantee efficient learning from exploration, and (2) longer responses
  do not necessarily lead to more effective exploration.
paper_ai_summary: Critique-GRPO, an RL framework combining numerical and natural language feedback, enhances LLM reasoning across tasks and outperforms existing methods.
paper_ai_keywords: reinforcement learning RL large language models LLMs scalar rewards performance plateaus self-reflection persistent failures natural language feedback critiques policy optimization Qwen2.5-7B-Base Qwen3-8B-Base pass@1 policy exploration entropy response length