title: Unleashing the Reasoning Potential of Pre-trained LLMs by Critique
    Fine-Tuning on One Problem
summary: We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess
  immense reasoning potential inherited from the pre-training stage. With
  reinforcement learning (RL), these models can improve dramatically on reasoning
  tasks. Recent studies have shown that even RL on a single problem can unleash
  these models' reasoning capabilities. However, RL is not only expensive but
  also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a
  critical question: Is there a more efficient way to unleash the reasoning
  potential of these powerful base LLMs? In this work, we demonstrate that
  Critique Fine-Tuning (CFT) on only one problem can effectively unleash the
  reasoning potential of LLMs. Our method constructs critique data by collecting
  diverse model-generated solutions to a single problem and using teacher LLMs to
  provide detailed critiques. We fine-tune Qwen and Llama family models, ranging
  from 1.5B to 14B parameters, on the CFT data and observe significant
  performance gains across diverse reasoning tasks. For example, with just 5 GPU
  hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six
  math benchmarks and 16% on three logic reasoning benchmarks. These results are
  comparable to or even surpass the results from RL with 20x less compute.
  Ablation studies reveal the robustness of one-shot CFT across different prompt
  problems. These results highlight one-shot CFT as a simple, general, and
  compute-efficient approach to unleashing the reasoning capabilities of modern
  LLMs.
paper_authors._id: 6840e7d81fadbc85ae3bdc0f
paper_authors.name: Yubo Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:51:07.108Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 6840e7d81fadbc85ae3bdc10
paper_authors.name: Ping Nie
paper_authors._id: 6840e7d81fadbc85ae3bdc11
paper_authors.name: Kai Zou
paper_authors._id: 6840e7d81fadbc85ae3bdc12
paper_authors.name: Lijun Wu
paper_authors._id: 6840e7d81fadbc85ae3bdc13
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:51:23.993Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: Unleashing the Reasoning Potential of Pre-trained LLMs by Critique
    Fine-Tuning on One Problem
paper_summary: We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess
  immense reasoning potential inherited from the pre-training stage. With
  reinforcement learning (RL), these models can improve dramatically on reasoning
  tasks. Recent studies have shown that even RL on a single problem can unleash
  these models' reasoning capabilities. However, RL is not only expensive but
  also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a
  critical question: Is there a more efficient way to unleash the reasoning
  potential of these powerful base LLMs? In this work, we demonstrate that
  Critique Fine-Tuning (CFT) on only one problem can effectively unleash the
  reasoning potential of LLMs. Our method constructs critique data by collecting
  diverse model-generated solutions to a single problem and using teacher LLMs to
  provide detailed critiques. We fine-tune Qwen and Llama family models, ranging
  from 1.5B to 14B parameters, on the CFT data and observe significant
  performance gains across diverse reasoning tasks. For example, with just 5 GPU
  hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six
  math benchmarks and 16% on three logic reasoning benchmarks. These results are
  comparable to or even surpass the results from RL with 20x less compute.
  Ablation studies reveal the robustness of one-shot CFT across different prompt
  problems. These results highlight one-shot CFT as a simple, general, and
  compute-efficient approach to unleashing the reasoning capabilities of modern
  LLMs.
paper_ai_summary: Critique Fine-Tuning on a single problem can efficiently enhance the reasoning capabilities of large language models with significant performance gains and reduced computational cost compared to reinforcement learning.
paper_ai_keywords: Critique Fine-Tuning teacher LLMs Qwen-Math Llama family models reasoning tasks one-shot CFT performance gains logic reasoning benchmarks math benchmarks prompt problems