Unleashing the Reasoning Potential of Pre-trained LLMs by Critique
  Fine-Tuning on One Problem
We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess
immense reasoning potential inherited from the pre-training stage. With
reinforcement learning (RL), these models can improve dramatically on reasoning
tasks. Recent studies have shown that even RL on a single problem can unleash
these models' reasoning capabilities. However, RL is not only expensive but
also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a
critical question: Is there a more efficient way to unleash the reasoning
potential of these powerful base LLMs? In this work, we demonstrate that
Critique Fine-Tuning (CFT) on only one problem can effectively unleash the
reasoning potential of LLMs. Our method constructs critique data by collecting
diverse model-generated solutions to a single problem and using teacher LLMs to
provide detailed critiques. We fine-tune Qwen and Llama family models, ranging
from 1.5B to 14B parameters, on the CFT data and observe significant
performance gains across diverse reasoning tasks. For example, with just 5 GPU
hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six
math benchmarks and 16% on three logic reasoning benchmarks. These results are
comparable to or even surpass the results from RL with 20x less compute.
Ablation studies reveal the robustness of one-shot CFT across different prompt
problems. These results highlight one-shot CFT as a simple, general, and
compute-efficient approach to unleashing the reasoning capabilities of modern
LLMs.
6840e7d81fadbc85ae3bdc0f
Yubo Wang
admin_assigned
2025-06-05T12:51:07.108Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
6840e7d81fadbc85ae3bdc10
Ping Nie
6840e7d81fadbc85ae3bdc11
Kai Zou
6840e7d81fadbc85ae3bdc12
Lijun Wu
6840e7d81fadbc85ae3bdc13
Wenhu Chen
admin_assigned
2025-06-05T12:51:23.993Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
Unleashing the Reasoning Potential of Pre-trained LLMs by Critique
  Fine-Tuning on One Problem
We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess
immense reasoning potential inherited from the pre-training stage. With
reinforcement learning (RL), these models can improve dramatically on reasoning
tasks. Recent studies have shown that even RL on a single problem can unleash
these models' reasoning capabilities. However, RL is not only expensive but
also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a
critical question: Is there a more efficient way to unleash the reasoning
potential of these powerful base LLMs? In this work, we demonstrate that
Critique Fine-Tuning (CFT) on only one problem can effectively unleash the
reasoning potential of LLMs. Our method constructs critique data by collecting
diverse model-generated solutions to a single problem and using teacher LLMs to
provide detailed critiques. We fine-tune Qwen and Llama family models, ranging
from 1.5B to 14B parameters, on the CFT data and observe significant
performance gains across diverse reasoning tasks. For example, with just 5 GPU
hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six
math benchmarks and 16% on three logic reasoning benchmarks. These results are
comparable to or even surpass the results from RL with 20x less compute.
Ablation studies reveal the robustness of one-shot CFT across different prompt
problems. These results highlight one-shot CFT as a simple, general, and
compute-efficient approach to unleashing the reasoning capabilities of modern
LLMs.
Critique Fine-Tuning on a single problem can efficiently enhance the reasoning capabilities of large language models with significant performance gains and reduced computational cost compared to reinforcement learning.
Critique Fine-Tuning teacher LLMs Qwen-Math Llama family models reasoning tasks one-shot CFT performance gains logic reasoning benchmarks math benchmarks prompt problems