title: Critique Fine-Tuning: Learning to Critique is More Effective than
    Learning to Imitate
summary: Supervised Fine-Tuning (SFT) is commonly used to train language models to
  imitate annotated responses for given instructions. In this paper, we challenge
  this paradigm and propose Critique Fine-Tuning (CFT), a strategy where models
  learn to critique noisy responses rather than simply imitate correct ones.
  Inspired by human learning processes that emphasize critical thinking, CFT
  encourages deeper analysis and nuanced understanding-traits often overlooked by
  standard SFT. To validate the effectiveness of CFT, we construct a 50K-sample
  dataset from WebInstruct, using GPT-4o as the teacher to generate critiques in
  the form of (input=[query; noisy response], output=critique). CFT on this
  dataset yields a consistent 4-10% improvement over SFT on six math benchmarks
  with different base models like Qwen2.5, Qwen2.5-Math and DeepSeek-Math. We
  further expand to MetaMath and NuminaMath datasets and observe similar gains
  over SFT. Notably, our Qwen2.5-Math-CFT model-trained on just 50K
  samples-matches or outperforms competitive models such as AceMath and
  Qwen2.5-Math-Instruct on most benchmarks, both of which use over 2M samples.
  Ablation studies show that CFT is robust to the source of noisy response and
  teacher critique model. Through these findings, we argue that critique-based
  training offers a more effective alternative to advance the reasoning of
  language models.
paper_authors._id: 679ae76cf211c66bd702f5d5
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:39:49.375Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 679ae76cf211c66bd702f5d6
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T15:17:01.780Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 679ae76cf211c66bd702f5d7
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-30T02:43:59.302Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: Critique Fine-Tuning: Learning to Critique is More Effective than
    Learning to Imitate
paper_summary: Supervised Fine-Tuning (SFT) is commonly used to train language models to
  imitate annotated responses for given instructions. In this paper, we challenge
  this paradigm and propose Critique Fine-Tuning (CFT), a strategy where models
  learn to critique noisy responses rather than simply imitate correct ones.
  Inspired by human learning processes that emphasize critical thinking, CFT
  encourages deeper analysis and nuanced understanding-traits often overlooked by
  standard SFT. To validate the effectiveness of CFT, we construct a 50K-sample
  dataset from WebInstruct, using GPT-4o as the teacher to generate critiques in
  the form of (input=[query; noisy response], output=critique). CFT on this
  dataset yields a consistent 4-10% improvement over SFT on six math benchmarks
  with different base models like Qwen2.5, Qwen2.5-Math and DeepSeek-Math. We
  further expand to MetaMath and NuminaMath datasets and observe similar gains
  over SFT. Notably, our Qwen2.5-Math-CFT model-trained on just 50K
  samples-matches or outperforms competitive models such as AceMath and
  Qwen2.5-Math-Instruct on most benchmarks, both of which use over 2M samples.
  Ablation studies show that CFT is robust to the source of noisy response and
  teacher critique model. Through these findings, we argue that critique-based
  training offers a more effective alternative to advance the reasoning of
  language models.
paper_ai_summary: Critique Fine-Tuning (CFT) improves language model performance by training them to critique noisy responses, showing better reasoning ability than standard Supervised Fine-Tuning (SFT).
paper_ai_keywords: Critique Fine-Tuning CFT Supervised Fine-Tuning SFT WebInstruct GPT-4 Qwen2.5 Qwen2.5-Math DeepSeek-Math MetaMath NuminaMath AceMath Qwen2.5-Math-Instruct