title: Thinking Preference Optimization
summary: Supervised Fine-Tuning (SFT) has been a go-to and effective method for
  enhancing long chain-of-thought (CoT) reasoning in relatively small LLMs by
  fine-tuning them with long CoT responses from larger LLMs. To continually
  improve reasoning abilities, we can either collect new high-quality long CoT
  reasoning SFT data or repeatedly train on existing SFT datasets. However,
  acquiring new long CoT SFT data is costly and limited, while repeated training
  often results in a performance plateau or decline. To further boost the
  performance with the SFT data, we propose Thinking Preference Optimization
  (ThinkPO), a simple yet effective post-SFT method that enhances long CoT
  reasoning without requiring new long CoT responses. Instead, ThinkPO utilizes
  readily available or easily obtainable short CoT reasoning responses as
  rejected answers and long CoT responses as chosen answers for the same
  question. It then applies direct preference optimization to encourage the model
  to favor longer reasoning outputs. Experiments show that ThinkPO further
  improves the reasoning performance of SFT-ed models, e.g. it increases math
  reasoning accuracy of SFT-ed models by 8.6% and output length by 25.9%.
  Notably, ThinkPO is capable of continually boosting the performance of the
  publicly distilled SFT model, e.g., increasing the official
  DeepSeek-R1-Distill-Qwen-7B's performance on MATH500 from 87.4% to 91.2%.
paper_authors._id: 67b6b014f7e569081326494f
paper_authors.name: Wang Yang
paper_authors._id: 67b6b014f7e5690813264950
paper_authors.name: Hongye Jin
paper_authors._id: 67b6b014f7e5690813264951
paper_authors.name: Jingfeng Yang
paper_authors._id: 67b6b014f7e5690813264952
paper_authors.name: Vipin Chaudhary
paper_authors._id: 67b6b014f7e5690813264953
paper_authors.name: Xiaotian Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:12:59.725Z
paper_authors._id: 650dde4ce14eeb01d42b37a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
paper_authors.fullname: Xiaotian Han
paper_authors.type: user
paper_authors.user: xiaotianhan
paper_title: Thinking Preference Optimization
paper_summary: Supervised Fine-Tuning (SFT) has been a go-to and effective method for
  enhancing long chain-of-thought (CoT) reasoning in relatively small LLMs by
  fine-tuning them with long CoT responses from larger LLMs. To continually
  improve reasoning abilities, we can either collect new high-quality long CoT
  reasoning SFT data or repeatedly train on existing SFT datasets. However,
  acquiring new long CoT SFT data is costly and limited, while repeated training
  often results in a performance plateau or decline. To further boost the
  performance with the SFT data, we propose Thinking Preference Optimization
  (ThinkPO), a simple yet effective post-SFT method that enhances long CoT
  reasoning without requiring new long CoT responses. Instead, ThinkPO utilizes
  readily available or easily obtainable short CoT reasoning responses as
  rejected answers and long CoT responses as chosen answers for the same
  question. It then applies direct preference optimization to encourage the model
  to favor longer reasoning outputs. Experiments show that ThinkPO further
  improves the reasoning performance of SFT-ed models, e.g. it increases math
  reasoning accuracy of SFT-ed models by 8.6% and output length by 25.9%.
  Notably, ThinkPO is capable of continually boosting the performance of the
  publicly distilled SFT model, e.g., increasing the official
  DeepSeek-R1-Distill-Qwen-7B's performance on MATH500 from 87.4% to 91.2%.
paper_ai_keywords: Thinking Preference Optimization (ThinkPO) CoT reasoning SFT (Supervised Fine-Tuning) long chain-of-thought direct preference optimization DeepSeek-R1-Distill-Qwen-7B MATH500