title: Self-Training with Direct Preference Optimization Improves
    Chain-of-Thought Reasoning
summary: Effective training of language models (LMs) for mathematical reasoning tasks
  demands high-quality supervised fine-tuning data. Besides obtaining annotations
  from human experts, a common alternative is sampling from larger and more
  powerful LMs. However, this knowledge distillation approach can be costly and
  unstable, particularly when relying on closed-source, proprietary LMs like
  GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate
  that the reasoning abilities of small-scale LMs can be enhanced through
  self-training, a process where models learn from their own outputs. We also
  show that the conventional self-training can be further augmented by a
  preference learning algorithm called Direct Preference Optimization (DPO). By
  integrating DPO into self-training, we leverage preference data to guide LMs
  towards more accurate and diverse chain-of-thought reasoning. We evaluate our
  method across various mathematical reasoning tasks using different base models.
  Our experiments show that this approach not only improves LMs' reasoning
  performance but also offers a more cost-effective and scalable solution
  compared to relying on large proprietary LMs.
paper_authors._id: 66a740d7577ca69aa4c1c4a8
paper_authors.name: Tianduo Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:38:18.265Z
paper_authors._id: 6352aa7b6cfb8f149814de5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666361939036-noauth.jpeg
paper_authors.fullname: Tianduo Wang
paper_authors.type: user
paper_authors.user: Tianduo
paper_authors._id: 66a740d7577ca69aa4c1c4a9
paper_authors.name: Shichen Li
paper_authors._id: 66a740d7577ca69aa4c1c4aa
paper_authors.name: Wei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:39:47.467Z
paper_authors._id: 6447f0ed19538c015b27ecb8
paper_authors.avatarUrl: /avatars/aae591acfe36e3297d57103d1bda12cf.svg
paper_authors.fullname: Wei Lu
paper_authors.type: user
paper_authors.user: weilu
paper_title: Self-Training with Direct Preference Optimization Improves
    Chain-of-Thought Reasoning
paper_summary: Effective training of language models (LMs) for mathematical reasoning tasks
  demands high-quality supervised fine-tuning data. Besides obtaining annotations
  from human experts, a common alternative is sampling from larger and more
  powerful LMs. However, this knowledge distillation approach can be costly and
  unstable, particularly when relying on closed-source, proprietary LMs like
  GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate
  that the reasoning abilities of small-scale LMs can be enhanced through
  self-training, a process where models learn from their own outputs. We also
  show that the conventional self-training can be further augmented by a
  preference learning algorithm called Direct Preference Optimization (DPO). By
  integrating DPO into self-training, we leverage preference data to guide LMs
  towards more accurate and diverse chain-of-thought reasoning. We evaluate our
  method across various mathematical reasoning tasks using different base models.
  Our experiments show that this approach not only improves LMs' reasoning
  performance but also offers a more cost-effective and scalable solution
  compared to relying on large proprietary LMs.
paper_ai_summary: Enhancing small-scale language models for mathematical reasoning through self-training and preference learning, resulting in improved performance and reduced costs compared to large proprietary models.
paper_ai_keywords: language models fine-tuning knowledge distillation self-training Direct Preference Optimization preference learning chain-of-thought reasoning mathematical reasoning tasks