title: sDPO: Don't Use Your Data All at Once
summary: As development of large language models (LLM) progresses, aligning them with
  human preferences has become increasingly important. We propose stepwise DPO
  (sDPO), an extension of the recently popularized direct preference optimization
  (DPO) for alignment tuning. This approach involves dividing the available
  preference datasets and utilizing them in a stepwise manner, rather than
  employing it all at once. We demonstrate that this method facilitates the use
  of more precisely aligned reference models within the DPO training framework.
  Furthermore, sDPO trains the final model to be more performant, even
  outperforming other popular LLMs with more parameters.
paper_authors._id: 6606447bf631d41ec45b1ac0
paper_authors.name: Dahyun Kim
paper_authors._id: 6606447bf631d41ec45b1ac1
paper_authors.name: Yungi Kim
paper_authors._id: 6606447bf631d41ec45b1ac2
paper_authors.name: Wonho Song
paper_authors._id: 6606447bf631d41ec45b1ac3
paper_authors.name: Hyeonwoo Kim
paper_authors._id: 6606447bf631d41ec45b1ac4
paper_authors.name: Yunsu Kim
paper_authors._id: 6606447bf631d41ec45b1ac5
paper_authors.name: Sanghoon Kim
paper_authors._id: 6606447bf631d41ec45b1ac6
paper_authors.name: Chanjun Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:21:22.718Z
paper_authors._id: 64b9fd6910430817357cd9b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9fd6910430817357cd9b2/Zm_Ngp20IlHlLIOFlNXPo.jpeg
paper_authors.fullname: Chanjun Park
paper_authors.type: user
paper_authors.user: Chanjun
paper_title: sDPO: Don't Use Your Data All at Once
paper_summary: As development of large language models (LLM) progresses, aligning them with
  human preferences has become increasingly important. We propose stepwise DPO
  (sDPO), an extension of the recently popularized direct preference optimization
  (DPO) for alignment tuning. This approach involves dividing the available
  preference datasets and utilizing them in a stepwise manner, rather than
  employing it all at once. We demonstrate that this method facilitates the use
  of more precisely aligned reference models within the DPO training framework.
  Furthermore, sDPO trains the final model to be more performant, even
  outperforming other popular LLMs with more parameters.
paper_ai_keywords: stepwise DPO direct preference optimization preference datasets reference models