title: Pre-DPO: Improving Data Utilization in Direct Preference Optimization
    Using a Guiding Reference Model
summary: Direct Preference Optimization (DPO) simplifies reinforcement learning from
  human feedback (RLHF) for large language models (LLMs) by directly optimizing
  human preferences without an explicit reward model. We find that during DPO
  training, the reference model plays the role of a data weight adjuster.
  However, the common practice of initializing the policy and reference models
  identically in DPO can lead to inefficient data utilization and impose a
  performance ceiling. Meanwhile, the lack of a reference model in Simple
  Preference Optimization (SimPO) reduces training robustness and necessitates
  stricter conditions to prevent catastrophic forgetting. In this work, we
  propose Pre-DPO, a simple yet effective DPO-based training paradigm that
  enhances preference optimization performance by leveraging a guiding reference
  model. This reference model provides foresight into the optimal policy state
  achievable through the training preference data, serving as a guiding mechanism
  that adaptively assigns higher weights to samples more suitable for the model
  and lower weights to those less suitable. Extensive experiments on AlpacaEval
  2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently
  improves the performance of both DPO and SimPO, without relying on external
  models or additional data.
paper_authors._id: 6809948944114def75aaeb7d
paper_authors.name: Junshu Pan
paper_authors._id: 6809948944114def75aaeb7e
paper_authors.name: Wei Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-24T11:26:03.812Z
paper_authors._id: 6468823272d9180d4ac90bdf
paper_authors.avatarUrl: /avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
paper_authors.fullname: Wei Shen
paper_authors.type: user
paper_authors.user: Swtheking
paper_authors._id: 6809948944114def75aaeb7f
paper_authors.name: Shulin Huang
paper_authors._id: 6809948944114def75aaeb80
paper_authors.name: Qiji Zhou
paper_authors._id: 6809948944114def75aaeb81
paper_authors.name: Yue Zhang
paper_title: Pre-DPO: Improving Data Utilization in Direct Preference Optimization
    Using a Guiding Reference Model
paper_summary: Direct Preference Optimization (DPO) simplifies reinforcement learning from
  human feedback (RLHF) for large language models (LLMs) by directly optimizing
  human preferences without an explicit reward model. We find that during DPO
  training, the reference model plays the role of a data weight adjuster.
  However, the common practice of initializing the policy and reference models
  identically in DPO can lead to inefficient data utilization and impose a
  performance ceiling. Meanwhile, the lack of a reference model in Simple
  Preference Optimization (SimPO) reduces training robustness and necessitates
  stricter conditions to prevent catastrophic forgetting. In this work, we
  propose Pre-DPO, a simple yet effective DPO-based training paradigm that
  enhances preference optimization performance by leveraging a guiding reference
  model. This reference model provides foresight into the optimal policy state
  achievable through the training preference data, serving as a guiding mechanism
  that adaptively assigns higher weights to samples more suitable for the model
  and lower weights to those less suitable. Extensive experiments on AlpacaEval
  2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently
  improves the performance of both DPO and SimPO, without relying on external
  models or additional data.
paper_ai_summary: Pre-DPO enhances preference optimization in RLHF for LLMs by using a guiding reference model to improve data utilization and performance.
paper_ai_keywords: Direct Preference Optimization DPO reinforcement learning from human feedback RLHF large language models LLMs reference model Simple Preference Optimization SimPO preference optimization AlpacaEval Arena-Hard