title: The Differences Between Direct Alignment Algorithms are a Blur
summary: Direct Alignment Algorithms (DAAs) simplify language model alignment by
  replacing reinforcement learning (RL) and reward modeling (RM) in Reinforcement
  Learning from Human Feedback (RLHF) with direct policy optimization. DAAs can
  be classified by their ranking losses (pairwise vs. pointwise), by the rewards
  used in those losses (e.g., likelihood ratios of policy and reference policy,
  or odds ratios), or by whether a Supervised Fine-Tuning (SFT) phase is required
  (two-stage vs. one-stage). We first show that one-stage methods underperform
  two-stage methods. To address this, we incorporate an explicit SFT phase and
  introduce the beta parameter, controlling the strength of preference
  optimization, into single-stage ORPO and ASFT. These modifications improve
  their performance in Alpaca Eval 2 by +3.46 (ORPO) and +8.27 (ASFT),
  matching two-stage methods like DPO. Further analysis reveals that the key
  factor is whether the approach uses pairwise or pointwise objectives, rather
  than the specific implicit reward or loss function. These results highlight the
  importance of careful evaluation to avoid premature claims of performance gains
  or overall superiority in alignment algorithms.
paper_authors._id: 67a1c1428747511e7b9a1965
paper_authors.name: Alexey Gorbatovski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:00.767Z
paper_authors._id: 62897fce5d9e25c10e4f319d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
paper_authors.fullname: Alexey Gorbatovski
paper_authors.type: user
paper_authors.user: Myashka
paper_authors._id: 67a1c1428747511e7b9a1966
paper_authors.name: Boris Shaposhnikov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T14:48:32.720Z
paper_authors._id: 637dd11dcbad6e62a5e39743
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
paper_authors.fullname: Boris Shaposhnikov
paper_authors.type: user
paper_authors.user: borisshapa
paper_authors._id: 67a1c1428747511e7b9a1967
paper_authors.name: Viacheslav Sinii
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:38:52.039Z
paper_authors._id: 6416272d986557e8cac64ece
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6416272d986557e8cac64ece/s3CLjNN_pGj-vJDcENFD2.jpeg
paper_authors.fullname: Viacheslav
paper_authors.type: user
paper_authors.user: ummagumm-a
paper_authors._id: 67a1c1428747511e7b9a1968
paper_authors.name: Alexey Malakhov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:38:54.121Z
paper_authors._id: 636e71b2b0ebc04888157b71
paper_authors.avatarUrl: /avatars/957ba705d470e3a01792741d7f0ff038.svg
paper_authors.fullname: Alexey Malakhov
paper_authors.type: user
paper_authors.user: ZeL1k7
paper_authors._id: 67a1c1428747511e7b9a1969
paper_authors.name: Daniil Gavrilov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:38:57.087Z
paper_authors._id: 62a9c8edc19f92ae443ab37f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
paper_authors.fullname: Daniil Gavrilov
paper_authors.type: user
paper_authors.user: kefirski
paper_title: The Differences Between Direct Alignment Algorithms are a Blur
paper_summary: Direct Alignment Algorithms (DAAs) simplify language model alignment by
  replacing reinforcement learning (RL) and reward modeling (RM) in Reinforcement
  Learning from Human Feedback (RLHF) with direct policy optimization. DAAs can
  be classified by their ranking losses (pairwise vs. pointwise), by the rewards
  used in those losses (e.g., likelihood ratios of policy and reference policy,
  or odds ratios), or by whether a Supervised Fine-Tuning (SFT) phase is required
  (two-stage vs. one-stage). We first show that one-stage methods underperform
  two-stage methods. To address this, we incorporate an explicit SFT phase and
  introduce the beta parameter, controlling the strength of preference
  optimization, into single-stage ORPO and ASFT. These modifications improve
  their performance in Alpaca Eval 2 by +3.46 (ORPO) and +8.27 (ASFT),
  matching two-stage methods like DPO. Further analysis reveals that the key
  factor is whether the approach uses pairwise or pointwise objectives, rather
  than the specific implicit reward or loss function. These results highlight the
  importance of careful evaluation to avoid premature claims of performance gains
  or overall superiority in alignment algorithms.
paper_ai_keywords: Direct Alignment Algorithms (DAAs) reinforcement learning (RL) reward modeling (RM) Reinforcement Learning from Human Feedback (RLHF) direct policy optimization ranking losses pairwise pointwise likelihood ratios odds ratios Supervised Fine-Tuning (SFT) one-stage two-stage $\beta$ parameter reference optimization single-stage ORPO single-stage ASFT DPO Alpaca Eval 2 pairwise objectives pointwise objectives