The Differences Between Direct Alignment Algorithms are a Blur
Direct Alignment Algorithms (DAAs) simplify language model alignment by
replacing reinforcement learning (RL) and reward modeling (RM) in Reinforcement
Learning from Human Feedback (RLHF) with direct policy optimization. DAAs can
be classified by their ranking losses (pairwise vs. pointwise), by the rewards
used in those losses (e.g., likelihood ratios of policy and reference policy,
or odds ratios), or by whether a Supervised Fine-Tuning (SFT) phase is required
(two-stage vs. one-stage). We first show that one-stage methods underperform
two-stage methods. To address this, we incorporate an explicit SFT phase and
introduce the beta parameter, controlling the strength of preference
optimization, into single-stage ORPO and ASFT. These modifications improve
their performance in Alpaca Eval 2 by +3.46 (ORPO) and +8.27 (ASFT),
matching two-stage methods like DPO. Further analysis reveals that the key
factor is whether the approach uses pairwise or pointwise objectives, rather
than the specific implicit reward or loss function. These results highlight the
importance of careful evaluation to avoid premature claims of performance gains
or overall superiority in alignment algorithms.
67a1c1428747511e7b9a1965
Alexey Gorbatovski
claimed_verified
2025-02-04T09:39:00.767Z
62897fce5d9e25c10e4f319d
https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
Alexey Gorbatovski
user
Myashka
67a1c1428747511e7b9a1966
Boris Shaposhnikov
claimed_verified
2025-02-04T14:48:32.720Z
637dd11dcbad6e62a5e39743
https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
Boris Shaposhnikov
user
borisshapa
67a1c1428747511e7b9a1967
Viacheslav Sinii
claimed_verified
2025-02-04T09:38:52.039Z
6416272d986557e8cac64ece
https://cdn-avatars.huggingface.co/v1/production/uploads/6416272d986557e8cac64ece/s3CLjNN_pGj-vJDcENFD2.jpeg
Viacheslav
user
ummagumm-a
67a1c1428747511e7b9a1968
Alexey Malakhov
claimed_verified
2025-02-04T09:38:54.121Z
636e71b2b0ebc04888157b71
/avatars/957ba705d470e3a01792741d7f0ff038.svg
Alexey Malakhov
user
ZeL1k7
67a1c1428747511e7b9a1969
Daniil Gavrilov
claimed_verified
2025-02-04T09:38:57.087Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
The Differences Between Direct Alignment Algorithms are a Blur
Direct Alignment Algorithms (DAAs) simplify language model alignment by
replacing reinforcement learning (RL) and reward modeling (RM) in Reinforcement
Learning from Human Feedback (RLHF) with direct policy optimization. DAAs can
be classified by their ranking losses (pairwise vs. pointwise), by the rewards
used in those losses (e.g., likelihood ratios of policy and reference policy,
or odds ratios), or by whether a Supervised Fine-Tuning (SFT) phase is required
(two-stage vs. one-stage). We first show that one-stage methods underperform
two-stage methods. To address this, we incorporate an explicit SFT phase and
introduce the beta parameter, controlling the strength of preference
optimization, into single-stage ORPO and ASFT. These modifications improve
their performance in Alpaca Eval 2 by +3.46 (ORPO) and +8.27 (ASFT),
matching two-stage methods like DPO. Further analysis reveals that the key
factor is whether the approach uses pairwise or pointwise objectives, rather
than the specific implicit reward or loss function. These results highlight the
importance of careful evaluation to avoid premature claims of performance gains
or overall superiority in alignment algorithms.
Direct Alignment Algorithms improve language model alignment by introducing a supervised fine-tuning phase and adjusting preference optimization strength, showing that ranking objectives are crucial for performance.
Direct Alignment Algorithms RLHF direct policy optimization ranking losses likelihood ratios odds ratios Supervised Fine-Tuning $\beta$ parameter single-stage ORPO ASFT pairwise objectives pointwise objectives Alpaca Eval 2 DPO