title: RePO: ReLU-based Preference Optimization
summary: Aligning large language models (LLMs) with human preferences is critical for
  real-world deployment, yet existing methods like RLHF face computational and
  stability challenges. While DPO establishes an offline paradigm with single
  hyperparameter beta, subsequent methods like SimPO reintroduce complexity
  through dual parameters (beta, gamma). We propose {ReLU-based Preference
  Optimization (RePO)}, a streamlined algorithm that eliminates beta via two
  advances: (1) retaining SimPO's reference-free margins but removing beta
  through gradient analysis, and (2) adopting a ReLU-based max-margin loss that
  naturally filters trivial pairs. Theoretically, RePO is characterized as
  SimPO's limiting case (beta to infty), where the logistic weighting
  collapses to binary thresholding, forming a convex envelope of the 0-1 loss.
  Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO
  and SimPO across multiple base models, requiring only one hyperparameter to
  tune.
paper_authors._id: 67cff321f2b1fe815dce3722
paper_authors.name: Junkang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:16:45.903Z
paper_authors._id: 664764e5d834283e7ff96d37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/664764e5d834283e7ff96d37/JBi3dvAUKqqGf_DjVORK7.jpeg
paper_authors.fullname: Junkang Wu
paper_authors.type: user
paper_authors.user: junkang0909
paper_authors._id: 67cff321f2b1fe815dce3723
paper_authors.name: Kexin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:16:08.313Z
paper_authors._id: 631fc126788f34547a9d7ee1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674202999503-631fc126788f34547a9d7ee1.png
paper_authors.fullname: Kexin Huang
paper_authors.type: user
paper_authors.user: kexinhuang
paper_authors._id: 67cff321f2b1fe815dce3724
paper_authors.name: Xue Wang
paper_authors._id: 67cff321f2b1fe815dce3725
paper_authors.name: Jinyang Gao
paper_authors._id: 67cff321f2b1fe815dce3726
paper_authors.name: Bolin Ding
paper_authors._id: 67cff321f2b1fe815dce3727
paper_authors.name: Jiancan Wu
paper_authors._id: 67cff321f2b1fe815dce3728
paper_authors.name: Xiangnan He
paper_authors._id: 67cff321f2b1fe815dce3729
paper_authors.name: Xiang Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T08:24:02.839Z
paper_authors._id: 65fca775fa59bdf4737b1a84
paper_authors.avatarUrl: /avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
paper_authors.fullname: Xiang Wang
paper_authors.type: user
paper_authors.user: xiangwang1223
paper_title: RePO: ReLU-based Preference Optimization
paper_summary: Aligning large language models (LLMs) with human preferences is critical for
  real-world deployment, yet existing methods like RLHF face computational and
  stability challenges. While DPO establishes an offline paradigm with single
  hyperparameter beta, subsequent methods like SimPO reintroduce complexity
  through dual parameters (beta, gamma). We propose {ReLU-based Preference
  Optimization (RePO)}, a streamlined algorithm that eliminates beta via two
  advances: (1) retaining SimPO's reference-free margins but removing beta
  through gradient analysis, and (2) adopting a ReLU-based max-margin loss that
  naturally filters trivial pairs. Theoretically, RePO is characterized as
  SimPO's limiting case (beta to infty), where the logistic weighting
  collapses to binary thresholding, forming a convex envelope of the 0-1 loss.
  Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO
  and SimPO across multiple base models, requiring only one hyperparameter to
  tune.
paper_ai_keywords: LLMS RLHF DPO SimPO ReLU-based Preference Optimization (RePO) reference-free margins gradient analysis ReLU-based max-margin loss convex envelope 0-1 loss