title: LiPO: Listwise Preference Optimization through Learning-to-Rank
summary: Aligning language models (LMs) with curated human feedback is critical to
  control their behaviors in real-world applications. Several recent policy
  optimization methods, such as DPO and SLiC, serve as promising alternatives to
  the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In
  practice, human feedback often comes in a format of a ranked list over multiple
  responses to amortize the cost of reading prompt. Multiple responses can also
  be ranked by reward models or AI feedback. There lacks such a study on directly
  fitting upon a list of responses. In this work, we formulate the LM alignment
  as a listwise ranking problem and describe the Listwise Preference Optimization
  (LiPO) framework, where the policy can potentially learn more effectively from
  a ranked list of plausible responses given the prompt. This view draws an
  explicit connection to Learning-to-Rank (LTR), where most existing preference
  optimization work can be mapped to existing ranking objectives, especially
  pairwise ones. Following this connection, we provide an examination of ranking
  objectives that are not well studied for LM alignment withDPO and SLiC as
  special cases when list size is two. In particular, we highlight a specific
  method, LiPO-{\lambda}, which leverages a state-of-the-art listwise ranking
  objective and weights each preference pair in a more advanced manner. We show
  that LiPO-{\lambda} can outperform DPO and SLiC by a clear margin on two
  preference alignment tasks.
paper_authors._id: 65c1b67a0ad80fd1f69b12a7
paper_authors.name: Tianqi Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-28T19:29:33.612Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 65c1b67a0ad80fd1f69b12a8
paper_authors.name: Zhen Qin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-18T08:17:53.197Z
paper_authors._id: 64a2bd9245dde1d6ac7b4d81
paper_authors.avatarUrl: /avatars/8a238a832bfcf14e7704a2152b254a0e.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: paulqin
paper_authors._id: 65c1b67a0ad80fd1f69b12a9
paper_authors.name: Junru Wu
paper_authors._id: 65c1b67a0ad80fd1f69b12aa
paper_authors.name: Jiaming Shen
paper_authors._id: 65c1b67a0ad80fd1f69b12ab
paper_authors.name: Misha Khalman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:05.847Z
paper_authors._id: 65c2ad2790703d2dbcf79471
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
paper_authors.fullname: Misha Khalman
paper_authors.type: user
paper_authors.user: khalman
paper_authors._id: 65c1b67a0ad80fd1f69b12ac
paper_authors.name: Rishabh Joshi
paper_authors._id: 65c1b67a0ad80fd1f69b12ad
paper_authors.name: Yao Zhao
paper_authors._id: 65c1b67a0ad80fd1f69b12ae
paper_authors.name: Mohammad Saleh
paper_authors._id: 65c1b67a0ad80fd1f69b12af
paper_authors.name: Simon Baumgartner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:30:17.442Z
paper_authors._id: 6560f697e0a7720b6ae377bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6560f697e0a7720b6ae377bc/8b-p-lQ7KV_VygBJ6Vf3_.jpeg
paper_authors.fullname: Simon Baumgartner
paper_authors.type: user
paper_authors.user: sens3
paper_authors._id: 65c1b67a0ad80fd1f69b12b0
paper_authors.name: Jialu Liu
paper_authors._id: 65c1b67a0ad80fd1f69b12b1
paper_authors.name: Peter J. Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:08.211Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_authors._id: 65c1b67a0ad80fd1f69b12b2
paper_authors.name: Xuanhui Wang
paper_title: LiPO: Listwise Preference Optimization through Learning-to-Rank
paper_summary: Aligning language models (LMs) with curated human feedback is critical to
  control their behaviors in real-world applications. Several recent policy
  optimization methods, such as DPO and SLiC, serve as promising alternatives to
  the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In
  practice, human feedback often comes in a format of a ranked list over multiple
  responses to amortize the cost of reading prompt. Multiple responses can also
  be ranked by reward models or AI feedback. There lacks such a study on directly
  fitting upon a list of responses. In this work, we formulate the LM alignment
  as a listwise ranking problem and describe the Listwise Preference Optimization
  (LiPO) framework, where the policy can potentially learn more effectively from
  a ranked list of plausible responses given the prompt. This view draws an
  explicit connection to Learning-to-Rank (LTR), where most existing preference
  optimization work can be mapped to existing ranking objectives, especially
  pairwise ones. Following this connection, we provide an examination of ranking
  objectives that are not well studied for LM alignment withDPO and SLiC as
  special cases when list size is two. In particular, we highlight a specific
  method, LiPO-{\lambda}, which leverages a state-of-the-art listwise ranking
  objective and weights each preference pair in a more advanced manner. We show
  that LiPO-{\lambda} can outperform DPO and SLiC by a clear margin on two
  preference alignment tasks.
paper_ai_summary: The LiPO framework, which addresses LM alignment as a listwise ranking problem, outperforms DPO and SLiC by effectively learning from ranked lists of responses.
paper_ai_keywords: Listwise Preference Optimization LiPO Learning-to-Rank preference optimization listwise ranking objective LiPO-λ