title: Towards a Unified View of Preference Learning for Large Language Models:
    A Survey
summary: Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
  the crucial factors to achieve success is aligning the LLM's output with human
  preferences. This alignment process often requires only a small amount of data
  to efficiently enhance the LLM's performance. While effective, research in this
  area spans multiple domains, and the methods involved are relatively complex to
  understand. The relationships between different methods have been
  under-explored, limiting the development of the preference alignment. In light
  of this, we break down the existing popular alignment strategies into different
  components and provide a unified framework to study the current alignment
  strategies, thereby establishing connections among them. In this survey, we
  decompose all the strategies in preference learning into four components:
  model, data, feedback, and algorithm. This unified view offers an in-depth
  understanding of existing alignment algorithms and also opens up possibilities
  to synergize the strengths of different strategies. Furthermore, we present
  detailed working examples of prevalent existing algorithms to facilitate a
  comprehensive understanding for the readers. Finally, based on our unified
  perspective, we explore the challenges and future research directions for
  aligning large language models with human preferences.
paper_authors._id: 66dec9c1140f04eb189084cb
paper_authors.name: Bofei Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T10:57:57.861Z
paper_authors._id: 65ae21adabf6d1ccb795e9a4
paper_authors.avatarUrl: /avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
paper_authors.fullname: Bofei Gao
paper_authors.type: user
paper_authors.user: KbsdJames
paper_authors._id: 66dec9c1140f04eb189084cc
paper_authors.name: Feifan Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:47.570Z
paper_authors._id: 6447ca6ca478b20f1755b294
paper_authors.avatarUrl: /avatars/5049856b5ed1b74533fff902e14b4c7c.svg
paper_authors.fullname: Feifan Song
paper_authors.type: user
paper_authors.user: songff
paper_authors._id: 66dec9c1140f04eb189084cd
paper_authors.name: Yibo Miao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:50:58.165Z
paper_authors._id: 64a139c098fad0c8a5a627a4
paper_authors.avatarUrl: /avatars/6eb508abd827d4a4f5abb6b24155b22d.svg
paper_authors.fullname: Yibo Miao
paper_authors.type: user
paper_authors.user: instro
paper_authors._id: 66dec9c1140f04eb189084ce
paper_authors.name: Zefan Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:51:04.590Z
paper_authors._id: 64b15284372d4340772a3dca
paper_authors.avatarUrl: /avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
paper_authors.fullname: Zefan Cai
paper_authors.type: user
paper_authors.user: ZefanCai
paper_authors._id: 66dec9c1140f04eb189084cf
paper_authors.name: Zhe Yang
paper_authors._id: 66dec9c1140f04eb189084d0
paper_authors.name: Liang Chen
paper_authors._id: 66dec9c1140f04eb189084d1
paper_authors.name: Helan Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:51:31.092Z
paper_authors._id: 6491aec41e161baf7894b54b
paper_authors.avatarUrl: /avatars/0af10e24669125e4329c003fc27b603e.svg
paper_authors.fullname: huhelan
paper_authors.type: user
paper_authors.user: hhhllan
paper_authors._id: 66dec9c1140f04eb189084d2
paper_authors.name: Runxin Xu
paper_authors._id: 66dec9c1140f04eb189084d3
paper_authors.name: Qingxiu Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:02:20.041Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 66dec9c1140f04eb189084d4
paper_authors.name: Ce Zheng
paper_authors._id: 66dec9c1140f04eb189084d5
paper_authors.name: Wen Xiao
paper_authors._id: 66dec9c1140f04eb189084d6
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T08:37:45.705Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66dec9c1140f04eb189084d7
paper_authors.name: Daoguang Zan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:53:15.350Z
paper_authors._id: 61527edf8b55dbdae72874fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
paper_authors.fullname: Daoguang Zan
paper_authors.type: user
paper_authors.user: Daoguang
paper_authors._id: 66dec9c1140f04eb189084d8
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:53:08.301Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 66dec9c1140f04eb189084d9
paper_authors.name: Bowen Yu
paper_authors._id: 66dec9c1140f04eb189084da
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:00:23.780Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 66dec9c1140f04eb189084db
paper_authors.name: Zeyu Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T14:22:33.625Z
paper_authors._id: 643fb14a18afbc4d1f3ebfb4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643fb14a18afbc4d1f3ebfb4/2qeL_qPSB9_MTzZf46ynJ.png
paper_authors.fullname: czy yente
paper_authors.type: user
paper_authors.user: cyente
paper_authors._id: 66dec9c1140f04eb189084dc
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:03:37.313Z
paper_authors._id: 64ccb9bfead94891d12aef42
paper_authors.avatarUrl: /avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
paper_authors.fullname: Yang Jian
paper_authors.type: user
paper_authors.user: CSJianYang
paper_authors._id: 66dec9c1140f04eb189084dd
paper_authors.name: Lei Sha
paper_authors._id: 66dec9c1140f04eb189084de
paper_authors.name: Houfeng Wang
paper_authors._id: 66dec9c1140f04eb189084df
paper_authors.name: Zhifang Sui
paper_authors._id: 66dec9c1140f04eb189084e0
paper_authors.name: Peiyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:51:54.896Z
paper_authors._id: 656873f33fd0bf1f82558695
paper_authors.avatarUrl: /avatars/7a085da2e2a91d7f41988501a573ebf9.svg
paper_authors.fullname: PEIYI, WANG
paper_authors.type: user
paper_authors.user: peiyiwang89
paper_authors._id: 66dec9c1140f04eb189084e1
paper_authors.name: Tianyu Liu
paper_authors._id: 66dec9c1140f04eb189084e2
paper_authors.name: Baobao Chang
paper_title: Towards a Unified View of Preference Learning for Large Language Models:
    A Survey
paper_summary: Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
  the crucial factors to achieve success is aligning the LLM's output with human
  preferences. This alignment process often requires only a small amount of data
  to efficiently enhance the LLM's performance. While effective, research in this
  area spans multiple domains, and the methods involved are relatively complex to
  understand. The relationships between different methods have been
  under-explored, limiting the development of the preference alignment. In light
  of this, we break down the existing popular alignment strategies into different
  components and provide a unified framework to study the current alignment
  strategies, thereby establishing connections among them. In this survey, we
  decompose all the strategies in preference learning into four components:
  model, data, feedback, and algorithm. This unified view offers an in-depth
  understanding of existing alignment algorithms and also opens up possibilities
  to synergize the strengths of different strategies. Furthermore, we present
  detailed working examples of prevalent existing algorithms to facilitate a
  comprehensive understanding for the readers. Finally, based on our unified
  perspective, we explore the challenges and future research directions for
  aligning large language models with human preferences.