Towards a Unified View of Preference Learning for Large Language Models:
  A Survey
Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
the crucial factors to achieve success is aligning the LLM's output with human
preferences. This alignment process often requires only a small amount of data
to efficiently enhance the LLM's performance. While effective, research in this
area spans multiple domains, and the methods involved are relatively complex to
understand. The relationships between different methods have been
under-explored, limiting the development of the preference alignment. In light
of this, we break down the existing popular alignment strategies into different
components and provide a unified framework to study the current alignment
strategies, thereby establishing connections among them. In this survey, we
decompose all the strategies in preference learning into four components:
model, data, feedback, and algorithm. This unified view offers an in-depth
understanding of existing alignment algorithms and also opens up possibilities
to synergize the strengths of different strategies. Furthermore, we present
detailed working examples of prevalent existing algorithms to facilitate a
comprehensive understanding for the readers. Finally, based on our unified
perspective, we explore the challenges and future research directions for
aligning large language models with human preferences.
66dec9c1140f04eb189084cb
Bofei Gao
claimed_verified
2024-09-09T10:57:57.861Z
65ae21adabf6d1ccb795e9a4
/avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
Bofei Gao
user
KbsdJames
66dec9c1140f04eb189084cc
Feifan Song
claimed_verified
2024-09-10T07:10:47.570Z
6447ca6ca478b20f1755b294
/avatars/5049856b5ed1b74533fff902e14b4c7c.svg
Feifan Song
user
songff
66dec9c1140f04eb189084cd
Yibo Miao
admin_assigned
2024-09-10T08:50:58.165Z
64a139c098fad0c8a5a627a4
https://cdn-avatars.huggingface.co/v1/production/uploads/64a139c098fad0c8a5a627a4/MgWortQS64cJOZ3pL4WyH.jpeg
Yibo Miao
user
instro
66dec9c1140f04eb189084ce
Zefan Cai
admin_assigned
2024-09-10T08:51:04.590Z
64b15284372d4340772a3dca
/avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
Zefan Cai
user
ZefanCai
66dec9c1140f04eb189084cf
Zhe Yang
66dec9c1140f04eb189084d0
Liang Chen
66dec9c1140f04eb189084d1
Helan Hu
admin_assigned
2024-09-10T08:51:31.092Z
6491aec41e161baf7894b54b
/avatars/0af10e24669125e4329c003fc27b603e.svg
huhelan
user
hhhllan
66dec9c1140f04eb189084d2
Runxin Xu
66dec9c1140f04eb189084d3
Qingxiu Dong
claimed_verified
2024-10-10T08:02:20.041Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Rosy24
user
Rsy24
66dec9c1140f04eb189084d4
Ce Zheng
66dec9c1140f04eb189084d5
Wen Xiao
66dec9c1140f04eb189084d6
Ge Zhang
claimed_verified
2024-09-10T08:37:45.705Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66dec9c1140f04eb189084d7
Daoguang Zan
admin_assigned
2024-09-10T08:53:15.350Z
61527edf8b55dbdae72874fa
https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
Daoguang Zan
user
Daoguang
66dec9c1140f04eb189084d8
Keming Lu
admin_assigned
2024-09-10T08:53:08.301Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
66dec9c1140f04eb189084d9
Bowen Yu
66dec9c1140f04eb189084da
Dayiheng Liu
admin_assigned
2024-09-10T09:00:23.780Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
66dec9c1140f04eb189084db
Zeyu Cui
claimed_verified
2024-09-23T14:22:33.625Z
643fb14a18afbc4d1f3ebfb4
https://cdn-avatars.huggingface.co/v1/production/uploads/643fb14a18afbc4d1f3ebfb4/2qeL_qPSB9_MTzZf46ynJ.png
czy yente
user
cyente
66dec9c1140f04eb189084dc
Jian Yang
claimed_verified
2024-10-14T19:03:37.313Z
64ccb9bfead94891d12aef42
/avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
Yang Jian
user
CSJianYang
66dec9c1140f04eb189084dd
Lei Sha
66dec9c1140f04eb189084de
Houfeng Wang
66dec9c1140f04eb189084df
Zhifang Sui
66dec9c1140f04eb189084e0
Peiyi Wang
admin_assigned
2024-09-10T08:51:54.896Z
656873f33fd0bf1f82558695
/avatars/7a085da2e2a91d7f41988501a573ebf9.svg
PEIYI, WANG
user
peiyiwang89
66dec9c1140f04eb189084e1
Tianyu Liu
66dec9c1140f04eb189084e2
Baobao Chang
Towards a Unified View of Preference Learning for Large Language Models:
  A Survey
Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
the crucial factors to achieve success is aligning the LLM's output with human
preferences. This alignment process often requires only a small amount of data
to efficiently enhance the LLM's performance. While effective, research in this
area spans multiple domains, and the methods involved are relatively complex to
understand. The relationships between different methods have been
under-explored, limiting the development of the preference alignment. In light
of this, we break down the existing popular alignment strategies into different
components and provide a unified framework to study the current alignment
strategies, thereby establishing connections among them. In this survey, we
decompose all the strategies in preference learning into four components:
model, data, feedback, and algorithm. This unified view offers an in-depth
understanding of existing alignment algorithms and also opens up possibilities
to synergize the strengths of different strategies. Furthermore, we present
detailed working examples of prevalent existing algorithms to facilitate a
comprehensive understanding for the readers. Finally, based on our unified
perspective, we explore the challenges and future research directions for
aligning large language models with human preferences.