Self-Boosting Large Language Models with Synthetic Preference Data
Through alignment with human preferences, Large Language Models (LLMs) have
advanced significantly in generating honest, harmless, and helpful responses.
However, collecting high-quality preference data is a resource-intensive and
creativity-demanding process, especially for the continual improvement of LLMs.
We introduce SynPO, a self-boosting paradigm that leverages synthetic
preference data for model alignment. SynPO employs an iterative mechanism
wherein a self-prompt generator creates diverse prompts, and a response
improver refines model responses progressively. This approach trains LLMs to
autonomously learn the generative rewards for their own outputs and eliminates
the need for large-scale annotation of prompts and human preferences. After
four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
in instruction-following abilities, achieving over 22.1% win rate improvements
on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
increase on the well-recognized Open LLM leaderboard.
6707453f257e19aee32b0a15
Qingxiu Dong
admin_assigned
2024-10-10T09:02:41.139Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Qingxiu Dong
user
Rsy24
6707453f257e19aee32b0a16
Li Dong
admin_assigned
2024-10-10T14:09:20.654Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
6707453f257e19aee32b0a17
Xingxing Zhang
admin_assigned
2024-10-10T09:03:01.763Z
64abbcff6cadc7aca584f71b
/avatars/fc6e85ad4a8befd133a37b411712c648.svg
Xingxing Zhang
user
THU-CHUNXIA
6707453f257e19aee32b0a18
Zhifang Sui
6707453f257e19aee32b0a19
Furu Wei
admin_assigned
2024-10-10T09:03:14.088Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Self-Boosting Large Language Models with Synthetic Preference Data
Through alignment with human preferences, Large Language Models (LLMs) have
advanced significantly in generating honest, harmless, and helpful responses.
However, collecting high-quality preference data is a resource-intensive and
creativity-demanding process, especially for the continual improvement of LLMs.
We introduce SynPO, a self-boosting paradigm that leverages synthetic
preference data for model alignment. SynPO employs an iterative mechanism
wherein a self-prompt generator creates diverse prompts, and a response
improver refines model responses progressively. This approach trains LLMs to
autonomously learn the generative rewards for their own outputs and eliminates
the need for large-scale annotation of prompts and human preferences. After
four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
in instruction-following abilities, achieving over 22.1% win rate improvements
on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
increase on the well-recognized Open LLM leaderboard.
SynPO, a self-boosting paradigm using synthetic preference data, enhances LLMs' instruction-following abilities and general performance without extensive human annotation.
Large Language Models LLMs SynPO self-boosting synthetic preference data self-prompt generator response improver generative rewards instruction-following abilities AlpacaEval ArenaHard Open LLM leaderboard