title: Self-Boosting Large Language Models with Synthetic Preference Data
summary: Through alignment with human preferences, Large Language Models (LLMs) have
  advanced significantly in generating honest, harmless, and helpful responses.
  However, collecting high-quality preference data is a resource-intensive and
  creativity-demanding process, especially for the continual improvement of LLMs.
  We introduce SynPO, a self-boosting paradigm that leverages synthetic
  preference data for model alignment. SynPO employs an iterative mechanism
  wherein a self-prompt generator creates diverse prompts, and a response
  improver refines model responses progressively. This approach trains LLMs to
  autonomously learn the generative rewards for their own outputs and eliminates
  the need for large-scale annotation of prompts and human preferences. After
  four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
  in instruction-following abilities, achieving over 22.1% win rate improvements
  on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
  performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
  increase on the well-recognized Open LLM leaderboard.
paper_authors._id: 6707453f257e19aee32b0a15
paper_authors.name: Qingxiu Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:02:41.139Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 6707453f257e19aee32b0a16
paper_authors.name: Li Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T14:09:20.654Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 6707453f257e19aee32b0a17
paper_authors.name: Xingxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:03:01.763Z
paper_authors._id: 64abbcff6cadc7aca584f71b
paper_authors.avatarUrl: /avatars/fc6e85ad4a8befd133a37b411712c648.svg
paper_authors.fullname: Xingxing Zhang
paper_authors.type: user
paper_authors.user: THU-CHUNXIA
paper_authors._id: 6707453f257e19aee32b0a18
paper_authors.name: Zhifang Sui
paper_authors._id: 6707453f257e19aee32b0a19
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:03:14.088Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Self-Boosting Large Language Models with Synthetic Preference Data
paper_summary: Through alignment with human preferences, Large Language Models (LLMs) have
  advanced significantly in generating honest, harmless, and helpful responses.
  However, collecting high-quality preference data is a resource-intensive and
  creativity-demanding process, especially for the continual improvement of LLMs.
  We introduce SynPO, a self-boosting paradigm that leverages synthetic
  preference data for model alignment. SynPO employs an iterative mechanism
  wherein a self-prompt generator creates diverse prompts, and a response
  improver refines model responses progressively. This approach trains LLMs to
  autonomously learn the generative rewards for their own outputs and eliminates
  the need for large-scale annotation of prompts and human preferences. After
  four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
  in instruction-following abilities, achieving over 22.1% win rate improvements
  on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
  performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
  increase on the well-recognized Open LLM leaderboard.
paper_ai_summary: SynPO, a self-boosting paradigm using synthetic preference data, enhances LLMs' instruction-following abilities and general performance without extensive human annotation.
paper_ai_keywords: Large Language Models LLMs SynPO self-boosting synthetic preference data self-prompt generator response improver generative rewards instruction-following abilities AlpacaEval ArenaHard Open LLM leaderboard