title: Xwin-LM: Strong and Scalable Alignment Practice for LLMs
summary: In this work, we present Xwin-LM, a comprehensive suite of alignment
  methodologies for large language models (LLMs). This suite encompasses several
  key techniques, including supervised finetuning (SFT), reward modeling (RM),
  rejection sampling finetuning (RS), and direct preference optimization (DPO).
  The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
  with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
  preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
  models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
  parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
  is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
  Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
  from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
  DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
  consistent and significant improvements across the pipeline, demonstrating the
  strength and scalability of Xwin-LM. The repository
  https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster
  community research.
paper_authors._id: 66594d86f1b4f073d80378d2
paper_authors.name: Bolin Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:52:41.072Z
paper_authors._id: 642149592cc2b3c39e7ef6c9
paper_authors.avatarUrl: /avatars/fed5d447375cd8c3b2bc9e096fe681de.svg
paper_authors.fullname: Bolin Ni
paper_authors.type: user
paper_authors.user: nbl97
paper_authors._id: 66594d86f1b4f073d80378d3
paper_authors.name: JingCheng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:52:47.834Z
paper_authors._id: 625026b7d2d191ac43320c5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
paper_authors.fullname: Jingcheng Hu
paper_authors.type: user
paper_authors.user: reign12
paper_authors._id: 66594d86f1b4f073d80378d4
paper_authors.name: Yixuan Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:53:01.864Z
paper_authors._id: 640efe54a92fedb0e84ee598
paper_authors.avatarUrl: /avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
paper_authors.fullname: Yixuan Wei
paper_authors.type: user
paper_authors.user: EasonWei
paper_authors._id: 66594d86f1b4f073d80378d5
paper_authors.name: Houwen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:53:10.851Z
paper_authors._id: 631952b6f18d0b5d999ca397
paper_authors.avatarUrl: /avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
paper_authors.fullname: Peng
paper_authors.type: user
paper_authors.user: Houwen
paper_authors._id: 66594d86f1b4f073d80378d6
paper_authors.name: Zheng Zhang
paper_authors._id: 66594d86f1b4f073d80378d7
paper_authors.name: Gaofeng Meng
paper_authors._id: 66594d86f1b4f073d80378d8
paper_authors.name: Han Hu
paper_title: Xwin-LM: Strong and Scalable Alignment Practice for LLMs
paper_summary: In this work, we present Xwin-LM, a comprehensive suite of alignment
  methodologies for large language models (LLMs). This suite encompasses several
  key techniques, including supervised finetuning (SFT), reward modeling (RM),
  rejection sampling finetuning (RS), and direct preference optimization (DPO).
  The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
  with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
  preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
  models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
  parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
  is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
  Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
  from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
  DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
  consistent and significant improvements across the pipeline, demonstrating the
  strength and scalability of Xwin-LM. The repository
  https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster
  community research.
paper_ai_keywords: supervised finetuning (SFT) reward modeling (RM) rejection sampling finetuning (RS) direct preference optimization (DPO) large language models (LLMs) Xwin-LM-SFT Xwin-Pair Xwin-RM Xwin-Set Xwin-LM-RS Xwin-LM-DPO AlpacaEval MT-bench