title: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
    Models
summary: Harnessing the power of human-annotated data through Supervised Fine-Tuning
  (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we
  delve into the prospect of growing a strong LLM out of a weak one without the
  need for acquiring additional human-annotated data. We propose a new
  fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a
  supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,
  where the LLM refines its capability by playing against instances of itself.
  More specifically, the LLM generates its own training data from its previous
  iterations, refining its policy by discerning these self-generated responses
  from those obtained from human-annotated data. Our method progressively
  elevates the LLM from a nascent model to a formidable one, unlocking the full
  potential of human-annotated demonstration data for SFT. Theoretically, we
  prove that the global optimum to the training objective function of our method
  is achieved only when the LLM policy aligns with the target data distribution.
  Empirically, we evaluate our method on several benchmark datasets including the
  HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our
  results show that SPIN can significantly improve the LLM's performance across a
  variety of benchmarks and even outperform models trained through direct
  preference optimization (DPO) supplemented with extra GPT-4 preference data.
  This sheds light on the promise of self-play, enabling the achievement of
  human-level performance in LLMs without the need for expert opponents.
paper_authors._id: 6594ba845070805494045542
paper_authors.name: Zixiang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T13:02:50.615Z
paper_authors._id: 64335a225277e3b24ef3de1a
paper_authors.avatarUrl: /avatars/535fc39476b46fe6f589dcd4d7ca7f01.svg
paper_authors.fullname: Zixiang Chen
paper_authors.type: user
paper_authors.user: chenzx19
paper_authors._id: 6594ba845070805494045543
paper_authors.name: Yihe Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-03T12:34:15.847Z
paper_authors._id: 642f4c789b2484d7d8551a93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
paper_authors.fullname: Yihe Deng
paper_authors.type: user
paper_authors.user: ydeng9
paper_authors._id: 6594ba845070805494045544
paper_authors.name: Huizhuo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:35:20.039Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 6594ba845070805494045545
paper_authors.name: Kaixuan Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T13:03:08.926Z
paper_authors._id: 656c1018e8bf55919a8234b0
paper_authors.avatarUrl: /avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
paper_authors.fullname: Kaixuan Ji
paper_authors.type: user
paper_authors.user: KaixuanJi
paper_authors._id: 6594ba845070805494045546
paper_authors.name: Quanquan Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-03T12:34:09.757Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
    Models
paper_summary: Harnessing the power of human-annotated data through Supervised Fine-Tuning
  (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we
  delve into the prospect of growing a strong LLM out of a weak one without the
  need for acquiring additional human-annotated data. We propose a new
  fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a
  supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,
  where the LLM refines its capability by playing against instances of itself.
  More specifically, the LLM generates its own training data from its previous
  iterations, refining its policy by discerning these self-generated responses
  from those obtained from human-annotated data. Our method progressively
  elevates the LLM from a nascent model to a formidable one, unlocking the full
  potential of human-annotated demonstration data for SFT. Theoretically, we
  prove that the global optimum to the training objective function of our method
  is achieved only when the LLM policy aligns with the target data distribution.
  Empirically, we evaluate our method on several benchmark datasets including the
  HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our
  results show that SPIN can significantly improve the LLM's performance across a
  variety of benchmarks and even outperform models trained through direct
  preference optimization (DPO) supplemented with extra GPT-4 preference data.
  This sheds light on the promise of self-play, enabling the achievement of
  human-level performance in LLMs without the need for expert opponents.
paper_ai_keywords: Supervised Fine-Tuning (SFT) Large Language Models (LLMs) Self-Play fIne-tuNing (SPIN) self-play mechanism self-generated responses target data distribution HuggingFace Open LLM Leaderboard MT-Bench Big-Bench direct preference optimization (DPO) GPT-4 preference data