Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models
Harnessing the power of human-annotated data through Supervised Fine-Tuning
(SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we
delve into the prospect of growing a strong LLM out of a weak one without the
need for acquiring additional human-annotated data. We propose a new
fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a
supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,
where the LLM refines its capability by playing against instances of itself.
More specifically, the LLM generates its own training data from its previous
iterations, refining its policy by discerning these self-generated responses
from those obtained from human-annotated data. Our method progressively
elevates the LLM from a nascent model to a formidable one, unlocking the full
potential of human-annotated demonstration data for SFT. Theoretically, we
prove that the global optimum to the training objective function of our method
is achieved only when the LLM policy aligns with the target data distribution.
Empirically, we evaluate our method on several benchmark datasets including the
HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our
results show that SPIN can significantly improve the LLM's performance across a
variety of benchmarks and even outperform models trained through direct
preference optimization (DPO) supplemented with extra GPT-4 preference data.
This sheds light on the promise of self-play, enabling the achievement of
human-level performance in LLMs without the need for expert opponents.
6594ba845070805494045542
Zixiang Chen
admin_assigned
2024-02-14T13:02:50.615Z
64335a225277e3b24ef3de1a
/avatars/535fc39476b46fe6f589dcd4d7ca7f01.svg
Zixiang Chen
user
chenzx19
6594ba845070805494045543
Yihe Deng
claimed_verified
2024-01-03T12:34:15.847Z
642f4c789b2484d7d8551a93
https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
Yihe Deng
user
ydeng9
6594ba845070805494045544
Huizhuo Yuan
claimed_verified
2024-02-19T09:35:20.039Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
6594ba845070805494045545
Kaixuan Ji
admin_assigned
2024-02-14T13:03:08.926Z
656c1018e8bf55919a8234b0
/avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
Kaixuan Ji
user
KaixuanJi
6594ba845070805494045546
Quanquan Gu
claimed_verified
2024-01-03T12:34:09.757Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models
Harnessing the power of human-annotated data through Supervised Fine-Tuning
(SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we
delve into the prospect of growing a strong LLM out of a weak one without the
need for acquiring additional human-annotated data. We propose a new
fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a
supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,
where the LLM refines its capability by playing against instances of itself.
More specifically, the LLM generates its own training data from its previous
iterations, refining its policy by discerning these self-generated responses
from those obtained from human-annotated data. Our method progressively
elevates the LLM from a nascent model to a formidable one, unlocking the full
potential of human-annotated demonstration data for SFT. Theoretically, we
prove that the global optimum to the training objective function of our method
is achieved only when the LLM policy aligns with the target data distribution.
Empirically, we evaluate our method on several benchmark datasets including the
HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our
results show that SPIN can significantly improve the LLM's performance across a
variety of benchmarks and even outperform models trained through direct
preference optimization (DPO) supplemented with extra GPT-4 preference data.
This sheds light on the promise of self-play, enabling the achievement of
human-level performance in LLMs without the need for expert opponents.
A novel self-play fine-tuning method (SPIN) enhances Large Language Models by generating its own training data and refining its policy, achieving performance on par with models trained using direct preference optimization and additional human data.
Large Language Models Supervised Fine-Tuning Self-Play fIne-tuNing SPIN self-play mechanism policy refinement HuggingFace Open LLM Leaderboard MT-Bench Big-Bench direct preference optimization