title: Self-play with Execution Feedback: Improving Instruction-following
    Capabilities of Large Language Models
summary: One core capability of large language models (LLMs) is to follow natural
  language instructions. However, the issue of automatically constructing
  high-quality training data to enhance the complex instruction-following
  abilities of LLMs without manual annotation remains unresolved. In this paper,
  we introduce AutoIF, the first scalable and reliable method for automatically
  generating instruction-following training data. AutoIF transforms the
  validation of instruction-following data quality into code verification,
  requiring LLMs to generate instructions, the corresponding code to check the
  correctness of the instruction responses, and unit test samples to verify the
  code's correctness. Then, execution feedback-based rejection sampling can
  generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from
  Human Feedback (RLHF) training. AutoIF achieves significant improvements across
  three training algorithms, SFT, Offline DPO, and Online DPO, when applied to
  the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and
  strong-to-weak distillation settings. Our code is publicly available at
  https://github.com/QwenLM/AutoIF.
paper_authors._id: 66755c0fd5ac75b591f746da
paper_authors.name: Guanting Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:22:03.427Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 66755c0fd5ac75b591f746db
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:22:09.957Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 66755c0fd5ac75b591f746dc
paper_authors.name: Chengpeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:22:19.159Z
paper_authors._id: 65294b334d7cf551ac50d6a6
paper_authors.avatarUrl: /avatars/75d21e20b711b871616ef3850bb900b7.svg
paper_authors.fullname: ChengpengLi
paper_authors.type: user
paper_authors.user: ChengpengLi
paper_authors._id: 66755c0fd5ac75b591f746dd
paper_authors.name: Tingyu Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:54.377Z
paper_authors._id: 653b5cb0d994e992e2831539
paper_authors.avatarUrl: /avatars/e6b917cb263f10a6d9c72ec539276428.svg
paper_authors.fullname: xiatingyu
paper_authors.type: user
paper_authors.user: xiatingyu
paper_authors._id: 66755c0fd5ac75b591f746de
paper_authors.name: Bowen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:32:26.389Z
paper_authors._id: 63d9d68c1cae35c27bf7a6a7
paper_authors.avatarUrl: /avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: Tigerph
paper_authors._id: 66755c0fd5ac75b591f746df
paper_authors.name: Chang Zhou
paper_authors._id: 66755c0fd5ac75b591f746e0
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:22:56.911Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: Self-play with Execution Feedback: Improving Instruction-following
    Capabilities of Large Language Models
paper_summary: One core capability of large language models (LLMs) is to follow natural
  language instructions. However, the issue of automatically constructing
  high-quality training data to enhance the complex instruction-following
  abilities of LLMs without manual annotation remains unresolved. In this paper,
  we introduce AutoIF, the first scalable and reliable method for automatically
  generating instruction-following training data. AutoIF transforms the
  validation of instruction-following data quality into code verification,
  requiring LLMs to generate instructions, the corresponding code to check the
  correctness of the instruction responses, and unit test samples to verify the
  code's correctness. Then, execution feedback-based rejection sampling can
  generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from
  Human Feedback (RLHF) training. AutoIF achieves significant improvements across
  three training algorithms, SFT, Offline DPO, and Online DPO, when applied to
  the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and
  strong-to-weak distillation settings. Our code is publicly available at
  https://github.com/QwenLM/AutoIF.
paper_ai_keywords: instruction-following AutoIF code verification Supervised Fine-Tuning (SFT) Reinforcement Learning from Human Feedback (RLHF) Offline DPO Online DPO self-alignment strong-to-weak distillation