title: SPaR: Self-Play with Tree-Search Refinement to Improve
    Instruction-Following in Large Language Models
summary: Instruction-following is a fundamental capability of language models,
  requiring the model to recognize even the most subtle requirements in the
  instructions and accurately reflect them in its output. Such an ability is
  well-suited for and often optimized by preference learning. However, existing
  methods often directly sample multiple independent responses from the model
  when creating preference pairs. Such practice can introduce content variations
  irrelevant to whether the instruction is precisely followed (e.g., different
  expressions about the same semantic), interfering with the goal of teaching
  models to recognize the key differences that lead to improved instruction
  following. In light of this, we introduce SPaR, a self-play framework
  integrating tree-search self-refinement to yield valid and comparable
  preference pairs free from distractions. By playing against itself, an LLM
  employs a tree-search strategy to refine its previous responses with respect to
  the instruction while minimizing unnecessary variations. Our experiments show
  that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
  GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
  Furthermore, SPaR demonstrates promising scalability and transferability,
  greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
  inference scaling in tree search would impact model performance. Our code and
  data are publicly available at https://github.com/thu-coai/SPaR.
paper_authors._id: 6760f48fa38bf4e171b85bb8
paper_authors.name: Jiale Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:03:44.419Z
paper_authors._id: 627626d42d26ac639e56f565
paper_authors.avatarUrl: /avatars/805c5f909f52656345b8bde486c9fa8f.svg
paper_authors.fullname: Jiale Cheng
paper_authors.type: user
paper_authors.user: CCCCCC
paper_authors._id: 6760f48fa38bf4e171b85bb9
paper_authors.name: Xiao Liu
paper_authors._id: 6760f48fa38bf4e171b85bba
paper_authors.name: Cunxiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:21:56.020Z
paper_authors._id: 65eaf755ab0a6a90da55ab58
paper_authors.avatarUrl: /avatars/a46890a9d067a913513edf3759f12c85.svg
paper_authors.fullname: Cunxiang Wang
paper_authors.type: user
paper_authors.user: wangcunxiang
paper_authors._id: 6760f48fa38bf4e171b85bbb
paper_authors.name: Xiaotao Gu
paper_authors._id: 6760f48fa38bf4e171b85bbc
paper_authors.name: Yida Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:22:10.606Z
paper_authors._id: 6445e2e83a0fa0e98cd0c995
paper_authors.avatarUrl: /avatars/5ab8e5fbfaa9e5b456fb94eaa7e62832.svg
paper_authors.fullname: Yida Lu
paper_authors.type: user
paper_authors.user: yida-lu
paper_authors._id: 6760f48fa38bf4e171b85bbd
paper_authors.name: Dan Zhang
paper_authors._id: 6760f48fa38bf4e171b85bbe
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:22:40.968Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 6760f48fa38bf4e171b85bbf
paper_authors.name: Jie Tang
paper_authors._id: 6760f48fa38bf4e171b85bc0
paper_authors.name: Hongning Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:22:48.091Z
paper_authors._id: 62ec23fcbd19e355478fc584
paper_authors.avatarUrl: /avatars/d7c567ef5f20bb3b9905cb5015d11e12.svg
paper_authors.fullname: Hongning Wang
paper_authors.type: user
paper_authors.user: howang
paper_authors._id: 6760f48fa38bf4e171b85bc1
paper_authors.name: Minlie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:23:16.966Z
paper_authors._id: 650d3b9ee65b12d3b97ccfbd
paper_authors.avatarUrl: /avatars/1e0ecb1c4175fe1240b0eaad3fcc61a6.svg
paper_authors.fullname: huangminlie
paper_authors.type: user
paper_authors.user: huangminlie
paper_title: SPaR: Self-Play with Tree-Search Refinement to Improve
    Instruction-Following in Large Language Models
paper_summary: Instruction-following is a fundamental capability of language models,
  requiring the model to recognize even the most subtle requirements in the
  instructions and accurately reflect them in its output. Such an ability is
  well-suited for and often optimized by preference learning. However, existing
  methods often directly sample multiple independent responses from the model
  when creating preference pairs. Such practice can introduce content variations
  irrelevant to whether the instruction is precisely followed (e.g., different
  expressions about the same semantic), interfering with the goal of teaching
  models to recognize the key differences that lead to improved instruction
  following. In light of this, we introduce SPaR, a self-play framework
  integrating tree-search self-refinement to yield valid and comparable
  preference pairs free from distractions. By playing against itself, an LLM
  employs a tree-search strategy to refine its previous responses with respect to
  the instruction while minimizing unnecessary variations. Our experiments show
  that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
  GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
  Furthermore, SPaR demonstrates promising scalability and transferability,
  greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
  inference scaling in tree search would impact model performance. Our code and
  data are publicly available at https://github.com/thu-coai/SPaR.
paper_ai_keywords: SPaR self-play framework tree-search self-refinement preference pairs LLM tree-search strategy IFEval benchmark GLM-4-9B LLaMA3-70B inference scaling