SPaR: Self-Play with Tree-Search Refinement to Improve
  Instruction-Following in Large Language Models
Instruction-following is a fundamental capability of language models,
requiring the model to recognize even the most subtle requirements in the
instructions and accurately reflect them in its output. Such an ability is
well-suited for and often optimized by preference learning. However, existing
methods often directly sample multiple independent responses from the model
when creating preference pairs. Such practice can introduce content variations
irrelevant to whether the instruction is precisely followed (e.g., different
expressions about the same semantic), interfering with the goal of teaching
models to recognize the key differences that lead to improved instruction
following. In light of this, we introduce SPaR, a self-play framework
integrating tree-search self-refinement to yield valid and comparable
preference pairs free from distractions. By playing against itself, an LLM
employs a tree-search strategy to refine its previous responses with respect to
the instruction while minimizing unnecessary variations. Our experiments show
that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
Furthermore, SPaR demonstrates promising scalability and transferability,
greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
inference scaling in tree search would impact model performance. Our code and
data are publicly available at https://github.com/thu-coai/SPaR.
6760f48fa38bf4e171b85bb8
Jiale Cheng
claimed_verified
2024-12-17T08:03:44.419Z
627626d42d26ac639e56f565
/avatars/805c5f909f52656345b8bde486c9fa8f.svg
Jiale Cheng
user
CCCCCC
6760f48fa38bf4e171b85bb9
Xiao Liu
6760f48fa38bf4e171b85bba
Cunxiang Wang
admin_assigned
2024-12-17T11:21:56.020Z
65eaf755ab0a6a90da55ab58
/avatars/a46890a9d067a913513edf3759f12c85.svg
Cunxiang Wang
user
wangcunxiang
6760f48fa38bf4e171b85bbb
Xiaotao Gu
claimed_verified
2025-07-02T09:25:02.732Z
6438bca4a5e10f6d58694b47
/avatars/3aeb25fbc73c5cab1265e13d11adfb76.svg
XG
user
xgeric
6760f48fa38bf4e171b85bbc
Yida Lu
admin_assigned
2024-12-17T11:22:10.606Z
6445e2e83a0fa0e98cd0c995
/avatars/5ab8e5fbfaa9e5b456fb94eaa7e62832.svg
Yida Lu
user
yida-lu
6760f48fa38bf4e171b85bbd
Dan Zhang
6760f48fa38bf4e171b85bbe
Yuxiao Dong
admin_assigned
2024-12-17T11:22:40.968Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
6760f48fa38bf4e171b85bbf
Jie Tang
6760f48fa38bf4e171b85bc0
Hongning Wang
admin_assigned
2024-12-17T11:22:48.091Z
62ec23fcbd19e355478fc584
/avatars/d7c567ef5f20bb3b9905cb5015d11e12.svg
Hongning Wang
user
howang
6760f48fa38bf4e171b85bc1
Minlie Huang
admin_assigned
2024-12-17T11:23:16.966Z
650d3b9ee65b12d3b97ccfbd
/avatars/1e0ecb1c4175fe1240b0eaad3fcc61a6.svg
huangminlie
user
huangminlie
SPaR: Self-Play with Tree-Search Refinement to Improve
  Instruction-Following in Large Language Models
Instruction-following is a fundamental capability of language models,
requiring the model to recognize even the most subtle requirements in the
instructions and accurately reflect them in its output. Such an ability is
well-suited for and often optimized by preference learning. However, existing
methods often directly sample multiple independent responses from the model
when creating preference pairs. Such practice can introduce content variations
irrelevant to whether the instruction is precisely followed (e.g., different
expressions about the same semantic), interfering with the goal of teaching
models to recognize the key differences that lead to improved instruction
following. In light of this, we introduce SPaR, a self-play framework
integrating tree-search self-refinement to yield valid and comparable
preference pairs free from distractions. By playing against itself, an LLM
employs a tree-search strategy to refine its previous responses with respect to
the instruction while minimizing unnecessary variations. Our experiments show
that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
Furthermore, SPaR demonstrates promising scalability and transferability,
greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
inference scaling in tree search would impact model performance. Our code and
data are publicly available at https://github.com/thu-coai/SPaR.
SPaR, a self-play framework using tree-search refinement, enhances instruction-following capabilities of LLMs by generating relevant preference pairs, leading to improved performance on benchmarks without sacrificing generalization.
self-play framework tree-search self-refinement instruction-following preference pairs IFEval benchmark general capabilities inference scaling