title: Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge
    with Curriculum Preference Learning
summary: Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique
  for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO
  have enabled LLMs to distill high-quality behaviors from MCTS, improving their
  reasoning performance. However, existing distillation methods underutilize the
  rich trajectory information generated by MCTS, limiting the potential for
  improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel
  pairwise training framework that enables LLMs to self-improve through MCTS
  behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via
  two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from
  child nodes sharing the same parent in the search tree, providing step-level
  information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL
  introduces curriculum preference learning, dynamically adjusting the training
  sequence of trajectory pairs in each offline training epoch to prioritize
  critical learning steps and mitigate overfitting. Experimental results on
  mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly
  outperforms previous MCTS behavior distillation methods, substantially boosting
  the reasoning capabilities of LLMs.
paper_authors._id: 67089cf0b51108162c256e77
paper_authors.name: Xiyao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:47:38.122Z
paper_authors._id: 655fed9fdef5905d38b84af3
paper_authors.avatarUrl: /avatars/2cda4182dfd11a1e94743639e62328ea.svg
paper_authors.fullname: Xiyao Wang
paper_authors.type: user
paper_authors.user: russwang
paper_authors._id: 67089cf0b51108162c256e78
paper_authors.name: Linfeng Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:47:15.198Z
paper_authors._id: 64c94eddcb2f1bf0e7db5a4d
paper_authors.avatarUrl: /avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
paper_authors.fullname: Linfeng Song
paper_authors.type: user
paper_authors.user: freesunshine0316
paper_authors._id: 67089cf0b51108162c256e79
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:27:57.244Z
paper_authors._id: 6414061f996b2e426f23df48
paper_authors.avatarUrl: /avatars/db0af49c5f017f7c455a076b8b2212b6.svg
paper_authors.fullname: Ye Tian
paper_authors.type: user
paper_authors.user: yetian
paper_authors._id: 67089cf0b51108162c256e7a
paper_authors.name: Dian Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:51:40.388Z
paper_authors._id: 63635bcb242f3b3a859a71f0
paper_authors.avatarUrl: /avatars/7fd623003a18e9252fabeca684b69afc.svg
paper_authors.fullname: Dian Yu
paper_authors.type: user
paper_authors.user: Pluie1503
paper_authors._id: 67089cf0b51108162c256e7b
paper_authors.name: Baolin Peng
paper_authors._id: 67089cf0b51108162c256e7c
paper_authors.name: Haitao Mi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:52:43.237Z
paper_authors._id: 65147a1426fbd558dbd08f1b
paper_authors.avatarUrl: /avatars/86574ee2d5c22e940be1c4e50be88675.svg
paper_authors.fullname: Haitao Mi
paper_authors.type: user
paper_authors.user: haitaominlp
paper_authors._id: 67089cf0b51108162c256e7d
paper_authors.name: Furong Huang
paper_authors._id: 67089cf0b51108162c256e7e
paper_authors.name: Dong Yu
paper_title: Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge
    with Curriculum Preference Learning
paper_summary: Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique
  for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO
  have enabled LLMs to distill high-quality behaviors from MCTS, improving their
  reasoning performance. However, existing distillation methods underutilize the
  rich trajectory information generated by MCTS, limiting the potential for
  improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel
  pairwise training framework that enables LLMs to self-improve through MCTS
  behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via
  two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from
  child nodes sharing the same parent in the search tree, providing step-level
  information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL
  introduces curriculum preference learning, dynamically adjusting the training
  sequence of trajectory pairs in each offline training epoch to prioritize
  critical learning steps and mitigate overfitting. Experimental results on
  mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly
  outperforms previous MCTS behavior distillation methods, substantially boosting
  the reasoning capabilities of LLMs.
paper_ai_keywords: Monte Carlo Tree Search (MCTS) AlphaLLM-CPL pairwise training framework trajectory pairs curriculum preference learning overfitting mathematical reasoning tasks reasoning capabilities