Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge
  with Curriculum Preference Learning
Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique
for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO
have enabled LLMs to distill high-quality behaviors from MCTS, improving their
reasoning performance. However, existing distillation methods underutilize the
rich trajectory information generated by MCTS, limiting the potential for
improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel
pairwise training framework that enables LLMs to self-improve through MCTS
behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via
two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from
child nodes sharing the same parent in the search tree, providing step-level
information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL
introduces curriculum preference learning, dynamically adjusting the training
sequence of trajectory pairs in each offline training epoch to prioritize
critical learning steps and mitigate overfitting. Experimental results on
mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly
outperforms previous MCTS behavior distillation methods, substantially boosting
the reasoning capabilities of LLMs.
67089cf0b51108162c256e77
Xiyao Wang
admin_assigned
2024-10-11T10:47:38.122Z
655fed9fdef5905d38b84af3
/avatars/2cda4182dfd11a1e94743639e62328ea.svg
Xiyao Wang
user
russwang
67089cf0b51108162c256e78
Linfeng Song
admin_assigned
2024-10-11T10:47:15.198Z
64c94eddcb2f1bf0e7db5a4d
/avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
Linfeng Song
user
freesunshine0316
67089cf0b51108162c256e79
Ye Tian
claimed_verified
2024-10-17T11:27:57.244Z
6414061f996b2e426f23df48
/avatars/db0af49c5f017f7c455a076b8b2212b6.svg
Ye Tian
user
yetian
67089cf0b51108162c256e7a
Dian Yu
admin_assigned
2024-10-11T10:51:40.388Z
63635bcb242f3b3a859a71f0
/avatars/7fd623003a18e9252fabeca684b69afc.svg
Dian Yu
user
Pluie1503
67089cf0b51108162c256e7b
Baolin Peng
67089cf0b51108162c256e7c
Haitao Mi
admin_assigned
2024-10-11T10:52:43.237Z
65147a1426fbd558dbd08f1b
/avatars/86574ee2d5c22e940be1c4e50be88675.svg
Haitao Mi
user
haitaominlp
67089cf0b51108162c256e7d
Furong Huang
67089cf0b51108162c256e7e
Dong Yu
Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge
  with Curriculum Preference Learning
Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique
for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO
have enabled LLMs to distill high-quality behaviors from MCTS, improving their
reasoning performance. However, existing distillation methods underutilize the
rich trajectory information generated by MCTS, limiting the potential for
improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel
pairwise training framework that enables LLMs to self-improve through MCTS
behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via
two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from
child nodes sharing the same parent in the search tree, providing step-level
information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL
introduces curriculum preference learning, dynamically adjusting the training
sequence of trajectory pairs in each offline training epoch to prioritize
critical learning steps and mitigate overfitting. Experimental results on
mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly
outperforms previous MCTS behavior distillation methods, substantially boosting
the reasoning capabilities of LLMs.
AlphaLLM-CPL, a new pairwise training framework, enhances LLM reasoning by distilling MCTS behaviors more effectively through stepwise trajectory pairs and curriculum preference learning.
Monte Carlo Tree Search MCTS LLMs SFT DPO behavior distillation stepwise trajectory pairs curriculum preference learning reasoning capabilities