title: Ensembling Large Language Models with Process Reward-Guided Tree Search
    for Better Complex Reasoning
summary: Despite recent advances in large language models, open-source models often
  struggle to consistently perform well on complex reasoning tasks. Existing
  ensemble methods, whether applied at the token or output levels, fail to
  address these challenges. In response, we present Language model Ensemble with
  Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level
  ensembling of language models. LE-MCTS formulates step-by-step reasoning with
  an ensemble of language models as a Markov decision process. In this framework,
  states represent intermediate reasoning paths, while actions consist of
  generating the next reasoning step using one of the language models selected
  from a predefined pool. Guided by a process-based reward model, LE-MCTS
  performs a tree search over the reasoning steps generated by different language
  models, identifying the most accurate reasoning chain. Experimental results on
  five mathematical reasoning benchmarks demonstrate that our approach
  outperforms both single language model decoding algorithms and language model
  ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the
  MATH and MQA datasets, respectively, highlighting its effectiveness in solving
  complex reasoning problems.
paper_authors._id: 676c0c92dd95830fd9d60010
paper_authors.name: Sungjin Park
paper_authors._id: 676c0c92dd95830fd9d60011
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:32:19.827Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 676c0c92dd95830fd9d60012
paper_authors.name: Yeyun Gong
paper_authors._id: 676c0c92dd95830fd9d60013
paper_authors.name: Edward Choi
paper_title: Ensembling Large Language Models with Process Reward-Guided Tree Search
    for Better Complex Reasoning
paper_summary: Despite recent advances in large language models, open-source models often
  struggle to consistently perform well on complex reasoning tasks. Existing
  ensemble methods, whether applied at the token or output levels, fail to
  address these challenges. In response, we present Language model Ensemble with
  Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level
  ensembling of language models. LE-MCTS formulates step-by-step reasoning with
  an ensemble of language models as a Markov decision process. In this framework,
  states represent intermediate reasoning paths, while actions consist of
  generating the next reasoning step using one of the language models selected
  from a predefined pool. Guided by a process-based reward model, LE-MCTS
  performs a tree search over the reasoning steps generated by different language
  models, identifying the most accurate reasoning chain. Experimental results on
  five mathematical reasoning benchmarks demonstrate that our approach
  outperforms both single language model decoding algorithms and language model
  ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the
  MATH and MQA datasets, respectively, highlighting its effectiveness in solving
  complex reasoning problems.
paper_ai_summary: A new ensemble framework combining language models with Monte Carlo Tree Search improves performance on complex mathematical reasoning tasks.
paper_ai_keywords: large language models ensemling token-based ensembling output-level ensembling process-level ensembling Markov decision process intermediate reasoning paths reward model Monte Carlo Tree Search LE-MCTS MATH dataset MQA dataset