Toward Self-Improvement of LLMs via Imagination, Searching, and
  Criticizing
Despite the impressive capabilities of Large Language Models (LLMs) on
various tasks, they still struggle with scenarios that involves complex
reasoning and planning. Recent work proposed advanced prompting techniques and
the necessity of fine-tuning with high-quality data to augment LLMs' reasoning
abilities. However, these approaches are inherently constrained by data
availability and quality. In light of this, self-correction and self-learning
emerge as viable solutions, employing strategies that allow LLMs to refine
their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs
in self-refining its response, particularly in complex reasoning and planning
task, remains dubious. In this paper, we introduce AlphaLLM for the
self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with
LLMs to establish a self-improving loop, thereby enhancing the capabilities of
LLMs without additional annotations. Drawing inspiration from the success of
AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM
for self-improvement, including data scarcity, the vastness search spaces of
language tasks, and the subjective nature of feedback in language tasks.
AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach
tailored for language tasks, and a trio of critic models for precise feedback.
Our experimental results in mathematical reasoning tasks demonstrate that
AlphaLLM significantly enhances the performance of LLMs without additional
annotations, showing the potential for self-improvement in LLMs.
6621d00f15c921cd65f1d070
Ye Tian
claimed_verified
2024-10-17T11:30:15.518Z
6414061f996b2e426f23df48
/avatars/db0af49c5f017f7c455a076b8b2212b6.svg
Ye Tian
user
yetian
6621d00f15c921cd65f1d071
Baolin Peng
admin_assigned
2024-04-19T08:19:08.546Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
6621d00f15c921cd65f1d072
Linfeng Song
admin_assigned
2024-04-19T08:19:30.406Z
64c94eddcb2f1bf0e7db5a4d
/avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
Linfeng Song
user
freesunshine0316
6621d00f15c921cd65f1d073
Lifeng Jin
6621d00f15c921cd65f1d074
Dian Yu
claimed_verified
2024-04-22T06:50:20.554Z
62d58fd53bf5e059f7cc3245
/avatars/7a4f3ee4a37245f67efd26749d66a706.svg
Dian Yu
user
yudian
6621d00f15c921cd65f1d075
Haitao Mi
admin_assigned
2024-04-19T08:21:14.239Z
65147a1426fbd558dbd08f1b
/avatars/86574ee2d5c22e940be1c4e50be88675.svg
Haitao Mi
user
haitaominlp
6621d00f15c921cd65f1d076
Dong Yu
Toward Self-Improvement of LLMs via Imagination, Searching, and
  Criticizing
Despite the impressive capabilities of Large Language Models (LLMs) on
various tasks, they still struggle with scenarios that involves complex
reasoning and planning. Recent work proposed advanced prompting techniques and
the necessity of fine-tuning with high-quality data to augment LLMs' reasoning
abilities. However, these approaches are inherently constrained by data
availability and quality. In light of this, self-correction and self-learning
emerge as viable solutions, employing strategies that allow LLMs to refine
their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs
in self-refining its response, particularly in complex reasoning and planning
task, remains dubious. In this paper, we introduce AlphaLLM for the
self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with
LLMs to establish a self-improving loop, thereby enhancing the capabilities of
LLMs without additional annotations. Drawing inspiration from the success of
AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM
for self-improvement, including data scarcity, the vastness search spaces of
language tasks, and the subjective nature of feedback in language tasks.
AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach
tailored for language tasks, and a trio of critic models for precise feedback.
Our experimental results in mathematical reasoning tasks demonstrate that
AlphaLLM significantly enhances the performance of LLMs without additional
annotations, showing the potential for self-improvement in LLMs.
AlphaLLM enhances Large Language Models through integration with Monte Carlo Tree Search and critic models, improving performance in complex reasoning tasks without additional annotations.
Large Language Models Monte Carlo Tree Search self-correction self-learning self-improving loop prompt synthesis critic models mathematical reasoning tasks