title: Toward Self-Improvement of LLMs via Imagination, Searching, and
    Criticizing
summary: Despite the impressive capabilities of Large Language Models (LLMs) on
  various tasks, they still struggle with scenarios that involves complex
  reasoning and planning. Recent work proposed advanced prompting techniques and
  the necessity of fine-tuning with high-quality data to augment LLMs' reasoning
  abilities. However, these approaches are inherently constrained by data
  availability and quality. In light of this, self-correction and self-learning
  emerge as viable solutions, employing strategies that allow LLMs to refine
  their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs
  in self-refining its response, particularly in complex reasoning and planning
  task, remains dubious. In this paper, we introduce AlphaLLM for the
  self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with
  LLMs to establish a self-improving loop, thereby enhancing the capabilities of
  LLMs without additional annotations. Drawing inspiration from the success of
  AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM
  for self-improvement, including data scarcity, the vastness search spaces of
  language tasks, and the subjective nature of feedback in language tasks.
  AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach
  tailored for language tasks, and a trio of critic models for precise feedback.
  Our experimental results in mathematical reasoning tasks demonstrate that
  AlphaLLM significantly enhances the performance of LLMs without additional
  annotations, showing the potential for self-improvement in LLMs.
paper_authors._id: 6621d00f15c921cd65f1d070
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:30:15.518Z
paper_authors._id: 6414061f996b2e426f23df48
paper_authors.avatarUrl: /avatars/db0af49c5f017f7c455a076b8b2212b6.svg
paper_authors.fullname: Ye Tian
paper_authors.type: user
paper_authors.user: yetian
paper_authors._id: 6621d00f15c921cd65f1d071
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:19:08.546Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 6621d00f15c921cd65f1d072
paper_authors.name: Linfeng Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:19:30.406Z
paper_authors._id: 64c94eddcb2f1bf0e7db5a4d
paper_authors.avatarUrl: /avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
paper_authors.fullname: Linfeng Song
paper_authors.type: user
paper_authors.user: freesunshine0316
paper_authors._id: 6621d00f15c921cd65f1d073
paper_authors.name: Lifeng Jin
paper_authors._id: 6621d00f15c921cd65f1d074
paper_authors.name: Dian Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:50:20.554Z
paper_authors._id: 62d58fd53bf5e059f7cc3245
paper_authors.avatarUrl: /avatars/7a4f3ee4a37245f67efd26749d66a706.svg
paper_authors.fullname: Dian Yu
paper_authors.type: user
paper_authors.user: yudian
paper_authors._id: 6621d00f15c921cd65f1d075
paper_authors.name: Haitao Mi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:21:14.239Z
paper_authors._id: 65147a1426fbd558dbd08f1b
paper_authors.avatarUrl: /avatars/86574ee2d5c22e940be1c4e50be88675.svg
paper_authors.fullname: Haitao Mi
paper_authors.type: user
paper_authors.user: haitaominlp
paper_authors._id: 6621d00f15c921cd65f1d076
paper_authors.name: Dong Yu
paper_title: Toward Self-Improvement of LLMs via Imagination, Searching, and
    Criticizing
paper_summary: Despite the impressive capabilities of Large Language Models (LLMs) on
  various tasks, they still struggle with scenarios that involves complex
  reasoning and planning. Recent work proposed advanced prompting techniques and
  the necessity of fine-tuning with high-quality data to augment LLMs' reasoning
  abilities. However, these approaches are inherently constrained by data
  availability and quality. In light of this, self-correction and self-learning
  emerge as viable solutions, employing strategies that allow LLMs to refine
  their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs
  in self-refining its response, particularly in complex reasoning and planning
  task, remains dubious. In this paper, we introduce AlphaLLM for the
  self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with
  LLMs to establish a self-improving loop, thereby enhancing the capabilities of
  LLMs without additional annotations. Drawing inspiration from the success of
  AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM
  for self-improvement, including data scarcity, the vastness search spaces of
  language tasks, and the subjective nature of feedback in language tasks.
  AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach
  tailored for language tasks, and a trio of critic models for precise feedback.
  Our experimental results in mathematical reasoning tasks demonstrate that
  AlphaLLM significantly enhances the performance of LLMs without additional
  annotations, showing the potential for self-improvement in LLMs.
paper_ai_summary: AlphaLLM enhances Large Language Models through integration with Monte Carlo Tree Search and critic models, improving performance in complex reasoning tasks without additional annotations.
paper_ai_keywords: Large Language Models Monte Carlo Tree Search self-correction self-learning self-improving loop prompt synthesis critic models mathematical reasoning tasks