title: Improving Autonomous AI Agents with Reflective Tree Search and
    Self-Learning
summary: Autonomous agents have demonstrated significant potential in automating
  complex multistep decision-making tasks. However, even state-of-the-art
  vision-language models (VLMs), such as GPT-4o, still fall short of human-level
  performance, particularly in intricate web environments and long-horizon
  planning tasks. To address these limitations, we introduce Reflective Monte
  Carlo Tree Search (R-MCTS), a novel test-time algorithm designed to enhance the
  ability of AI agents, e.g., powered by GPT-4o, to explore decision space on the
  fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive
  reflection, allowing agents to learn from past interactions and dynamically
  improve their search efficiency; and 2) using multi-agent debate to provide
  reliable state evaluation. Moreover, we improve the agent's performance by
  fine-tuning GPT-4o through self-learning, using R-MCTS generated tree
  traversals without any human-provided labels. On the challenging VisualWebArena
  benchmark, our GPT-4o-based R-MCTS agent achieves a 6% to 30% relative
  improvement across various tasks compared to the previous state-of-the-art.
  Additionally, we show that the knowledge gained from test-time search can be
  effectively transferred back to GPT-4o via fine-tuning. The fine-tuned GPT-4o
  matches 97% of R-MCTS's performance while reducing compute usage by a factor of
  four at test time. Furthermore, qualitative results reveal that the fine-tuned
  GPT-4o model demonstrates the ability to explore the environment, evaluate a
  state, and backtrack to viable ones when it detects that the current state
  cannot lead to success. Moreover, our work demonstrates the compute scaling
  properties in both training - data collection with R-MCTS - and testing time.
  These results suggest a promising research direction to enhance VLMs' reasoning
  and planning capabilities for agentic applications via test-time search and
  self-learning.
paper_authors._id: 6700297bf2b11c7dd21a5fea
paper_authors.name: Xiao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:36:25.439Z
paper_authors._id: 6234fd736dcfc5fe9f5b8601
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647639915850-noauth.jpeg
paper_authors.fullname: Xiao Yu
paper_authors.type: user
paper_authors.user: jasonyux
paper_authors._id: 6700297bf2b11c7dd21a5feb
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:01:44.640Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 6700297bf2b11c7dd21a5fec
paper_authors.name: Vineeth Vajipey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:01:12.557Z
paper_authors._id: 65c3df25dfc61faac86d553a
paper_authors.avatarUrl: /avatars/0ce07f4ab9f3f43f4468d6a6fa7dc255.svg
paper_authors.fullname: Vineeth Vajipey
paper_authors.type: user
paper_authors.user: vineethvajipey
paper_authors._id: 6700297bf2b11c7dd21a5fed
paper_authors.name: Hao Cheng
paper_authors._id: 6700297bf2b11c7dd21a5fee
paper_authors.name: Michel Galley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:00:54.189Z
paper_authors._id: 5f6bed855e78cc6b0ed31dee
paper_authors.avatarUrl: /avatars/d67edb079fa3fe7ea6f2081f7a3afe9e.svg
paper_authors.fullname: Michel Galley
paper_authors.type: user
paper_authors.user: mgalley
paper_authors._id: 6700297bf2b11c7dd21a5fef
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:00:47.513Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6700297bf2b11c7dd21a5ff0
paper_authors.name: Zhou Yu
paper_title: Improving Autonomous AI Agents with Reflective Tree Search and
    Self-Learning
paper_summary: Autonomous agents have demonstrated significant potential in automating
  complex multistep decision-making tasks. However, even state-of-the-art
  vision-language models (VLMs), such as GPT-4o, still fall short of human-level
  performance, particularly in intricate web environments and long-horizon
  planning tasks. To address these limitations, we introduce Reflective Monte
  Carlo Tree Search (R-MCTS), a novel test-time algorithm designed to enhance the
  ability of AI agents, e.g., powered by GPT-4o, to explore decision space on the
  fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive
  reflection, allowing agents to learn from past interactions and dynamically
  improve their search efficiency; and 2) using multi-agent debate to provide
  reliable state evaluation. Moreover, we improve the agent's performance by
  fine-tuning GPT-4o through self-learning, using R-MCTS generated tree
  traversals without any human-provided labels. On the challenging VisualWebArena
  benchmark, our GPT-4o-based R-MCTS agent achieves a 6% to 30% relative
  improvement across various tasks compared to the previous state-of-the-art.
  Additionally, we show that the knowledge gained from test-time search can be
  effectively transferred back to GPT-4o via fine-tuning. The fine-tuned GPT-4o
  matches 97% of R-MCTS's performance while reducing compute usage by a factor of
  four at test time. Furthermore, qualitative results reveal that the fine-tuned
  GPT-4o model demonstrates the ability to explore the environment, evaluate a
  state, and backtrack to viable ones when it detects that the current state
  cannot lead to success. Moreover, our work demonstrates the compute scaling
  properties in both training - data collection with R-MCTS - and testing time.
  These results suggest a promising research direction to enhance VLMs' reasoning
  and planning capabilities for agentic applications via test-time search and
  self-learning.
paper_ai_keywords: Reflective Monte Carlo Tree Search (R-MCTS) contrastive reflection multi-agent debate self-learning tree traversals VisualWebArena benchmark compute usage training - data collection testing time compute scaling properties