Improving Autonomous AI Agents with Reflective Tree Search and
  Self-Learning
Autonomous agents have demonstrated significant potential in automating
complex multistep decision-making tasks. However, even state-of-the-art
vision-language models (VLMs), such as GPT-4o, still fall short of human-level
performance, particularly in intricate web environments and long-horizon
planning tasks. To address these limitations, we introduce Reflective Monte
Carlo Tree Search (R-MCTS), a novel test-time algorithm designed to enhance the
ability of AI agents, e.g., powered by GPT-4o, to explore decision space on the
fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive
reflection, allowing agents to learn from past interactions and dynamically
improve their search efficiency; and 2) using multi-agent debate to provide
reliable state evaluation. Moreover, we improve the agent's performance by
fine-tuning GPT-4o through self-learning, using R-MCTS generated tree
traversals without any human-provided labels. On the challenging VisualWebArena
benchmark, our GPT-4o-based R-MCTS agent achieves a 6% to 30% relative
improvement across various tasks compared to the previous state-of-the-art.
Additionally, we show that the knowledge gained from test-time search can be
effectively transferred back to GPT-4o via fine-tuning. The fine-tuned GPT-4o
matches 97% of R-MCTS's performance while reducing compute usage by a factor of
four at test time. Furthermore, qualitative results reveal that the fine-tuned
GPT-4o model demonstrates the ability to explore the environment, evaluate a
state, and backtrack to viable ones when it detects that the current state
cannot lead to success. Moreover, our work demonstrates the compute scaling
properties in both training - data collection with R-MCTS - and testing time.
These results suggest a promising research direction to enhance VLMs' reasoning
and planning capabilities for agentic applications via test-time search and
self-learning.
6700297bf2b11c7dd21a5fea
Xiao Yu
claimed_verified
2024-11-04T09:36:25.439Z
6234fd736dcfc5fe9f5b8601
https://cdn-avatars.huggingface.co/v1/production/uploads/1647639915850-noauth.jpeg
Xiao Yu
user
jasonyux
6700297bf2b11c7dd21a5feb
Baolin Peng
admin_assigned
2024-10-07T18:01:44.640Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
6700297bf2b11c7dd21a5fec
Vineeth Vajipey
admin_assigned
2024-10-07T18:01:12.557Z
65c3df25dfc61faac86d553a
/avatars/0ce07f4ab9f3f43f4468d6a6fa7dc255.svg
Vineeth Vajipey
user
vineethvajipey
6700297bf2b11c7dd21a5fed
Hao Cheng
6700297bf2b11c7dd21a5fee
Michel Galley
admin_assigned
2024-10-07T18:00:54.189Z
5f6bed855e78cc6b0ed31dee
/avatars/d67edb079fa3fe7ea6f2081f7a3afe9e.svg
Michel Galley
user
mgalley
6700297bf2b11c7dd21a5fef
Jianfeng Gao
admin_assigned
2024-10-07T18:00:47.513Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
6700297bf2b11c7dd21a5ff0
Zhou Yu
Improving Autonomous AI Agents with Reflective Tree Search and
  Self-Learning
Autonomous agents have demonstrated significant potential in automating
complex multistep decision-making tasks. However, even state-of-the-art
vision-language models (VLMs), such as GPT-4o, still fall short of human-level
performance, particularly in intricate web environments and long-horizon
planning tasks. To address these limitations, we introduce Reflective Monte
Carlo Tree Search (R-MCTS), a novel test-time algorithm designed to enhance the
ability of AI agents, e.g., powered by GPT-4o, to explore decision space on the
fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive
reflection, allowing agents to learn from past interactions and dynamically
improve their search efficiency; and 2) using multi-agent debate to provide
reliable state evaluation. Moreover, we improve the agent's performance by
fine-tuning GPT-4o through self-learning, using R-MCTS generated tree
traversals without any human-provided labels. On the challenging VisualWebArena
benchmark, our GPT-4o-based R-MCTS agent achieves a 6% to 30% relative
improvement across various tasks compared to the previous state-of-the-art.
Additionally, we show that the knowledge gained from test-time search can be
effectively transferred back to GPT-4o via fine-tuning. The fine-tuned GPT-4o
matches 97% of R-MCTS's performance while reducing compute usage by a factor of
four at test time. Furthermore, qualitative results reveal that the fine-tuned
GPT-4o model demonstrates the ability to explore the environment, evaluate a
state, and backtrack to viable ones when it detects that the current state
cannot lead to success. Moreover, our work demonstrates the compute scaling
properties in both training - data collection with R-MCTS - and testing time.
These results suggest a promising research direction to enhance VLMs' reasoning
and planning capabilities for agentic applications via test-time search and
self-learning.
Reflective Monte Carlo Tree Search (R-MCTS) enhances vision-language models' decision-making abilities in complex web environments through contrastive reflection and multi-agent debate, leading to significant performance improvements and efficient fine-tuning.
Reflective Monte Carlo Tree Search R-MCTS contrastive reflection multi-agent debate Vision-Language Models VLMs GPT-4o VisualWebArena self-learning tree traversals compute scaling