title: Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation
    Model Internet Agents
summary: The vision of a broadly capable and goal-directed agent, such as an
  Internet-browsing agent in the digital world and a household humanoid in the
  physical world, has rapidly advanced, thanks to the generalization capability
  of foundation models. Such a generalist agent needs to have a large and diverse
  skill repertoire, such as finding directions between two travel locations and
  buying specific items from the Internet. If each skill needs to be specified
  manually through a fixed set of human-annotated instructions, the agent's skill
  repertoire will necessarily be limited due to the quantity and diversity of
  human-annotated instructions. In this work, we address this challenge by
  proposing Proposer-Agent-Evaluator, an effective learning system that enables
  foundation model agents to autonomously discover and practice skills in the
  wild. At the heart of PAE is a context-aware task proposer that autonomously
  proposes tasks for the agent to practice with context information of the
  environment such as user demos or even just the name of the website itself for
  Internet-browsing agents. Then, the agent policy attempts those tasks with
  thoughts and actual grounded operations in the real world with resulting
  trajectories evaluated by an autonomous VLM-based success evaluator. The
  success evaluation serves as the reward signal for the agent to refine its
  policies through RL. We validate PAE on challenging vision-based web
  navigation, using both real-world and self-hosted websites from WebVoyager and
  WebArena.To the best of our knowledge, this work represents the first effective
  learning system to apply autonomous task proposal with RL for agents that
  generalizes real-world human-annotated benchmarks with SOTA performances. Our
  open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
paper_authors._id: 6762d6291d6d92e318dfb0f4
paper_authors.name: Yifei Zhou
paper_authors._id: 6762d6291d6d92e318dfb0f5
paper_authors.name: Qianlan Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-18T18:06:53.583Z
paper_authors._id: 662c11c4d64b8a3bee132130
paper_authors.avatarUrl: /avatars/6c669c1d175bfc582e93f02c95b18f1e.svg
paper_authors.fullname: Qianlan Yang
paper_authors.type: user
paper_authors.user: yanQval
paper_authors._id: 6762d6291d6d92e318dfb0f6
paper_authors.name: Kaixiang Lin
paper_authors._id: 6762d6291d6d92e318dfb0f7
paper_authors.name: Min Bai
paper_authors._id: 6762d6291d6d92e318dfb0f8
paper_authors.name: Xiong Zhou
paper_authors._id: 6762d6291d6d92e318dfb0f9
paper_authors.name: Yu-Xiong Wang
paper_authors._id: 6762d6291d6d92e318dfb0fa
paper_authors.name: Sergey Levine
paper_authors._id: 6762d6291d6d92e318dfb0fb
paper_authors.name: Erran Li
paper_title: Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation
    Model Internet Agents
paper_summary: The vision of a broadly capable and goal-directed agent, such as an
  Internet-browsing agent in the digital world and a household humanoid in the
  physical world, has rapidly advanced, thanks to the generalization capability
  of foundation models. Such a generalist agent needs to have a large and diverse
  skill repertoire, such as finding directions between two travel locations and
  buying specific items from the Internet. If each skill needs to be specified
  manually through a fixed set of human-annotated instructions, the agent's skill
  repertoire will necessarily be limited due to the quantity and diversity of
  human-annotated instructions. In this work, we address this challenge by
  proposing Proposer-Agent-Evaluator, an effective learning system that enables
  foundation model agents to autonomously discover and practice skills in the
  wild. At the heart of PAE is a context-aware task proposer that autonomously
  proposes tasks for the agent to practice with context information of the
  environment such as user demos or even just the name of the website itself for
  Internet-browsing agents. Then, the agent policy attempts those tasks with
  thoughts and actual grounded operations in the real world with resulting
  trajectories evaluated by an autonomous VLM-based success evaluator. The
  success evaluation serves as the reward signal for the agent to refine its
  policies through RL. We validate PAE on challenging vision-based web
  navigation, using both real-world and self-hosted websites from WebVoyager and
  WebArena.To the best of our knowledge, this work represents the first effective
  learning system to apply autonomous task proposal with RL for agents that
  generalizes real-world human-annotated benchmarks with SOTA performances. Our
  open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
paper_ai_summary: A learning system (Proposer-Agent-Evaluator) enables foundation models to autonomously discover and practice skills by proposing tasks, attempting them, and being evaluated by a success evaluator using reinforcement learning.
paper_ai_keywords: Proposer-Agent-Evaluator context-aware task proposer agent policy VLM-based success evaluator reinforcement learning vision-based web navigation SOTA performances