title: CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web
    Navigation
summary: While much work on web agents emphasizes the promise of autonomously
  performing tasks on behalf of users, in reality, agents often fall short on
  complex tasks in real-world contexts and modeling user preference. This
  presents an opportunity for humans to collaborate with the agent and leverage
  the agent's capabilities effectively. We propose CowPilot, a framework
  supporting autonomous as well as human-agent collaborative web navigation, and
  evaluation across task success and task efficiency. CowPilot reduces the number
  of steps humans need to perform by allowing agents to propose next steps, while
  users are able to pause, reject, or take alternative actions. During execution,
  users can interleave their actions with the agent by overriding suggestions or
  resuming agent control when needed. We conducted case studies on five common
  websites and found that the human-agent collaborative mode achieves the highest
  success rate of 95% while requiring humans to perform only 15.2% of the total
  steps. Even with human interventions during task execution, the agent
  successfully drives up to half of task success on its own. CowPilot can serve
  as a useful tool for data collection and agent evaluation across websites,
  which we believe will enable research in how users and agents can work
  together. Video demonstrations are available at
  https://oaishi.github.io/cowpilot.html
paper_authors._id: 679d17693f3f5f82f3541388
paper_authors.name: Faria Huq
paper_authors._id: 679d17693f3f5f82f3541389
paper_authors.name: Zora Zhiruo Wang
paper_authors._id: 679d17693f3f5f82f354138a
paper_authors.name: Frank F. Xu
paper_authors._id: 679d17693f3f5f82f354138b
paper_authors.name: Tianyue Ou
paper_authors._id: 679d17693f3f5f82f354138c
paper_authors.name: Shuyan Zhou
paper_authors._id: 679d17693f3f5f82f354138d
paper_authors.name: Jeffrey P. Bigham
paper_authors._id: 679d17693f3f5f82f354138e
paper_authors.name: Graham Neubig
paper_title: CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web
    Navigation
paper_summary: While much work on web agents emphasizes the promise of autonomously
  performing tasks on behalf of users, in reality, agents often fall short on
  complex tasks in real-world contexts and modeling user preference. This
  presents an opportunity for humans to collaborate with the agent and leverage
  the agent's capabilities effectively. We propose CowPilot, a framework
  supporting autonomous as well as human-agent collaborative web navigation, and
  evaluation across task success and task efficiency. CowPilot reduces the number
  of steps humans need to perform by allowing agents to propose next steps, while
  users are able to pause, reject, or take alternative actions. During execution,
  users can interleave their actions with the agent by overriding suggestions or
  resuming agent control when needed. We conducted case studies on five common
  websites and found that the human-agent collaborative mode achieves the highest
  success rate of 95% while requiring humans to perform only 15.2% of the total
  steps. Even with human interventions during task execution, the agent
  successfully drives up to half of task success on its own. CowPilot can serve
  as a useful tool for data collection and agent evaluation across websites,
  which we believe will enable research in how users and agents can work
  together. Video demonstrations are available at
  https://oaishi.github.io/cowpilot.html