CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web
  Navigation
While much work on web agents emphasizes the promise of autonomously
performing tasks on behalf of users, in reality, agents often fall short on
complex tasks in real-world contexts and modeling user preference. This
presents an opportunity for humans to collaborate with the agent and leverage
the agent's capabilities effectively. We propose CowPilot, a framework
supporting autonomous as well as human-agent collaborative web navigation, and
evaluation across task success and task efficiency. CowPilot reduces the number
of steps humans need to perform by allowing agents to propose next steps, while
users are able to pause, reject, or take alternative actions. During execution,
users can interleave their actions with the agent by overriding suggestions or
resuming agent control when needed. We conducted case studies on five common
websites and found that the human-agent collaborative mode achieves the highest
success rate of 95% while requiring humans to perform only 15.2% of the total
steps. Even with human interventions during task execution, the agent
successfully drives up to half of task success on its own. CowPilot can serve
as a useful tool for data collection and agent evaluation across websites,
which we believe will enable research in how users and agents can work
together. Video demonstrations are available at
https://oaishi.github.io/cowpilot.html
679d17693f3f5f82f3541388
Faria Huq
679d17693f3f5f82f3541389
Zora Zhiruo Wang
679d17693f3f5f82f354138a
Frank F. Xu
679d17693f3f5f82f354138b
Tianyue Ou
679d17693f3f5f82f354138c
Shuyan Zhou
679d17693f3f5f82f354138d
Jeffrey P. Bigham
679d17693f3f5f82f354138e
Graham Neubig
CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web
  Navigation
While much work on web agents emphasizes the promise of autonomously
performing tasks on behalf of users, in reality, agents often fall short on
complex tasks in real-world contexts and modeling user preference. This
presents an opportunity for humans to collaborate with the agent and leverage
the agent's capabilities effectively. We propose CowPilot, a framework
supporting autonomous as well as human-agent collaborative web navigation, and
evaluation across task success and task efficiency. CowPilot reduces the number
of steps humans need to perform by allowing agents to propose next steps, while
users are able to pause, reject, or take alternative actions. During execution,
users can interleave their actions with the agent by overriding suggestions or
resuming agent control when needed. We conducted case studies on five common
websites and found that the human-agent collaborative mode achieves the highest
success rate of 95% while requiring humans to perform only 15.2% of the total
steps. Even with human interventions during task execution, the agent
successfully drives up to half of task success on its own. CowPilot can serve
as a useful tool for data collection and agent evaluation across websites,
which we believe will enable research in how users and agents can work
together. Video demonstrations are available at
https://oaishi.github.io/cowpilot.html
CowPilot is a framework that combines human-agent collaboration in web navigation, demonstrating high task success and efficiency by reducing user steps.
web agents autonomous navigation human-agent collaboration task success task efficiency web navigation agent suggestions user interventions