Agent S: An Open Agentic Framework that Uses Computers Like a Human
We present Agent S, an open agentic framework that enables autonomous
interaction with computers through a Graphical User Interface (GUI), aimed at
transforming human-computer interaction by automating complex, multi-step
tasks. Agent S aims to address three key challenges in automating computer
tasks: acquiring domain-specific knowledge, planning over long task horizons,
and handling dynamic, non-uniform interfaces. To this end, Agent S introduces
experience-augmented hierarchical planning, which learns from external
knowledge search and internal experience retrieval at multiple levels,
facilitating efficient task planning and subtask execution. In addition, it
employs an Agent-Computer Interface (ACI) to better elicit the reasoning and
control capabilities of GUI agents based on Multimodal Large Language Models
(MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the
baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves
a new state-of-the-art. Comprehensive analysis highlights the effectiveness of
individual components and provides insights for future improvements.
Furthermore, Agent S demonstrates broad generalizability to different operating
systems on a newly-released WindowsAgentArena benchmark. Code available at
https://github.com/simular-ai/Agent-S.
6708aabfd7c32e3238f26076
Saaket Agashe
6708aabfd7c32e3238f26077
Jiuzhou Han
admin_assigned
2024-10-11T10:46:17.834Z
63b0e5a7f2eb87a4d695398a
/avatars/3a03fbb3edbb4aad848cd63c0bce6853.svg
Jiuzhou Han
user
Jiuzhouh
6708aabfd7c32e3238f26078
Shuyu Gan
6708aabfd7c32e3238f26079
Jiachen Yang
admin_assigned
2024-10-11T10:46:05.846Z
65edff6233c279253952e0bd
/avatars/3130f6f9873ae0a943631e56f6d8d341.svg
Jiachen Yang
user
jc-y42
6708aabfd7c32e3238f2607a
Ang Li
6708aabfd7c32e3238f2607b
Xin Eric Wang
admin_assigned
2024-10-11T10:45:36.565Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Agent S: An Open Agentic Framework that Uses Computers Like a Human
We present Agent S, an open agentic framework that enables autonomous
interaction with computers through a Graphical User Interface (GUI), aimed at
transforming human-computer interaction by automating complex, multi-step
tasks. Agent S aims to address three key challenges in automating computer
tasks: acquiring domain-specific knowledge, planning over long task horizons,
and handling dynamic, non-uniform interfaces. To this end, Agent S introduces
experience-augmented hierarchical planning, which learns from external
knowledge search and internal experience retrieval at multiple levels,
facilitating efficient task planning and subtask execution. In addition, it
employs an Agent-Computer Interface (ACI) to better elicit the reasoning and
control capabilities of GUI agents based on Multimodal Large Language Models
(MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the
baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves
a new state-of-the-art. Comprehensive analysis highlights the effectiveness of
individual components and provides insights for future improvements.
Furthermore, Agent S demonstrates broad generalizability to different operating
systems on a newly-released WindowsAgentArena benchmark. Code available at
https://github.com/simular-ai/Agent-S.
Agent S, a framework for autonomous GUI interactions, enhances task automation with experience-augmented hierarchical planning and Multimodal Large Language Models.
Graphical User Interface GUI hierarchical planning external knowledge search internal experience retrieval Agent-Computer Interface ACI Multimodal Large Language Models MLLMs experience-augmented hierarchical planning