title: Agent S: An Open Agentic Framework that Uses Computers Like a Human
summary: We present Agent S, an open agentic framework that enables autonomous
  interaction with computers through a Graphical User Interface (GUI), aimed at
  transforming human-computer interaction by automating complex, multi-step
  tasks. Agent S aims to address three key challenges in automating computer
  tasks: acquiring domain-specific knowledge, planning over long task horizons,
  and handling dynamic, non-uniform interfaces. To this end, Agent S introduces
  experience-augmented hierarchical planning, which learns from external
  knowledge search and internal experience retrieval at multiple levels,
  facilitating efficient task planning and subtask execution. In addition, it
  employs an Agent-Computer Interface (ACI) to better elicit the reasoning and
  control capabilities of GUI agents based on Multimodal Large Language Models
  (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the
  baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves
  a new state-of-the-art. Comprehensive analysis highlights the effectiveness of
  individual components and provides insights for future improvements.
  Furthermore, Agent S demonstrates broad generalizability to different operating
  systems on a newly-released WindowsAgentArena benchmark. Code available at
  https://github.com/simular-ai/Agent-S.
paper_authors._id: 6708aabfd7c32e3238f26076
paper_authors.name: Saaket Agashe
paper_authors._id: 6708aabfd7c32e3238f26077
paper_authors.name: Jiuzhou Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:46:17.834Z
paper_authors._id: 63b0e5a7f2eb87a4d695398a
paper_authors.avatarUrl: /avatars/3a03fbb3edbb4aad848cd63c0bce6853.svg
paper_authors.fullname: Jiuzhou Han
paper_authors.type: user
paper_authors.user: Jiuzhouh
paper_authors._id: 6708aabfd7c32e3238f26078
paper_authors.name: Shuyu Gan
paper_authors._id: 6708aabfd7c32e3238f26079
paper_authors.name: Jiachen Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:46:05.846Z
paper_authors._id: 65edff6233c279253952e0bd
paper_authors.avatarUrl: /avatars/3130f6f9873ae0a943631e56f6d8d341.svg
paper_authors.fullname: Jiachen Yang
paper_authors.type: user
paper_authors.user: jc-y42
paper_authors._id: 6708aabfd7c32e3238f2607a
paper_authors.name: Ang Li
paper_authors._id: 6708aabfd7c32e3238f2607b
paper_authors.name: Xin Eric Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:45:36.565Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Agent S: An Open Agentic Framework that Uses Computers Like a Human
paper_summary: We present Agent S, an open agentic framework that enables autonomous
  interaction with computers through a Graphical User Interface (GUI), aimed at
  transforming human-computer interaction by automating complex, multi-step
  tasks. Agent S aims to address three key challenges in automating computer
  tasks: acquiring domain-specific knowledge, planning over long task horizons,
  and handling dynamic, non-uniform interfaces. To this end, Agent S introduces
  experience-augmented hierarchical planning, which learns from external
  knowledge search and internal experience retrieval at multiple levels,
  facilitating efficient task planning and subtask execution. In addition, it
  employs an Agent-Computer Interface (ACI) to better elicit the reasoning and
  control capabilities of GUI agents based on Multimodal Large Language Models
  (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the
  baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves
  a new state-of-the-art. Comprehensive analysis highlights the effectiveness of
  individual components and provides insights for future improvements.
  Furthermore, Agent S demonstrates broad generalizability to different operating
  systems on a newly-released WindowsAgentArena benchmark. Code available at
  https://github.com/simular-ai/Agent-S.
paper_ai_keywords: Graphical User Interface (GUI) experience-augmented hierarchical planning external knowledge internal experience retrieval Agent-Computer Interface (ACI) Multimodal Large Language Models (MLLMs) OSWorld benchmark WindowsAgentArena benchmark