title: Agent S2: A Compositional Generalist-Specialist Framework for Computer
    Use Agents
summary: Computer use agents automate digital tasks by directly interacting with
  graphical user interfaces (GUIs) on computers and mobile devices, offering
  significant potential to enhance human productivity by completing an open-ended
  space of user queries. However, current agents face significant challenges:
  imprecise grounding of GUI elements, difficulties with long-horizon task
  planning, and performance bottlenecks from relying on single generalist models
  for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
  compositional framework that delegates cognitive responsibilities across
  various generalist and specialist models. We propose a novel
  Mixture-of-Grounding technique to achieve precise GUI localization and
  introduce Proactive Hierarchical Planning, dynamically refining action plans at
  multiple temporal scales in response to evolving observations. Evaluations
  demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
  on three prominent computer use benchmarks. Specifically, Agent S2 achieves
  18.9% and 32.7% relative improvements over leading baseline agents such as
  Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
  Moreover, Agent S2 generalizes effectively to other operating systems and
  applications, surpassing previous best methods by 52.8% on WindowsAgentArena
  and by 16.52% on AndroidWorld relatively. Code available at
  https://github.com/simular-ai/Agent-S.
paper_authors._id: 67ec9340913c072638c16bbf
paper_authors.name: Saaket Agashe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:49:55.477Z
paper_authors._id: 627837664f2afdc41bbd622a
paper_authors.avatarUrl: /avatars/b7bfc4fb77830bba71839c04a4aeea64.svg
paper_authors.fullname: Saaket Agashe
paper_authors.type: user
paper_authors.user: saa1605
paper_authors._id: 67ec9340913c072638c16bc0
paper_authors.name: Kyle Wong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T08:01:17.795Z
paper_authors._id: 6526e6a4329ee9e7cb3045e5
paper_authors.avatarUrl: /avatars/ee9c10d6bd6b3fc96313cb538c94f678.svg
paper_authors.fullname: Kyle Wong
paper_authors.type: user
paper_authors.user: ewveggies
paper_authors._id: 67ec9340913c072638c16bc1
paper_authors.name: Vincent Tu
paper_authors._id: 67ec9340913c072638c16bc2
paper_authors.name: Jiachen Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:46:55.173Z
paper_authors._id: 65edff6233c279253952e0bd
paper_authors.avatarUrl: /avatars/3130f6f9873ae0a943631e56f6d8d341.svg
paper_authors.fullname: Jiachen Yang
paper_authors.type: user
paper_authors.user: jc-y42
paper_authors._id: 67ec9340913c072638c16bc3
paper_authors.name: Ang Li
paper_authors._id: 67ec9340913c072638c16bc4
paper_authors.name: Xin Eric Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:46:30.920Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Agent S2: A Compositional Generalist-Specialist Framework for Computer
    Use Agents
paper_summary: Computer use agents automate digital tasks by directly interacting with
  graphical user interfaces (GUIs) on computers and mobile devices, offering
  significant potential to enhance human productivity by completing an open-ended
  space of user queries. However, current agents face significant challenges:
  imprecise grounding of GUI elements, difficulties with long-horizon task
  planning, and performance bottlenecks from relying on single generalist models
  for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
  compositional framework that delegates cognitive responsibilities across
  various generalist and specialist models. We propose a novel
  Mixture-of-Grounding technique to achieve precise GUI localization and
  introduce Proactive Hierarchical Planning, dynamically refining action plans at
  multiple temporal scales in response to evolving observations. Evaluations
  demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
  on three prominent computer use benchmarks. Specifically, Agent S2 achieves
  18.9% and 32.7% relative improvements over leading baseline agents such as
  Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
  Moreover, Agent S2 generalizes effectively to other operating systems and
  applications, surpassing previous best methods by 52.8% on WindowsAgentArena
  and by 16.52% on AndroidWorld relatively. Code available at
  https://github.com/simular-ai/Agent-S.
paper_ai_summary: Agent S2, a compositional framework using Mixture-of-Grounding and Proactive Hierarchical Planning, achieves state-of-the-art performance in computer use automation across various benchmarks and operating systems.
paper_ai_keywords: Mixture-of-Grounding Proactive Hierarchical Planning compositional framework