title: Scaling Computer-Use Grounding via User Interface Decomposition and
    Synthesis
summary: Graphical user interface (GUI) grounding, the ability to map natural language
  instructions to specific actions on graphical user interfaces, remains a
  critical bottleneck in computer use agent development. Current benchmarks
  oversimplify grounding tasks as short referring expressions, failing to capture
  the complexity of real-world interactions that require software commonsense,
  layout understanding, and fine-grained manipulation capabilities. To address
  these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising
  564 finely annotated samples across diverse task types including text matching,
  element recognition, layout understanding, and precise manipulation.
  Additionally, we synthesize and release the largest computer use grounding
  dataset Jedi, which contains 4 million examples through multi-perspective
  decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its
  effectiveness by outperforming existing approaches on ScreenSpot-v2,
  ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved
  grounding with Jedi directly enhances agentic capabilities of general
  foundation models on complex computer tasks, improving from 5% to 27% on
  OSWorld. Through detailed ablation studies, we identify key factors
  contributing to grounding performance and verify that combining specialized
  data for different interface elements enables compositional generalization to
  novel interfaces. All benchmark, data, checkpoints, and code are open-sourced
  and available at https://osworld-grounding.github.io.
paper_authors._id: 682c12b44040343163ca7e2a
paper_authors.name: Tianbao Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:09.634Z
paper_authors._id: 618767e4238063b4615d042b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
paper_authors.fullname: Tianbao Xie
paper_authors.type: user
paper_authors.user: tianbaoxiexxx
paper_authors._id: 682c12b44040343163ca7e2b
paper_authors.name: Jiaqi Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:06.695Z
paper_authors._id: 66eeeb2ae65d94c88e9af620
paper_authors.avatarUrl: /avatars/a25657d634878e9d53ada19feb38149a.svg
paper_authors.fullname: Jiaqi Deng
paper_authors.type: user
paper_authors.user: MillanK
paper_authors._id: 682c12b44040343163ca7e2c
paper_authors.name: Xiaochuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:04.603Z
paper_authors._id: 64b103cf372d434077206750
paper_authors.avatarUrl: /avatars/ba0eb4fc712a8b9b93ceb30d11859ec2.svg
paper_authors.fullname: Xiaochuan Li
paper_authors.type: user
paper_authors.user: lixiaochuan2020
paper_authors._id: 682c12b44040343163ca7e2d
paper_authors.name: Junlin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:02.029Z
paper_authors._id: 66ed083acaf696884760729a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RgPe99BqsJsHUWoXO1qtS.jpeg
paper_authors.fullname: Nick Yang
paper_authors.type: user
paper_authors.user: RadioBlue
paper_authors._id: 682c12b44040343163ca7e2e
paper_authors.name: Haoyuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:29:32.375Z
paper_authors._id: 653927c90dfecb4b26bc5885
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653927c90dfecb4b26bc5885/lM7d75M0CzVrb6HVz1dDD.png
paper_authors.fullname: Fred Wu
paper_authors.type: user
paper_authors.user: HappySixHF
paper_authors._id: 682c12b44040343163ca7e2f
paper_authors.name: Jixuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:24:36.390Z
paper_authors._id: 6465941d0e6c7618f615675b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
paper_authors.fullname: Jixuan Chen
paper_authors.type: user
paper_authors.user: Mayome
paper_authors._id: 682c12b44040343163ca7e30
paper_authors.name: Wenjing Hu
paper_authors._id: 682c12b44040343163ca7e31
paper_authors.name: Xinyuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:08:42.493Z
paper_authors._id: 67b327cdd4665a0448eef7d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67b327cdd4665a0448eef7d5/_B5Z9MCa_qiFrDj1axKlz.png
paper_authors.fullname: Xinyuan Wang
paper_authors.type: user
paper_authors.user: xywang626
paper_authors._id: 682c12b44040343163ca7e32
paper_authors.name: Yuhui Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:25:12.279Z
paper_authors._id: 6602869253a0518b2a98cafd
paper_authors.avatarUrl: /avatars/c14b5953a716f42c83ad28147f8308ae.svg
paper_authors.fullname: Yuhui Xu
paper_authors.type: user
paper_authors.user: yuhuixu
paper_authors._id: 682c12b44040343163ca7e33
paper_authors.name: Zekun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:25:30.953Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 682c12b44040343163ca7e34
paper_authors.name: Yiheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:25:52.551Z
paper_authors._id: 601d29ab913ad3afd7b7ddb8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
paper_authors.fullname: Yiheng Xu
paper_authors.type: user
paper_authors.user: ranpox
paper_authors._id: 682c12b44040343163ca7e35
paper_authors.name: Junli Wang
paper_authors._id: 682c12b44040343163ca7e36
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:26:21.701Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_authors._id: 682c12b44040343163ca7e37
paper_authors.name: Tao Yu
paper_authors._id: 682c12b44040343163ca7e38
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:26:15.939Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_title: Scaling Computer-Use Grounding via User Interface Decomposition and
    Synthesis
paper_summary: Graphical user interface (GUI) grounding, the ability to map natural language
  instructions to specific actions on graphical user interfaces, remains a
  critical bottleneck in computer use agent development. Current benchmarks
  oversimplify grounding tasks as short referring expressions, failing to capture
  the complexity of real-world interactions that require software commonsense,
  layout understanding, and fine-grained manipulation capabilities. To address
  these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising
  564 finely annotated samples across diverse task types including text matching,
  element recognition, layout understanding, and precise manipulation.
  Additionally, we synthesize and release the largest computer use grounding
  dataset Jedi, which contains 4 million examples through multi-perspective
  decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its
  effectiveness by outperforming existing approaches on ScreenSpot-v2,
  ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved
  grounding with Jedi directly enhances agentic capabilities of general
  foundation models on complex computer tasks, improving from 5% to 27% on
  OSWorld. Through detailed ablation studies, we identify key factors
  contributing to grounding performance and verify that combining specialized
  data for different interface elements enables compositional generalization to
  novel interfaces. All benchmark, data, checkpoints, and code are open-sourced
  and available at https://osworld-grounding.github.io.
paper_ai_summary: A new benchmark OSWorld-G and dataset Jedi enhance GUI grounding by capturing complex interactions, leading to improved performance in computer use agents.
paper_ai_keywords: GUI grounding referring expressions software commonsense layout understanding fine-grained manipulation OSWorld-G Jedi multi-scale models ScreenSpot-v2 ScreenSpot-Pro compositional generalization