Scaling Computer-Use Grounding via User Interface Decomposition and
  Synthesis
Graphical user interface (GUI) grounding, the ability to map natural language
instructions to specific actions on graphical user interfaces, remains a
critical bottleneck in computer use agent development. Current benchmarks
oversimplify grounding tasks as short referring expressions, failing to capture
the complexity of real-world interactions that require software commonsense,
layout understanding, and fine-grained manipulation capabilities. To address
these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising
564 finely annotated samples across diverse task types including text matching,
element recognition, layout understanding, and precise manipulation.
Additionally, we synthesize and release the largest computer use grounding
dataset Jedi, which contains 4 million examples through multi-perspective
decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its
effectiveness by outperforming existing approaches on ScreenSpot-v2,
ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved
grounding with Jedi directly enhances agentic capabilities of general
foundation models on complex computer tasks, improving from 5% to 27% on
OSWorld. Through detailed ablation studies, we identify key factors
contributing to grounding performance and verify that combining specialized
data for different interface elements enables compositional generalization to
novel interfaces. All benchmark, data, checkpoints, and code are open-sourced
and available at https://osworld-grounding.github.io.
682c12b44040343163ca7e2a
Tianbao Xie
claimed_verified
2025-05-20T07:20:09.634Z
618767e4238063b4615d042b
https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
Tianbao Xie
user
tianbaoxiexxx
682c12b44040343163ca7e2b
Jiaqi Deng
claimed_verified
2025-05-20T07:20:06.695Z
66eeeb2ae65d94c88e9af620
/avatars/a25657d634878e9d53ada19feb38149a.svg
Jiaqi Deng
user
MillanK
682c12b44040343163ca7e2c
Xiaochuan Li
claimed_verified
2025-05-20T07:20:04.603Z
64b103cf372d434077206750
/avatars/ba0eb4fc712a8b9b93ceb30d11859ec2.svg
Xiaochuan Li
user
lixiaochuan2020
682c12b44040343163ca7e2d
Junlin Yang
claimed_verified
2025-05-20T07:20:02.029Z
66ed083acaf696884760729a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RgPe99BqsJsHUWoXO1qtS.jpeg
Nick Yang
user
RadioBlue
682c12b44040343163ca7e2e
Haoyuan Wu
claimed_verified
2025-05-20T19:29:32.375Z
653927c90dfecb4b26bc5885
https://cdn-avatars.huggingface.co/v1/production/uploads/653927c90dfecb4b26bc5885/lM7d75M0CzVrb6HVz1dDD.png
Fred Wu
user
HappySixHF
682c12b44040343163ca7e2f
Jixuan Chen
admin_assigned
2025-05-20T08:24:36.390Z
6465941d0e6c7618f615675b
https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
Jixuan Chen
user
Mayome
682c12b44040343163ca7e30
Wenjing Hu
682c12b44040343163ca7e31
Xinyuan Wang
claimed_verified
2025-05-26T14:08:42.493Z
67b327cdd4665a0448eef7d5
https://cdn-avatars.huggingface.co/v1/production/uploads/67b327cdd4665a0448eef7d5/_B5Z9MCa_qiFrDj1axKlz.png
Xinyuan Wang
user
xywang626
682c12b44040343163ca7e32
Yuhui Xu
admin_assigned
2025-05-20T08:25:12.279Z
6602869253a0518b2a98cafd
/avatars/c14b5953a716f42c83ad28147f8308ae.svg
Yuhui Xu
user
yuhuixu
682c12b44040343163ca7e33
Zekun Wang
admin_assigned
2025-05-20T08:25:30.953Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
682c12b44040343163ca7e34
Yiheng Xu
admin_assigned
2025-05-20T08:25:52.551Z
601d29ab913ad3afd7b7ddb8
https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
Yiheng Xu
user
ranpox
682c12b44040343163ca7e35
Junli Wang
682c12b44040343163ca7e36
Doyen Sahoo
admin_assigned
2025-05-20T08:26:21.701Z
65f84fd980481173afd91233
/avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
Doyen
user
doyensahoo
682c12b44040343163ca7e37
Tao Yu
682c12b44040343163ca7e38
Caiming Xiong
admin_assigned
2025-05-20T08:26:15.939Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
Scaling Computer-Use Grounding via User Interface Decomposition and
  Synthesis
Graphical user interface (GUI) grounding, the ability to map natural language
instructions to specific actions on graphical user interfaces, remains a
critical bottleneck in computer use agent development. Current benchmarks
oversimplify grounding tasks as short referring expressions, failing to capture
the complexity of real-world interactions that require software commonsense,
layout understanding, and fine-grained manipulation capabilities. To address
these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising
564 finely annotated samples across diverse task types including text matching,
element recognition, layout understanding, and precise manipulation.
Additionally, we synthesize and release the largest computer use grounding
dataset Jedi, which contains 4 million examples through multi-perspective
decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its
effectiveness by outperforming existing approaches on ScreenSpot-v2,
ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved
grounding with Jedi directly enhances agentic capabilities of general
foundation models on complex computer tasks, improving from 5% to 27% on
OSWorld. Through detailed ablation studies, we identify key factors
contributing to grounding performance and verify that combining specialized
data for different interface elements enables compositional generalization to
novel interfaces. All benchmark, data, checkpoints, and code are open-sourced
and available at https://osworld-grounding.github.io.
A new benchmark OSWorld-G and dataset Jedi enhance GUI grounding by capturing complex interactions, leading to improved performance in computer use agents.
GUI grounding referring expressions software commonsense layout understanding fine-grained manipulation OSWorld-G Jedi multi-scale models ScreenSpot-v2 ScreenSpot-Pro compositional generalization