UI-TARS: Pioneering Automated GUI Interaction with Native Agents
This paper introduces UI-TARS, a native GUI agent model that solely perceives
the screenshots as input and performs human-like interactions (e.g., keyboard
and mouse operations). Unlike prevailing agent frameworks that depend on
heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts
and workflows, UI-TARS is an end-to-end model that outperforms these
sophisticated frameworks. Experiments demonstrate its superior performance:
UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating
perception, grounding, and GUI task execution. Notably, in the OSWorld
benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15
steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld,
UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several
key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of
GUI screenshots for context-aware understanding of UI elements and precise
captioning; (2) Unified Action Modeling, which standardizes actions into a
unified space across platforms and achieves precise grounding and interaction
through large-scale action traces; (3) System-2 Reasoning, which incorporates
deliberate reasoning into multi-step decision making, involving multiple
reasoning patterns such as task decomposition, reflection thinking, milestone
recognition, etc. (4) Iterative Training with Reflective Online Traces, which
addresses the data bottleneck by automatically collecting, filtering, and
reflectively refining new interaction traces on hundreds of virtual machines.
Through iterative training and reflection tuning, UI-TARS continuously learns
from its mistakes and adapts to unforeseen situations with minimal human
intervention. We also analyze the evolution path of GUI agents to guide the
further development of this domain.
679078f902b4d94b0f2347c1
Yujia Qin
admin_assigned
2025-01-22T15:34:29.986Z
643f37cce9d063936912048b
/avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
Yujia Qin
user
YujiaHi
679078f902b4d94b0f2347c2
Yining Ye
admin_assigned
2025-01-22T15:34:38.046Z
636a0803fbc31ee68fed16bd
/avatars/331073d0fee901b2b9d93287fabe64fc.svg
yining ye
user
yeyn2001
679078f902b4d94b0f2347c3
Junjie Fang
679078f902b4d94b0f2347c4
Haoming Wang
claimed_verified
2025-01-23T09:18:11.031Z
678a19ba39c63f336d24cc27
/avatars/5bec449236ac7d4a0936ef0dd4046761.svg
Haoming Wang
user
MingComplex
679078f902b4d94b0f2347c5
Shihao Liang
admin_assigned
2025-01-22T15:35:23.362Z
64181962db24526c7c9b519a
/avatars/168af397b5e26d0be2652da8a46511f0.svg
liang
user
shihaoliang
679078f902b4d94b0f2347c6
Shizuo Tian
admin_assigned
2025-01-22T15:35:32.442Z
63ede3b7518f6e69a3572c54
/avatars/86e3afd3a8344735ef0b186a313bca2c.svg
Shizuo Tian
user
BlitherBoom
679078f902b4d94b0f2347c7
Junda Zhang
admin_assigned
2025-01-22T15:35:41.058Z
678b879c71d0276e3c3de55a
/avatars/3eb78e8f4bd7ea6e566fdb89a0d6ccb5.svg
Junda Zhang
user
junda-zhang
679078f902b4d94b0f2347c8
Jiahao Li
679078f902b4d94b0f2347c9
Yunxin Li
admin_assigned
2025-01-22T15:36:20.394Z
62fdb01bc1588e1d4c6c1a7c
/avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
Yunxin Li
user
YunxinLi
679078f902b4d94b0f2347ca
Shijue Huang
admin_assigned
2025-01-22T15:36:28.456Z
64ce05c631c655ff8a2e183c
/avatars/f2de7f8a1348b05f46946085e3e9718e.svg
Shijue Huang
user
JoeYing
679078f902b4d94b0f2347cb
Wanjun Zhong
admin_assigned
2025-01-22T15:36:36.061Z
643f956635e2b54a42e7feba
/avatars/c6185f81ae8499ae866ad451c1cbf43b.svg
Wanjun Zhong
user
WanjunZhong
679078f902b4d94b0f2347cc
Kuanye Li
679078f902b4d94b0f2347cd
Jiale Yang
679078f902b4d94b0f2347ce
Yu Miao
679078f902b4d94b0f2347cf
Woyu Lin
679078f902b4d94b0f2347d0
Longxiang Liu
679078f902b4d94b0f2347d1
Xu Jiang
679078f902b4d94b0f2347d2
Qianli Ma
679078f902b4d94b0f2347d3
Jingyu Li
admin_assigned
2025-01-22T15:38:27.051Z
64828d5a01de428ee8abe522
/avatars/6e59f550988ff8ef6239cb5d41dc5e71.svg
li
user
jingyuli
679078f902b4d94b0f2347d4
Xiaojun Xiao
admin_assigned
2025-01-22T15:38:35.794Z
64f466ff32285228200da712
/avatars/2f65939da5cc197784303878f826a0f8.svg
Xiaojun Xiao
user
Dante1018
679078f902b4d94b0f2347d5
Kai Cai
679078f902b4d94b0f2347d6
Chuang Li
679078f902b4d94b0f2347d7
Yaowei Zheng
claimed_verified
2025-03-12T08:50:22.586Z
642fef28a043f0ac7defa8a9
https://cdn-avatars.huggingface.co/v1/production/uploads/642fef28a043f0ac7defa8a9/RwOEkuj3fOnOA54tGR7Ea.png
Yaowei Zheng
user
hiyouga
679078f902b4d94b0f2347d8
Chaolin Jin
679078f902b4d94b0f2347d9
Chen Li
claimed_verified
2025-04-20T15:04:35.959Z
62ff2e3ca80e701897a9d9ff
/avatars/07379a617fff31f41f9cf514845287b4.svg
Chen Li
user
helio9cn
679078f902b4d94b0f2347da
Xiao Zhou
679078f902b4d94b0f2347db
Minchao Wang
admin_assigned
2025-01-22T15:44:07.899Z
64611d34846a6c8c8302885e
/avatars/f4bfc0705aead7b6255c53fe1ef569b2.svg
minchao wang
user
MinChaos
679078f902b4d94b0f2347dc
Haoli Chen
679078f902b4d94b0f2347dd
Zhaojian Li
679078f902b4d94b0f2347de
Haihua Yang
679078f902b4d94b0f2347df
Haifeng Liu
679078f902b4d94b0f2347e0
Feng Lin
679078f902b4d94b0f2347e1
Tao Peng
679078f902b4d94b0f2347e2
Xin Liu
679078f902b4d94b0f2347e3
Guang Shi
admin_assigned
2025-01-22T15:38:51.568Z
63561bfdbcf42eac0b8f13cf
/avatars/6c126e2681b930e0bad3255358fc6e48.svg
Guang Shi
user
anyuzx
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
This paper introduces UI-TARS, a native GUI agent model that solely perceives
the screenshots as input and performs human-like interactions (e.g., keyboard
and mouse operations). Unlike prevailing agent frameworks that depend on
heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts
and workflows, UI-TARS is an end-to-end model that outperforms these
sophisticated frameworks. Experiments demonstrate its superior performance:
UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating
perception, grounding, and GUI task execution. Notably, in the OSWorld
benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15
steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld,
UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several
key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of
GUI screenshots for context-aware understanding of UI elements and precise
captioning; (2) Unified Action Modeling, which standardizes actions into a
unified space across platforms and achieves precise grounding and interaction
through large-scale action traces; (3) System-2 Reasoning, which incorporates
deliberate reasoning into multi-step decision making, involving multiple
reasoning patterns such as task decomposition, reflection thinking, milestone
recognition, etc. (4) Iterative Training with Reflective Online Traces, which
addresses the data bottleneck by automatically collecting, filtering, and
reflectively refining new interaction traces on hundreds of virtual machines.
Through iterative training and reflection tuning, UI-TARS continuously learns
from its mistakes and adapts to unforeseen situations with minimal human
intervention. We also analyze the evolution path of GUI agents to guide the
further development of this domain.
UI-TARS, a native GUI agent model using screenshots as input, outperforms commercial models in various benchmarks through enhanced perception, unified action modeling, system-2 reasoning, and iterative training with reflective online traces.
native GUI agent model context-aware understanding precise captioning unified action modeling system-2 reasoning task decomposition reflection thinking milestone recognition iterative training reflective online traces