title: UI-TARS: Pioneering Automated GUI Interaction with Native Agents
summary: This paper introduces UI-TARS, a native GUI agent model that solely perceives
  the screenshots as input and performs human-like interactions (e.g., keyboard
  and mouse operations). Unlike prevailing agent frameworks that depend on
  heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts
  and workflows, UI-TARS is an end-to-end model that outperforms these
  sophisticated frameworks. Experiments demonstrate its superior performance:
  UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating
  perception, grounding, and GUI task execution. Notably, in the OSWorld
  benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15
  steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld,
  UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several
  key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of
  GUI screenshots for context-aware understanding of UI elements and precise
  captioning; (2) Unified Action Modeling, which standardizes actions into a
  unified space across platforms and achieves precise grounding and interaction
  through large-scale action traces; (3) System-2 Reasoning, which incorporates
  deliberate reasoning into multi-step decision making, involving multiple
  reasoning patterns such as task decomposition, reflection thinking, milestone
  recognition, etc. (4) Iterative Training with Reflective Online Traces, which
  addresses the data bottleneck by automatically collecting, filtering, and
  reflectively refining new interaction traces on hundreds of virtual machines.
  Through iterative training and reflection tuning, UI-TARS continuously learns
  from its mistakes and adapts to unforeseen situations with minimal human
  intervention. We also analyze the evolution path of GUI agents to guide the
  further development of this domain.
paper_authors._id: 679078f902b4d94b0f2347c1
paper_authors.name: Yujia Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:34:29.986Z
paper_authors._id: 643f37cce9d063936912048b
paper_authors.avatarUrl: /avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
paper_authors.fullname: Yujia Qin
paper_authors.type: user
paper_authors.user: YujiaHi
paper_authors._id: 679078f902b4d94b0f2347c2
paper_authors.name: Yining Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:34:38.046Z
paper_authors._id: 636a0803fbc31ee68fed16bd
paper_authors.avatarUrl: /avatars/331073d0fee901b2b9d93287fabe64fc.svg
paper_authors.fullname: yining ye
paper_authors.type: user
paper_authors.user: yeyn2001
paper_authors._id: 679078f902b4d94b0f2347c3
paper_authors.name: Junjie Fang
paper_authors._id: 679078f902b4d94b0f2347c4
paper_authors.name: Haoming Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:18:11.031Z
paper_authors._id: 678a19ba39c63f336d24cc27
paper_authors.avatarUrl: /avatars/5bec449236ac7d4a0936ef0dd4046761.svg
paper_authors.fullname: Haoming Wang
paper_authors.type: user
paper_authors.user: MingComplex
paper_authors._id: 679078f902b4d94b0f2347c5
paper_authors.name: Shihao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:35:23.362Z
paper_authors._id: 64181962db24526c7c9b519a
paper_authors.avatarUrl: /avatars/168af397b5e26d0be2652da8a46511f0.svg
paper_authors.fullname: liang
paper_authors.type: user
paper_authors.user: shihaoliang
paper_authors._id: 679078f902b4d94b0f2347c6
paper_authors.name: Shizuo Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:35:32.442Z
paper_authors._id: 63ede3b7518f6e69a3572c54
paper_authors.avatarUrl: /avatars/86e3afd3a8344735ef0b186a313bca2c.svg
paper_authors.fullname: Shizuo Tian
paper_authors.type: user
paper_authors.user: BlitherBoom
paper_authors._id: 679078f902b4d94b0f2347c7
paper_authors.name: Junda Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:35:41.058Z
paper_authors._id: 678b879c71d0276e3c3de55a
paper_authors.avatarUrl: /avatars/3eb78e8f4bd7ea6e566fdb89a0d6ccb5.svg
paper_authors.fullname: Junda Zhang
paper_authors.type: user
paper_authors.user: junda-zhang
paper_authors._id: 679078f902b4d94b0f2347c8
paper_authors.name: Jiahao Li
paper_authors._id: 679078f902b4d94b0f2347c9
paper_authors.name: Yunxin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:36:20.394Z
paper_authors._id: 62fdb01bc1588e1d4c6c1a7c
paper_authors.avatarUrl: /avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
paper_authors.fullname: Yunxin Li
paper_authors.type: user
paper_authors.user: YunxinLi
paper_authors._id: 679078f902b4d94b0f2347ca
paper_authors.name: Shijue Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:36:28.456Z
paper_authors._id: 64ce05c631c655ff8a2e183c
paper_authors.avatarUrl: /avatars/f2de7f8a1348b05f46946085e3e9718e.svg
paper_authors.fullname: Shijue Huang
paper_authors.type: user
paper_authors.user: JoeYing
paper_authors._id: 679078f902b4d94b0f2347cb
paper_authors.name: Wanjun Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:36:36.061Z
paper_authors._id: 643f956635e2b54a42e7feba
paper_authors.avatarUrl: /avatars/c6185f81ae8499ae866ad451c1cbf43b.svg
paper_authors.fullname: Wanjun Zhong
paper_authors.type: user
paper_authors.user: WanjunZhong
paper_authors._id: 679078f902b4d94b0f2347cc
paper_authors.name: Kuanye Li
paper_authors._id: 679078f902b4d94b0f2347cd
paper_authors.name: Jiale Yang
paper_authors._id: 679078f902b4d94b0f2347ce
paper_authors.name: Yu Miao
paper_authors._id: 679078f902b4d94b0f2347cf
paper_authors.name: Woyu Lin
paper_authors._id: 679078f902b4d94b0f2347d0
paper_authors.name: Longxiang Liu
paper_authors._id: 679078f902b4d94b0f2347d1
paper_authors.name: Xu Jiang
paper_authors._id: 679078f902b4d94b0f2347d2
paper_authors.name: Qianli Ma
paper_authors._id: 679078f902b4d94b0f2347d3
paper_authors.name: Jingyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:38:27.051Z
paper_authors._id: 64828d5a01de428ee8abe522
paper_authors.avatarUrl: /avatars/6e59f550988ff8ef6239cb5d41dc5e71.svg
paper_authors.fullname: li
paper_authors.type: user
paper_authors.user: jingyuli
paper_authors._id: 679078f902b4d94b0f2347d4
paper_authors.name: Xiaojun Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:38:35.794Z
paper_authors._id: 64f466ff32285228200da712
paper_authors.avatarUrl: /avatars/2f65939da5cc197784303878f826a0f8.svg
paper_authors.fullname: Xiaojun Xiao
paper_authors.type: user
paper_authors.user: Dante1018
paper_authors._id: 679078f902b4d94b0f2347d5
paper_authors.name: Kai Cai
paper_authors._id: 679078f902b4d94b0f2347d6
paper_authors.name: Chuang Li
paper_authors._id: 679078f902b4d94b0f2347d7
paper_authors.name: Yaowei Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:50:22.586Z
paper_authors._id: 642fef28a043f0ac7defa8a9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642fef28a043f0ac7defa8a9/RwOEkuj3fOnOA54tGR7Ea.png
paper_authors.fullname: Yaowei Zheng
paper_authors.type: user
paper_authors.user: hiyouga
paper_authors._id: 679078f902b4d94b0f2347d8
paper_authors.name: Chaolin Jin
paper_authors._id: 679078f902b4d94b0f2347d9
paper_authors.name: Chen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:04:35.959Z
paper_authors._id: 62ff2e3ca80e701897a9d9ff
paper_authors.avatarUrl: /avatars/07379a617fff31f41f9cf514845287b4.svg
paper_authors.fullname: Chen Li
paper_authors.type: user
paper_authors.user: helio9cn
paper_authors._id: 679078f902b4d94b0f2347da
paper_authors.name: Xiao Zhou
paper_authors._id: 679078f902b4d94b0f2347db
paper_authors.name: Minchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:44:07.899Z
paper_authors._id: 64611d34846a6c8c8302885e
paper_authors.avatarUrl: /avatars/f4bfc0705aead7b6255c53fe1ef569b2.svg
paper_authors.fullname: minchao wang
paper_authors.type: user
paper_authors.user: MinChaos
paper_authors._id: 679078f902b4d94b0f2347dc
paper_authors.name: Haoli Chen
paper_authors._id: 679078f902b4d94b0f2347dd
paper_authors.name: Zhaojian Li
paper_authors._id: 679078f902b4d94b0f2347de
paper_authors.name: Haihua Yang
paper_authors._id: 679078f902b4d94b0f2347df
paper_authors.name: Haifeng Liu
paper_authors._id: 679078f902b4d94b0f2347e0
paper_authors.name: Feng Lin
paper_authors._id: 679078f902b4d94b0f2347e1
paper_authors.name: Tao Peng
paper_authors._id: 679078f902b4d94b0f2347e2
paper_authors.name: Xin Liu
paper_authors._id: 679078f902b4d94b0f2347e3
paper_authors.name: Guang Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:38:51.568Z
paper_authors._id: 63561bfdbcf42eac0b8f13cf
paper_authors.avatarUrl: /avatars/6c126e2681b930e0bad3255358fc6e48.svg
paper_authors.fullname: Guang Shi
paper_authors.type: user
paper_authors.user: anyuzx
paper_title: UI-TARS: Pioneering Automated GUI Interaction with Native Agents
paper_summary: This paper introduces UI-TARS, a native GUI agent model that solely perceives
  the screenshots as input and performs human-like interactions (e.g., keyboard
  and mouse operations). Unlike prevailing agent frameworks that depend on
  heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts
  and workflows, UI-TARS is an end-to-end model that outperforms these
  sophisticated frameworks. Experiments demonstrate its superior performance:
  UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating
  perception, grounding, and GUI task execution. Notably, in the OSWorld
  benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15
  steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld,
  UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several
  key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of
  GUI screenshots for context-aware understanding of UI elements and precise
  captioning; (2) Unified Action Modeling, which standardizes actions into a
  unified space across platforms and achieves precise grounding and interaction
  through large-scale action traces; (3) System-2 Reasoning, which incorporates
  deliberate reasoning into multi-step decision making, involving multiple
  reasoning patterns such as task decomposition, reflection thinking, milestone
  recognition, etc. (4) Iterative Training with Reflective Online Traces, which
  addresses the data bottleneck by automatically collecting, filtering, and
  reflectively refining new interaction traces on hundreds of virtual machines.
  Through iterative training and reflection tuning, UI-TARS continuously learns
  from its mistakes and adapts to unforeseen situations with minimal human
  intervention. We also analyze the evolution path of GUI agents to guide the
  further development of this domain.
paper_ai_summary: UI-TARS, a native GUI agent model using screenshots as input, outperforms commercial models in various benchmarks through enhanced perception, unified action modeling, system-2 reasoning, and iterative training with reflective online traces.
paper_ai_keywords: native GUI agent model context-aware understanding precise captioning unified action modeling system-2 reasoning task decomposition reflection thinking milestone recognition iterative training reflective online traces