title: OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
summary: Existing efforts in building GUI agents heavily rely on the availability of
  robust commercial Vision-Language Models (VLMs) such as GPT-4o and
  GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
  to their significant performance lag compared to their closed-source
  counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
  scenarios. To facilitate future research in this area, we developed OS-Atlas -
  a foundational GUI action model that excels at GUI grounding and OOD agentic
  tasks through innovations in both data and modeling. We have invested
  significant engineering effort in developing an open-source toolkit for
  synthesizing GUI grounding data across multiple platforms, including Windows,
  Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
  the largest open-source cross-platform GUI grounding corpus to date, which
  contains over 13 million GUI elements. This dataset, combined with innovations
  in model training, provides a solid foundation for OS-Atlas to understand GUI
  screenshots and generalize to unseen interfaces. Through extensive evaluation
  across six benchmarks spanning three different platforms (mobile, desktop, and
  web), OS-Atlas demonstrates significant performance improvements over previous
  state-of-the-art models. Our evaluation also uncovers valuable insights into
  continuously improving and scaling the agentic capabilities of open-source
  VLMs.
paper_authors._id: 67286aaf6c2c5bca2617d7b4
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T14:18:19.133Z
paper_authors._id: 6280e830e99dccaac4bbfde5
paper_authors.avatarUrl: /avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
paper_authors.fullname: Zhiyong Wu
paper_authors.type: user
paper_authors.user: zy001
paper_authors._id: 67286aaf6c2c5bca2617d7b5
paper_authors.name: Zhenyu Wu
paper_authors._id: 67286aaf6c2c5bca2617d7b6
paper_authors.name: Fangzhi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T14:18:52.631Z
paper_authors._id: 64e6cf78ecce34cb442dc889
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
paper_authors.fullname: Fangzhi Xu
paper_authors.type: user
paper_authors.user: xufangzhi
paper_authors._id: 67286aaf6c2c5bca2617d7b7
paper_authors.name: Yian Wang
paper_authors._id: 67286aaf6c2c5bca2617d7b8
paper_authors.name: Qiushi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T11:37:21.047Z
paper_authors._id: 6064a0eeb1703ddba0d458b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
paper_authors.fullname: Qiushi
paper_authors.type: user
paper_authors.user: QiushiSun
paper_authors._id: 67286aaf6c2c5bca2617d7b9
paper_authors.name: Chengyou Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T14:19:11.993Z
paper_authors._id: 6602548a68d519ed324b47c5
paper_authors.avatarUrl: /avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
paper_authors.fullname: ChengyouJia
paper_authors.type: user
paper_authors.user: ChengyouJia
paper_authors._id: 67286aaf6c2c5bca2617d7ba
paper_authors.name: Kanzhi Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T14:19:17.294Z
paper_authors._id: 63340dbbd92c5842ae71d1e9
paper_authors.avatarUrl: /avatars/3a3182996bd41b526dcbfa8687d91963.svg
paper_authors.fullname: Kanzhi Cheng
paper_authors.type: user
paper_authors.user: cckevinn
paper_authors._id: 67286aaf6c2c5bca2617d7bb
paper_authors.name: Zichen Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:35:44.330Z
paper_authors._id: 642b9861bb77f8456634b048
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
paper_authors.fullname: Zichen Ding
paper_authors.type: user
paper_authors.user: heroding77
paper_authors._id: 67286aaf6c2c5bca2617d7bc
paper_authors.name: Liheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T14:19:29.363Z
paper_authors._id: 6561824484a9fbe322b9abc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6561824484a9fbe322b9abc3/omS3D6PzGBD7kc4S7bOIO.png
paper_authors.fullname: LIHENG CHEN
paper_authors.type: user
paper_authors.user: Lemaqwq
paper_authors._id: 67286aaf6c2c5bca2617d7bd
paper_authors.name: Paul Pu Liang
paper_authors._id: 67286aaf6c2c5bca2617d7be
paper_authors.name: Yu Qiao
paper_title: OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
paper_summary: Existing efforts in building GUI agents heavily rely on the availability of
  robust commercial Vision-Language Models (VLMs) such as GPT-4o and
  GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
  to their significant performance lag compared to their closed-source
  counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
  scenarios. To facilitate future research in this area, we developed OS-Atlas -
  a foundational GUI action model that excels at GUI grounding and OOD agentic
  tasks through innovations in both data and modeling. We have invested
  significant engineering effort in developing an open-source toolkit for
  synthesizing GUI grounding data across multiple platforms, including Windows,
  Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
  the largest open-source cross-platform GUI grounding corpus to date, which
  contains over 13 million GUI elements. This dataset, combined with innovations
  in model training, provides a solid foundation for OS-Atlas to understand GUI
  screenshots and generalize to unseen interfaces. Through extensive evaluation
  across six benchmarks spanning three different platforms (mobile, desktop, and
  web), OS-Atlas demonstrates significant performance improvements over previous
  state-of-the-art models. Our evaluation also uncovers valuable insights into
  continuously improving and scaling the agentic capabilities of open-source
  VLMs.
paper_ai_summary: OS-Atlas improves GUI agent performance through a large open-source GUI grounding dataset and model innovations.
paper_ai_keywords: GUI agents Vision-Language Models (VLMs) GUI grounding Out-Of-Distribution (OOD) open-source toolkit GUI screenshots benchmarks