title: ShowUI: One Vision-Language-Action Model for GUI Visual Agent
summary: Building Graphical User Interface (GUI) assistants holds significant promise
  for enhancing human workflow productivity. While most agents are
  language-based, relying on closed-source API with text-rich meta-information
  (e.g., HTML or accessibility tree), they show limitations in perceiving UI
  visuals as humans do, highlighting the need for GUI visual agents. In this
  work, we develop a vision-language-action model in digital world, namely
  ShowUI, which features the following innovations: (i) UI-Guided Visual Token
  Selection to reduce computational costs by formulating screenshots as an UI
  connected graph, adaptively identifying their redundant relationship and serve
  as the criteria for token selection during self-attention blocks; (ii)
  Interleaved Vision-Language-Action Streaming that flexibly unifies diverse
  needs within GUI tasks, enabling effective management of visual-action history
  in navigation or pairing multi-turn query-action sequences per screenshot to
  enhance training efficiency; (iii) Small-scale High-quality GUI
  Instruction-following Datasets by careful data curation and employing a
  resampling strategy to address significant data type imbalances. With above
  components, ShowUI, a lightweight 2B model using 256K data, achieves a strong
  75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection
  further reduces 33% of redundant visual tokens during training and speeds up
  the performance by 1.4x. Navigation experiments across web Mind2Web, mobile
  AITW, and online MiniWob environments further underscore the effectiveness and
  potential of our model in advancing GUI visual agents. The models are available
  at https://github.com/showlab/ShowUI.
paper_authors._id: 67468ee8e36c38e099c5c0d1
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:16.879Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 67468ee8e36c38e099c5c0d2
paper_authors.name: Linjie Li
paper_authors._id: 67468ee8e36c38e099c5c0d3
paper_authors.name: Difei Gao
paper_authors._id: 67468ee8e36c38e099c5c0d4
paper_authors.name: Zhengyuan Yang
paper_authors._id: 67468ee8e36c38e099c5c0d5
paper_authors.name: Shiwei Wu
paper_authors._id: 67468ee8e36c38e099c5c0d6
paper_authors.name: Zechen Bai
paper_authors._id: 67468ee8e36c38e099c5c0d7
paper_authors.name: Weixian Lei
paper_authors._id: 67468ee8e36c38e099c5c0d8
paper_authors.name: Lijuan Wang
paper_authors._id: 67468ee8e36c38e099c5c0d9
paper_authors.name: Mike Zheng Shou
paper_title: ShowUI: One Vision-Language-Action Model for GUI Visual Agent
paper_summary: Building Graphical User Interface (GUI) assistants holds significant promise
  for enhancing human workflow productivity. While most agents are
  language-based, relying on closed-source API with text-rich meta-information
  (e.g., HTML or accessibility tree), they show limitations in perceiving UI
  visuals as humans do, highlighting the need for GUI visual agents. In this
  work, we develop a vision-language-action model in digital world, namely
  ShowUI, which features the following innovations: (i) UI-Guided Visual Token
  Selection to reduce computational costs by formulating screenshots as an UI
  connected graph, adaptively identifying their redundant relationship and serve
  as the criteria for token selection during self-attention blocks; (ii)
  Interleaved Vision-Language-Action Streaming that flexibly unifies diverse
  needs within GUI tasks, enabling effective management of visual-action history
  in navigation or pairing multi-turn query-action sequences per screenshot to
  enhance training efficiency; (iii) Small-scale High-quality GUI
  Instruction-following Datasets by careful data curation and employing a
  resampling strategy to address significant data type imbalances. With above
  components, ShowUI, a lightweight 2B model using 256K data, achieves a strong
  75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection
  further reduces 33% of redundant visual tokens during training and speeds up
  the performance by 1.4x. Navigation experiments across web Mind2Web, mobile
  AITW, and online MiniWob environments further underscore the effectiveness and
  potential of our model in advancing GUI visual agents. The models are available
  at https://github.com/showlab/ShowUI.
paper_ai_keywords: UI-Guided Visual Token Selection Self-attention blocks Interleaved Vision-Language-Action Streaming Visual-action history Query-symmetry sequences GUI Instruction-following Datasets Resampling strategy Zero-shot screenshot grounding Redundant visual tokens Navigation experiments Web Mind2Web Mobile AITW Online MiniWob