title: OmniParser for Pure Vision Based GUI Agent
summary: The recent success of large vision language models shows great potential in
  driving the agent system operating on user interfaces. However, we argue that
  the power multimodal models like GPT-4V as a general agent on multiple
  operating systems across different applications is largely underestimated due
  to the lack of a robust screen parsing technique capable of: 1) reliably
  identifying interactable icons within the user interface, and 2) understanding
  the semantics of various elements in a screenshot and accurately associate the
  intended action with the corresponding region on the screen. To fill these
  gaps, we introduce OmniParser, a comprehensive method for parsing user
  interface screenshots into structured elements, which significantly enhances
  the ability of GPT-4V to generate actions that can be accurately grounded in
  the corresponding regions of the interface. We first curated an interactable
  icon detection dataset using popular webpages and an icon description dataset.
  These datasets were utilized to fine-tune specialized models: a detection model
  to parse interactable regions on the screen and a caption model to extract the
  functional semantics of the detected elements. OmniParser
  significantly improves GPT-4V's performance on ScreenSpot benchmark. And on
  Mind2Web and AITW benchmark, OmniParser with screenshot only input
  outperforms the GPT-4V baselines requiring additional information outside of
  screenshot.
paper_authors._id: 66ac4558169efd8637e3e4a7
paper_authors.name: Yadong Lu
paper_authors._id: 66ac4558169efd8637e3e4a8
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:22:30.221Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 66ac4558169efd8637e3e4a9
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:22:37.359Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 66ac4558169efd8637e3e4aa
paper_authors.name: Ahmed Awadallah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:22:51.189Z
paper_authors._id: 6557a41ee74d1065f7d05c51
paper_authors.avatarUrl: /avatars/b29e57fc61d1d3012637f84b82c14818.svg
paper_authors.fullname: Ahmed Awadallah
paper_authors.type: user
paper_authors.user: AhmedAwadallah
paper_title: OmniParser for Pure Vision Based GUI Agent
paper_summary: The recent success of large vision language models shows great potential in
  driving the agent system operating on user interfaces. However, we argue that
  the power multimodal models like GPT-4V as a general agent on multiple
  operating systems across different applications is largely underestimated due
  to the lack of a robust screen parsing technique capable of: 1) reliably
  identifying interactable icons within the user interface, and 2) understanding
  the semantics of various elements in a screenshot and accurately associate the
  intended action with the corresponding region on the screen. To fill these
  gaps, we introduce OmniParser, a comprehensive method for parsing user
  interface screenshots into structured elements, which significantly enhances
  the ability of GPT-4V to generate actions that can be accurately grounded in
  the corresponding regions of the interface. We first curated an interactable
  icon detection dataset using popular webpages and an icon description dataset.
  These datasets were utilized to fine-tune specialized models: a detection model
  to parse interactable regions on the screen and a caption model to extract the
  functional semantics of the detected elements. OmniParser
  significantly improves GPT-4V's performance on ScreenSpot benchmark. And on
  Mind2Web and AITW benchmark, OmniParser with screenshot only input
  outperforms the GPT-4V baselines requiring additional information outside of
  screenshot.
paper_ai_summary: OmniParser enhances GPT-4V's interaction with user interfaces by accurately parsing screenshots into actionable elements, significantly improving performance on key benchmarks.
paper_ai_keywords: vision language models multimodal models GPT-4V screen parsing interactable icons semantic elements detection model caption model ScreenSpot benchmark Mind2Web benchmark AITW benchmark