title: VideoGUI: A Benchmark for GUI Automation from Instructional Videos
summary: Graphical User Interface (GUI) automation holds significant promise for
  enhancing human productivity by assisting with computer tasks. Existing task
  formulations primarily focus on simple tasks that can be specified by a single,
  language-only instruction, such as "Insert a new slide." In this work, we
  introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI
  assistants on visual-centric GUI tasks. Sourced from high-quality web
  instructional videos, our benchmark focuses on tasks involving professional and
  novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex
  activities (e.g., video editing). VideoGUI evaluates GUI assistants through a
  hierarchical process, allowing for identification of the specific levels at
  which they may fail: (i) high-level planning: reconstruct procedural subtasks
  from visual conditions without language descriptions; (ii) middle-level
  planning: generate sequences of precise action narrations based on visual state
  (i.e., screenshot) and goals; (iii) atomic action execution: perform specific
  actions such as accurately clicking designated elements. For each level, we
  design evaluation metrics across individual dimensions to provide clear
  signals, such as individual performance in clicking, dragging, typing, and
  scrolling for atomic action execution. Our evaluation on VideoGUI reveals that
  even the SoTA large multimodal model GPT4o performs poorly on visual-centric
  GUI tasks, especially for high-level planning.
paper_authors._id: 666fbd99bc840e6748785265
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T16:15:20.666Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 666fbd99bc840e6748785266
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T16:22:52.728Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 666fbd99bc840e6748785267
paper_authors.name: Difei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T16:22:59.802Z
paper_authors._id: 661abfba70ec567e154fbb00
paper_authors.avatarUrl: /avatars/46c004650ba15525ccf02d40c9314085.svg
paper_authors.fullname: Difei Gao
paper_authors.type: user
paper_authors.user: QuStar
paper_authors._id: 666fbd99bc840e6748785268
paper_authors.name: Qinchen WU
paper_authors._id: 666fbd99bc840e6748785269
paper_authors.name: Mingyi Yan
paper_authors._id: 666fbd99bc840e674878526a
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T16:23:22.034Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 666fbd99bc840e674878526b
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T16:23:28.369Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 666fbd99bc840e674878526c
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T16:23:42.360Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: VideoGUI: A Benchmark for GUI Automation from Instructional Videos
paper_summary: Graphical User Interface (GUI) automation holds significant promise for
  enhancing human productivity by assisting with computer tasks. Existing task
  formulations primarily focus on simple tasks that can be specified by a single,
  language-only instruction, such as "Insert a new slide." In this work, we
  introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI
  assistants on visual-centric GUI tasks. Sourced from high-quality web
  instructional videos, our benchmark focuses on tasks involving professional and
  novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex
  activities (e.g., video editing). VideoGUI evaluates GUI assistants through a
  hierarchical process, allowing for identification of the specific levels at
  which they may fail: (i) high-level planning: reconstruct procedural subtasks
  from visual conditions without language descriptions; (ii) middle-level
  planning: generate sequences of precise action narrations based on visual state
  (i.e., screenshot) and goals; (iii) atomic action execution: perform specific
  actions such as accurately clicking designated elements. For each level, we
  design evaluation metrics across individual dimensions to provide clear
  signals, such as individual performance in clicking, dragging, typing, and
  scrolling for atomic action execution. Our evaluation on VideoGUI reveals that
  even the SoTA large multimodal model GPT4o performs poorly on visual-centric
  GUI tasks, especially for high-level planning.
paper_ai_keywords: VideoGUI multi-modal benchmark GUI assistants visual-centric GUI tasks procedural subtasks visual conditions language descriptions sequences of precise action narrations visual state screenshot goals atomic action execution clicking designated elements evaluation metrics individual performance dragging typing scrolling SoTA large multimodal model GPT4o