title: Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
    Grounding
summary: Graphical User Interfaces (GUIs) are central to our interaction with digital
  devices. Recently, growing efforts have been made to build models for various
  GUI understanding tasks. However, these efforts largely overlook an important
  GUI-referring task: screen reading based on user-indicated points, which we
  name the Screen Point-and-Read (SPR) task. This task is predominantly handled
  by rigid accessible screen reading tools, in great need of new models driven by
  advancements in Multimodal Large Language Models (MLLMs). In this paper, we
  propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
  to address the SPR task. Based on the input point coordinate and the
  corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
  Tree. Based on the tree, our ToL agent not only comprehends the content of the
  indicated area but also articulates the layout and spatial relationships
  between elements. Such layout information is crucial for accurately
  interpreting information on the screen, distinguishing our ToL agent from other
  screen reading tools. We also thoroughly evaluate the ToL agent against other
  baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
  web, and operating systems. Last but not least, we test the ToL agent on mobile
  GUI navigation tasks, demonstrating its utility in identifying incorrect
  actions along the path of agent execution trajectories. Code and data:
  screen-point-and-read.github.io
paper_authors._id: 667e21ecddca7a5233b55dcf
paper_authors.name: Yue Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:57:37.569Z
paper_authors._id: 656b8a62e8bf55919a6a2d61
paper_authors.avatarUrl: /avatars/26b75683dff73c51f966e9c15f04fd9f.svg
paper_authors.fullname: Yue Fan
paper_authors.type: user
paper_authors.user: yfan1997
paper_authors._id: 667e21ecddca7a5233b55dd0
paper_authors.name: Lei Ding
paper_authors._id: 667e21ecddca7a5233b55dd1
paper_authors.name: Ching-Chen Kuo
paper_authors._id: 667e21ecddca7a5233b55dd2
paper_authors.name: Shan Jiang
paper_authors._id: 667e21ecddca7a5233b55dd3
paper_authors.name: Yang Zhao
paper_authors._id: 667e21ecddca7a5233b55dd4
paper_authors.name: Xinze Guan
paper_authors._id: 667e21ecddca7a5233b55dd5
paper_authors.name: Jie Yang
paper_authors._id: 667e21ecddca7a5233b55dd6
paper_authors.name: Yi Zhang
paper_authors._id: 667e21ecddca7a5233b55dd7
paper_authors.name: Xin Eric Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:02:53.091Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
    Grounding
paper_summary: Graphical User Interfaces (GUIs) are central to our interaction with digital
  devices. Recently, growing efforts have been made to build models for various
  GUI understanding tasks. However, these efforts largely overlook an important
  GUI-referring task: screen reading based on user-indicated points, which we
  name the Screen Point-and-Read (SPR) task. This task is predominantly handled
  by rigid accessible screen reading tools, in great need of new models driven by
  advancements in Multimodal Large Language Models (MLLMs). In this paper, we
  propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
  to address the SPR task. Based on the input point coordinate and the
  corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
  Tree. Based on the tree, our ToL agent not only comprehends the content of the
  indicated area but also articulates the layout and spatial relationships
  between elements. Such layout information is crucial for accurately
  interpreting information on the screen, distinguishing our ToL agent from other
  screen reading tools. We also thoroughly evaluate the ToL agent against other
  baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
  web, and operating systems. Last but not least, we test the ToL agent on mobile
  GUI navigation tasks, demonstrating its utility in identifying incorrect
  actions along the path of agent execution trajectories. Code and data:
  screen-point-and-read.github.io
paper_ai_summary: A Tree-of-Lens agent addresses the Screen Point-and-Read task by utilizing a novel grounding mechanism to interpret content and layout in GUI screenshots.
paper_ai_keywords: Multimodal Large Language Models Tree-of-Lens Hierarchical Layout Tree screen reading GUI understanding SPR benchmark mobile GUI navigation