Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
  Grounding
Graphical User Interfaces (GUIs) are central to our interaction with digital
devices. Recently, growing efforts have been made to build models for various
GUI understanding tasks. However, these efforts largely overlook an important
GUI-referring task: screen reading based on user-indicated points, which we
name the Screen Point-and-Read (SPR) task. This task is predominantly handled
by rigid accessible screen reading tools, in great need of new models driven by
advancements in Multimodal Large Language Models (MLLMs). In this paper, we
propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
to address the SPR task. Based on the input point coordinate and the
corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
Tree. Based on the tree, our ToL agent not only comprehends the content of the
indicated area but also articulates the layout and spatial relationships
between elements. Such layout information is crucial for accurately
interpreting information on the screen, distinguishing our ToL agent from other
screen reading tools. We also thoroughly evaluate the ToL agent against other
baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
web, and operating systems. Last but not least, we test the ToL agent on mobile
GUI navigation tasks, demonstrating its utility in identifying incorrect
actions along the path of agent execution trajectories. Code and data:
screen-point-and-read.github.io
667e21ecddca7a5233b55dcf
Yue Fan
claimed_verified
2024-06-30T20:57:37.569Z
656b8a62e8bf55919a6a2d61
/avatars/26b75683dff73c51f966e9c15f04fd9f.svg
Yue Fan
user
yfan1997
667e21ecddca7a5233b55dd0
Lei Ding
667e21ecddca7a5233b55dd1
Ching-Chen Kuo
667e21ecddca7a5233b55dd2
Shan Jiang
667e21ecddca7a5233b55dd3
Yang Zhao
667e21ecddca7a5233b55dd4
Xinze Guan
667e21ecddca7a5233b55dd5
Jie Yang
667e21ecddca7a5233b55dd6
Yi Zhang
667e21ecddca7a5233b55dd7
Xin Eric Wang
claimed_verified
2024-06-28T08:02:53.091Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
  Grounding
Graphical User Interfaces (GUIs) are central to our interaction with digital
devices. Recently, growing efforts have been made to build models for various
GUI understanding tasks. However, these efforts largely overlook an important
GUI-referring task: screen reading based on user-indicated points, which we
name the Screen Point-and-Read (SPR) task. This task is predominantly handled
by rigid accessible screen reading tools, in great need of new models driven by
advancements in Multimodal Large Language Models (MLLMs). In this paper, we
propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
to address the SPR task. Based on the input point coordinate and the
corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
Tree. Based on the tree, our ToL agent not only comprehends the content of the
indicated area but also articulates the layout and spatial relationships
between elements. Such layout information is crucial for accurately
interpreting information on the screen, distinguishing our ToL agent from other
screen reading tools. We also thoroughly evaluate the ToL agent against other
baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
web, and operating systems. Last but not least, we test the ToL agent on mobile
GUI navigation tasks, demonstrating its utility in identifying incorrect
actions along the path of agent execution trajectories. Code and data:
screen-point-and-read.github.io
A Tree-of-Lens agent addresses the Screen Point-and-Read task by utilizing a novel grounding mechanism to interpret content and layout in GUI screenshots.
Multimodal Large Language Models Tree-of-Lens Hierarchical Layout Tree screen reading GUI understanding SPR benchmark mobile GUI navigation