title: Harnessing Webpage UIs for Text-Rich Visual Understanding
summary: Text-rich visual understanding-the ability to process environments where
  dense textual content is integrated with visuals-is crucial for multimodal
  large language models (MLLMs) to interact effectively with structured
  environments. To enhance this capability, we propose synthesizing general
  multimodal instructions from webpage UIs using text-based large language models
  (LLMs). Despite lacking direct visual input, text-based LLMs are able to
  process structured text representations from webpage accessibility trees. These
  instructions are then paired with UI screenshots to train multimodal models. We
  introduce MultiUI, a dataset containing 7.3 million samples from 1 million
  websites, covering diverse multimodal tasks and UI layouts. Models trained on
  MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
  VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
  Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
  non-UI domains, such as document understanding, OCR, and chart interpretation.
  These results highlight the broad applicability of web UI data for advancing
  text-rich visual understanding across various scenarios.
paper_authors._id: 6711c43e0babf0ff2ec5f194
paper_authors.name: Junpeng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:38.288Z
paper_authors._id: 6512a2e284bedffb5ac5a511
paper_authors.avatarUrl: /avatars/e798f2c8633d77aea41fce684d749390.svg
paper_authors.fullname: Junpeng Liu
paper_authors.type: user
paper_authors.user: jeepliu
paper_authors._id: 6711c43e0babf0ff2ec5f195
paper_authors.name: Tianyue Ou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T09:57:06.209Z
paper_authors._id: 65403d8781a8731a1c09a584
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65403d8781a8731a1c09a584/858OEZnMxbJleFiI1UgGs.jpeg
paper_authors.fullname: Tianyue Ou
paper_authors.type: user
paper_authors.user: oottyy
paper_authors._id: 6711c43e0babf0ff2ec5f196
paper_authors.name: Yifan Song
paper_authors._id: 6711c43e0babf0ff2ec5f197
paper_authors.name: Yuxiao Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:33:26.181Z
paper_authors._id: 6500bbf5e102da55f9ed43fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6500bbf5e102da55f9ed43fc/QZ6EAFV2CStFsILmTJw5D.jpeg
paper_authors.fullname: Yuxiao Qu
paper_authors.type: user
paper_authors.user: CohenQu
paper_authors._id: 6711c43e0babf0ff2ec5f198
paper_authors.name: Wai Lam
paper_authors._id: 6711c43e0babf0ff2ec5f199
paper_authors.name: Chenyan Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:33:36.903Z
paper_authors._id: 617ae49bc0ff6006217aa22e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635543252231-617ae49bc0ff6006217aa22e.jpeg
paper_authors.fullname: Chenyan Xiong
paper_authors.type: user
paper_authors.user: xiongchenyan
paper_authors._id: 6711c43e0babf0ff2ec5f19a
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:33:42.084Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 6711c43e0babf0ff2ec5f19b
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:33:49.506Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 6711c43e0babf0ff2ec5f19c
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:35.934Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_title: Harnessing Webpage UIs for Text-Rich Visual Understanding
paper_summary: Text-rich visual understanding-the ability to process environments where
  dense textual content is integrated with visuals-is crucial for multimodal
  large language models (MLLMs) to interact effectively with structured
  environments. To enhance this capability, we propose synthesizing general
  multimodal instructions from webpage UIs using text-based large language models
  (LLMs). Despite lacking direct visual input, text-based LLMs are able to
  process structured text representations from webpage accessibility trees. These
  instructions are then paired with UI screenshots to train multimodal models. We
  introduce MultiUI, a dataset containing 7.3 million samples from 1 million
  websites, covering diverse multimodal tasks and UI layouts. Models trained on
  MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
  VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
  Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
  non-UI domains, such as document understanding, OCR, and chart interpretation.
  These results highlight the broad applicability of web UI data for advancing
  text-rich visual understanding across various scenarios.
paper_ai_summary: MultiUI, a dataset from webpage UIs, enhances multimodal models' ability to handle text-rich visual environments, improving performance on web and non-web tasks.
paper_ai_keywords: multimodal large language models MLLMs text-based large language models LLMs accessibility trees multimodal instructions UI screenshots VisualWebBench Mind2Web action accuracy document understanding OCR chart interpretation