title: Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
summary: Recent advancements in multimodal large language models (MLLMs) have been
  noteworthy, yet, these general-domain MLLMs often fall short in their ability
  to comprehend and interact effectively with user interface (UI) screens. In
  this paper, we present Ferret-UI, a new MLLM tailored for enhanced
  understanding of mobile UI screens, equipped with referring, grounding, and
  reasoning capabilities. Given that UI screens typically exhibit a more
  elongated aspect ratio and contain smaller objects of interest (e.g., icons,
  texts) than natural images, we incorporate "any resolution" on top of Ferret to
  magnify details and leverage enhanced visual features. Specifically, each
  screen is divided into 2 sub-images based on the original aspect ratio (i.e.,
  horizontal division for portrait screens and vertical division for landscape
  screens). Both sub-images are encoded separately before being sent to LLMs. We
  meticulously gather training samples from an extensive range of elementary UI
  tasks, such as icon recognition, find text, and widget listing. These samples
  are formatted for instruction-following with region annotations to facilitate
  precise referring and grounding. To augment the model's reasoning ability, we
  further compile a dataset for advanced tasks, including detailed description,
  perception/interaction conversations, and function inference. After training on
  the curated datasets, Ferret-UI exhibits outstanding comprehension of UI
  screens and the capability to execute open-ended instructions. For model
  evaluation, we establish a comprehensive benchmark encompassing all the
  aforementioned tasks. Ferret-UI excels not only beyond most open-source UI
  MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.
paper_authors._id: 6614b91239e6b179c4a0f946
paper_authors.name: Keen You
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:12:15.037Z
paper_authors._id: 6150a95100a52e55be2994eb
paper_authors.avatarUrl: /avatars/c8e2f85a00ca54e1984a692f29f3096f.svg
paper_authors.fullname: Keen You
paper_authors.type: user
paper_authors.user: keenyou
paper_authors._id: 6614b91239e6b179c4a0f947
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:48:01.045Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 6614b91239e6b179c4a0f948
paper_authors.name: Eldon Schoop
paper_authors._id: 6614b91239e6b179c4a0f949
paper_authors.name: Floris Weers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:48:11.807Z
paper_authors._id: 65f41f1d6724ea19d7bc9052
paper_authors.avatarUrl: /avatars/4ccd6279aac3a18e1b23642a1a549e2f.svg
paper_authors.fullname: Floris Weers
paper_authors.type: user
paper_authors.user: fweers
paper_authors._id: 6614b91239e6b179c4a0f94a
paper_authors.name: Amanda Swearngin
paper_authors._id: 6614b91239e6b179c4a0f94b
paper_authors.name: Jeffrey Nichols
paper_authors._id: 6614b91239e6b179c4a0f94c
paper_authors.name: Yinfei Yang
paper_authors._id: 6614b91239e6b179c4a0f94d
paper_authors.name: Zhe Gan
paper_title: Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
paper_summary: Recent advancements in multimodal large language models (MLLMs) have been
  noteworthy, yet, these general-domain MLLMs often fall short in their ability
  to comprehend and interact effectively with user interface (UI) screens. In
  this paper, we present Ferret-UI, a new MLLM tailored for enhanced
  understanding of mobile UI screens, equipped with referring, grounding, and
  reasoning capabilities. Given that UI screens typically exhibit a more
  elongated aspect ratio and contain smaller objects of interest (e.g., icons,
  texts) than natural images, we incorporate "any resolution" on top of Ferret to
  magnify details and leverage enhanced visual features. Specifically, each
  screen is divided into 2 sub-images based on the original aspect ratio (i.e.,
  horizontal division for portrait screens and vertical division for landscape
  screens). Both sub-images are encoded separately before being sent to LLMs. We
  meticulously gather training samples from an extensive range of elementary UI
  tasks, such as icon recognition, find text, and widget listing. These samples
  are formatted for instruction-following with region annotations to facilitate
  precise referring and grounding. To augment the model's reasoning ability, we
  further compile a dataset for advanced tasks, including detailed description,
  perception/interaction conversations, and function inference. After training on
  the curated datasets, Ferret-UI exhibits outstanding comprehension of UI
  screens and the capability to execute open-ended instructions. For model
  evaluation, we establish a comprehensive benchmark encompassing all the
  aforementioned tasks. Ferret-UI excels not only beyond most open-source UI
  MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.
paper_ai_summary: Ferret-UI, a multimodal large language model tailored for mobile UI screens, enhances understanding and interaction through region annotations and a comprehensive dataset of UI tasks, outperforming existing models including GPT-4V.
paper_ai_keywords: multimodal large language models MLLMs UI screens referring grounding reasoning capabilities any resolution sub-images region annotations instruction-following detailed description perception/interaction conversations function inference benchmark