title: LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
summary: Text-rich images, where text serves as the central visual element guiding the
  overall understanding, are prevalent in real-world applications, such as
  presentation slides, scanned documents, and webpage snapshots. Tasks involving
  multiple text-rich images are especially challenging, as they require not only
  understanding the content of individual images but reasoning about
  inter-relationships and logical flows across multiple visual inputs. Despite
  the importance of these scenarios, current multimodal large language models
  (MLLMs) struggle to handle such tasks due to two key challenges: (1) the
  scarcity of high-quality instruction tuning datasets for text-rich multi-image
  scenarios, and (2) the difficulty in balancing image resolution with visual
  feature sequence length. To address these challenges, we propose \OurMethod, a
  MLLM designed specifically for handling vision-language tasks involving
  multiple text-rich images. First, we curated about one million high-quality
  multimodal instruction-tuning data, tailored to text-rich, multi-image
  scenarios. Second, we developed an adaptive high-resolution multi-image
  encoding module to dynamically optimize the allocation of visual sequence
  length based on the original aspect ratios and resolutions of the input images.
  Experiments across a wide range of benchmarks demonstrate our model's superior
  capabilities in text-rich, multi-image evaluations and competitive performance
  in general domain evaluations.
paper_authors._id: 66fe1c9f81382f012c968fbe
paper_authors.name: Mengzhao Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:56.605Z
paper_authors._id: 642bca844d7a550711e7beac
paper_authors.avatarUrl: /avatars/9f740ceb7c97b3abf819a1f2d58c7154.svg
paper_authors.fullname: JillJia
paper_authors.type: user
paper_authors.user: JillJia
paper_authors._id: 66fe1c9f81382f012c968fbf
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T08:46:36.047Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 66fe1c9f81382f012c968fc0
paper_authors.name: Kaixin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T08:46:55.518Z
paper_authors._id: 64ae4f6280f308a395fd7c19
paper_authors.avatarUrl: /avatars/5f1330f8187cd5e66aa517303659f110.svg
paper_authors.fullname: Kaixin Ma
paper_authors.type: user
paper_authors.user: kaixinm
paper_authors._id: 66fe1c9f81382f012c968fc1
paper_authors.name: Tianqing Fang
paper_authors._id: 66fe1c9f81382f012c968fc2
paper_authors.name: Zhihan Zhang
paper_authors._id: 66fe1c9f81382f012c968fc3
paper_authors.name: Siru Ouyang
paper_authors._id: 66fe1c9f81382f012c968fc4
paper_authors.name: Hongming Zhang
paper_authors._id: 66fe1c9f81382f012c968fc5
paper_authors.name: Meng Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:55:28.939Z
paper_authors._id: 62cdc931c589e4a9e2411d39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
paper_authors.fullname: Meng Jiang
paper_authors.type: user
paper_authors.user: mjiang89
paper_authors._id: 66fe1c9f81382f012c968fc6
paper_authors.name: Dong Yu
paper_title: LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
paper_summary: Text-rich images, where text serves as the central visual element guiding the
  overall understanding, are prevalent in real-world applications, such as
  presentation slides, scanned documents, and webpage snapshots. Tasks involving
  multiple text-rich images are especially challenging, as they require not only
  understanding the content of individual images but reasoning about
  inter-relationships and logical flows across multiple visual inputs. Despite
  the importance of these scenarios, current multimodal large language models
  (MLLMs) struggle to handle such tasks due to two key challenges: (1) the
  scarcity of high-quality instruction tuning datasets for text-rich multi-image
  scenarios, and (2) the difficulty in balancing image resolution with visual
  feature sequence length. To address these challenges, we propose \OurMethod, a
  MLLM designed specifically for handling vision-language tasks involving
  multiple text-rich images. First, we curated about one million high-quality
  multimodal instruction-tuning data, tailored to text-rich, multi-image
  scenarios. Second, we developed an adaptive high-resolution multi-image
  encoding module to dynamically optimize the allocation of visual sequence
  length based on the original aspect ratios and resolutions of the input images.
  Experiments across a wide range of benchmarks demonstrate our model's superior
  capabilities in text-rich, multi-image evaluations and competitive performance
  in general domain evaluations.
paper_ai_keywords: multimodal large language models (MLLMs) instruction tuning datasets visual sequence length adaptive high-resolution multi-image encoding module