LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
Text-rich images, where text serves as the central visual element guiding the
overall understanding, are prevalent in real-world applications, such as
presentation slides, scanned documents, and webpage snapshots. Tasks involving
multiple text-rich images are especially challenging, as they require not only
understanding the content of individual images but reasoning about
inter-relationships and logical flows across multiple visual inputs. Despite
the importance of these scenarios, current multimodal large language models
(MLLMs) struggle to handle such tasks due to two key challenges: (1) the
scarcity of high-quality instruction tuning datasets for text-rich multi-image
scenarios, and (2) the difficulty in balancing image resolution with visual
feature sequence length. To address these challenges, we propose \OurMethod, a
MLLM designed specifically for handling vision-language tasks involving
multiple text-rich images. First, we curated about one million high-quality
multimodal instruction-tuning data, tailored to text-rich, multi-image
scenarios. Second, we developed an adaptive high-resolution multi-image
encoding module to dynamically optimize the allocation of visual sequence
length based on the original aspect ratios and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior
capabilities in text-rich, multi-image evaluations and competitive performance
in general domain evaluations.
66fe1c9f81382f012c968fbe
Mengzhao Jia
claimed_verified
2024-10-04T07:31:56.605Z
642bca844d7a550711e7beac
/avatars/9f740ceb7c97b3abf819a1f2d58c7154.svg
JillJia
user
JillJia
66fe1c9f81382f012c968fbf
Wenhao Yu
admin_assigned
2024-10-03T08:46:36.047Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
66fe1c9f81382f012c968fc0
Kaixin Ma
admin_assigned
2024-10-03T08:46:55.518Z
64ae4f6280f308a395fd7c19
/avatars/5f1330f8187cd5e66aa517303659f110.svg
Kaixin Ma
user
kaixinm
66fe1c9f81382f012c968fc1
Tianqing Fang
66fe1c9f81382f012c968fc2
Zhihan Zhang
66fe1c9f81382f012c968fc3
Siru Ouyang
66fe1c9f81382f012c968fc4
Hongming Zhang
66fe1c9f81382f012c968fc5
Meng Jiang
claimed_verified
2024-10-31T09:55:28.939Z
62cdc931c589e4a9e2411d39
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
Meng Jiang
user
mjiang89
66fe1c9f81382f012c968fc6
Dong Yu
LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
Text-rich images, where text serves as the central visual element guiding the
overall understanding, are prevalent in real-world applications, such as
presentation slides, scanned documents, and webpage snapshots. Tasks involving
multiple text-rich images are especially challenging, as they require not only
understanding the content of individual images but reasoning about
inter-relationships and logical flows across multiple visual inputs. Despite
the importance of these scenarios, current multimodal large language models
(MLLMs) struggle to handle such tasks due to two key challenges: (1) the
scarcity of high-quality instruction tuning datasets for text-rich multi-image
scenarios, and (2) the difficulty in balancing image resolution with visual
feature sequence length. To address these challenges, we propose \OurMethod, a
MLLM designed specifically for handling vision-language tasks involving
multiple text-rich images. First, we curated about one million high-quality
multimodal instruction-tuning data, tailored to text-rich, multi-image
scenarios. Second, we developed an adaptive high-resolution multi-image
encoding module to dynamically optimize the allocation of visual sequence
length based on the original aspect ratios and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior
capabilities in text-rich, multi-image evaluations and competitive performance
in general domain evaluations.
A model designed for multimodal tasks involving multiple text-rich images addresses dataset scarcity and visual feature balancing, outperforming existing models in specialized and general evaluations.
multimodal large language models high-quality multimodal instruction-tuning data adaptive high-resolution multi-image encoding module visual sequence length vision-language tasks text-rich images