Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks
<|reference_start|>Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks: Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs. Despite the importance of these scenarios, current multimodal large language models (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. To address these challenges, we propose Leopard, a MLLM designed specifically for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we developed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios and resolutions of the input images. Experiments across a wide range of benchmarks demonstrate our model's superior capabilities in text-rich, multi-image evaluations and competitive performance in general domain evaluations.<|reference_end|>
@article{jia2024leopard:,
    title={Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks},
    author={Mengzhao Jia, Wenhao Yu, Kaixin Ma, Tianqing Fang, Zhihan Zhang, Siru
  Ouyang, Hongming Zhang, Meng Jiang, Dong Yu},
    journal={arXiv preprint arXiv:2410.01744},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01744},
    primaryClass={cs.CV cs.CL}
}