MMLongBench: Benchmarking Long-Context Vision-Language Models
  Effectively and Thoroughly
The rapid extension of context windows in large vision-language models has
given rise to long-context vision-language models (LCVLMs), which are capable
of handling hundreds of images with interleaved text tokens in a single forward
pass. In this work, we introduce MMLongBench, the first benchmark covering a
diverse set of long-context vision-language tasks, to evaluate LCVLMs
effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning
five different categories of downstream tasks, such as Visual RAG and Many-Shot
ICL. It also provides broad coverage of image types, including various natural
and synthetic images. To assess the robustness of the models to different input
lengths, all examples are delivered at five standardized input lengths (8K-128K
tokens) via a cross-modal tokenization scheme that combines vision patches and
text tokens. Through a thorough benchmarking of 46 closed-source and
open-source LCVLMs, we provide a comprehensive analysis of the current models'
vision-language long-context ability. Our results show that: i) performance on
a single task is a weak proxy for overall long-context capability; ii) both
closed-source and open-source models face challenges in long-context
vision-language tasks, indicating substantial room for future improvement; iii)
models with stronger reasoning ability tend to exhibit better long-context
performance. By offering wide task coverage, various image types, and rigorous
length control, MMLongBench provides the missing foundation for diagnosing and
advancing the next generation of LCVLMs.
682adaf581c740ab4aabc5a3
Zhaowei Wang
claimed_verified
2025-05-19T13:09:12.290Z
62281c11236b7b2eefa7f198
https://cdn-avatars.huggingface.co/v1/production/uploads/62281c11236b7b2eefa7f198/xuTCVMUahpg4Mfb7L62sm.jpeg
Zhaowei Wang
user
ZhaoweiWang
682adaf581c740ab4aabc5a4
Wenhao Yu
682adaf581c740ab4aabc5a5
Xiyu Ren
682adaf581c740ab4aabc5a6
Jipeng Zhang
682adaf581c740ab4aabc5a7
Yu Zhao
claimed_verified
2025-05-19T13:09:10.234Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
682adaf581c740ab4aabc5a8
Rohit Saxena
claimed_verified
2025-05-19T13:09:08.362Z
657ccbf2869d5bb0e53b482f
/avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
Rohit Saxena
user
rohitsaxena
682adaf581c740ab4aabc5a9
Liang Cheng
682adaf581c740ab4aabc5aa
Ginny Wong
682adaf581c740ab4aabc5ab
Simon See
682adaf581c740ab4aabc5ac
Pasquale Minervini
claimed_verified
2025-05-21T10:30:09.684Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
682adaf581c740ab4aabc5ad
Yangqiu Song
682adaf581c740ab4aabc5ae
Mark Steedman
MMLongBench: Benchmarking Long-Context Vision-Language Models
  Effectively and Thoroughly
The rapid extension of context windows in large vision-language models has
given rise to long-context vision-language models (LCVLMs), which are capable
of handling hundreds of images with interleaved text tokens in a single forward
pass. In this work, we introduce MMLongBench, the first benchmark covering a
diverse set of long-context vision-language tasks, to evaluate LCVLMs
effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning
five different categories of downstream tasks, such as Visual RAG and Many-Shot
ICL. It also provides broad coverage of image types, including various natural
and synthetic images. To assess the robustness of the models to different input
lengths, all examples are delivered at five standardized input lengths (8K-128K
tokens) via a cross-modal tokenization scheme that combines vision patches and
text tokens. Through a thorough benchmarking of 46 closed-source and
open-source LCVLMs, we provide a comprehensive analysis of the current models'
vision-language long-context ability. Our results show that: i) performance on
a single task is a weak proxy for overall long-context capability; ii) both
closed-source and open-source models face challenges in long-context
vision-language tasks, indicating substantial room for future improvement; iii)
models with stronger reasoning ability tend to exhibit better long-context
performance. By offering wide task coverage, various image types, and rigorous
length control, MMLongBench provides the missing foundation for diagnosing and
advancing the next generation of LCVLMs.
MMLongBench is a benchmark that evaluates long-context vision-language models across various tasks, image types, and input lengths, revealing that single-task performance is insufficient for gauging overall capability.
long-context vision-language models LCVLMs Visual RAG Many-Shot ICL cross-modal tokenization vision patches reasoning ability