title: MMLongBench: Benchmarking Long-Context Vision-Language Models
    Effectively and Thoroughly
summary: The rapid extension of context windows in large vision-language models has
  given rise to long-context vision-language models (LCVLMs), which are capable
  of handling hundreds of images with interleaved text tokens in a single forward
  pass. In this work, we introduce MMLongBench, the first benchmark covering a
  diverse set of long-context vision-language tasks, to evaluate LCVLMs
  effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning
  five different categories of downstream tasks, such as Visual RAG and Many-Shot
  ICL. It also provides broad coverage of image types, including various natural
  and synthetic images. To assess the robustness of the models to different input
  lengths, all examples are delivered at five standardized input lengths (8K-128K
  tokens) via a cross-modal tokenization scheme that combines vision patches and
  text tokens. Through a thorough benchmarking of 46 closed-source and
  open-source LCVLMs, we provide a comprehensive analysis of the current models'
  vision-language long-context ability. Our results show that: i) performance on
  a single task is a weak proxy for overall long-context capability; ii) both
  closed-source and open-source models face challenges in long-context
  vision-language tasks, indicating substantial room for future improvement; iii)
  models with stronger reasoning ability tend to exhibit better long-context
  performance. By offering wide task coverage, various image types, and rigorous
  length control, MMLongBench provides the missing foundation for diagnosing and
  advancing the next generation of LCVLMs.
paper_authors._id: 682adaf581c740ab4aabc5a3
paper_authors.name: Zhaowei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:12.290Z
paper_authors._id: 62281c11236b7b2eefa7f198
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62281c11236b7b2eefa7f198/xuTCVMUahpg4Mfb7L62sm.jpeg
paper_authors.fullname: Zhaowei Wang
paper_authors.type: user
paper_authors.user: ZhaoweiWang
paper_authors._id: 682adaf581c740ab4aabc5a4
paper_authors.name: Wenhao Yu
paper_authors._id: 682adaf581c740ab4aabc5a5
paper_authors.name: Xiyu Ren
paper_authors._id: 682adaf581c740ab4aabc5a6
paper_authors.name: Jipeng Zhang
paper_authors._id: 682adaf581c740ab4aabc5a7
paper_authors.name: Yu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:10.234Z
paper_authors._id: 654a500d9b8bd6406d431c0d
paper_authors.avatarUrl: /avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
paper_authors.fullname: Yu Zhao
paper_authors.type: user
paper_authors.user: yuzhaouoe
paper_authors._id: 682adaf581c740ab4aabc5a8
paper_authors.name: Rohit Saxena
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:08.362Z
paper_authors._id: 657ccbf2869d5bb0e53b482f
paper_authors.avatarUrl: /avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
paper_authors.fullname: Rohit Saxena
paper_authors.type: user
paper_authors.user: rohitsaxena
paper_authors._id: 682adaf581c740ab4aabc5a9
paper_authors.name: Liang Cheng
paper_authors._id: 682adaf581c740ab4aabc5aa
paper_authors.name: Ginny Wong
paper_authors._id: 682adaf581c740ab4aabc5ab
paper_authors.name: Simon See
paper_authors._id: 682adaf581c740ab4aabc5ac
paper_authors.name: Pasquale Minervini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T10:30:09.684Z
paper_authors._id: 61001311e043e15c13412d30
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
paper_authors.fullname: Pasquale Minervini
paper_authors.type: user
paper_authors.user: pminervini
paper_authors._id: 682adaf581c740ab4aabc5ad
paper_authors.name: Yangqiu Song
paper_authors._id: 682adaf581c740ab4aabc5ae
paper_authors.name: Mark Steedman
paper_title: MMLongBench: Benchmarking Long-Context Vision-Language Models
    Effectively and Thoroughly
paper_summary: The rapid extension of context windows in large vision-language models has
  given rise to long-context vision-language models (LCVLMs), which are capable
  of handling hundreds of images with interleaved text tokens in a single forward
  pass. In this work, we introduce MMLongBench, the first benchmark covering a
  diverse set of long-context vision-language tasks, to evaluate LCVLMs
  effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning
  five different categories of downstream tasks, such as Visual RAG and Many-Shot
  ICL. It also provides broad coverage of image types, including various natural
  and synthetic images. To assess the robustness of the models to different input
  lengths, all examples are delivered at five standardized input lengths (8K-128K
  tokens) via a cross-modal tokenization scheme that combines vision patches and
  text tokens. Through a thorough benchmarking of 46 closed-source and
  open-source LCVLMs, we provide a comprehensive analysis of the current models'
  vision-language long-context ability. Our results show that: i) performance on
  a single task is a weak proxy for overall long-context capability; ii) both
  closed-source and open-source models face challenges in long-context
  vision-language tasks, indicating substantial room for future improvement; iii)
  models with stronger reasoning ability tend to exhibit better long-context
  performance. By offering wide task coverage, various image types, and rigorous
  length control, MMLongBench provides the missing foundation for diagnosing and
  advancing the next generation of LCVLMs.
paper_ai_summary: MMLongBench is a benchmark that evaluates long-context vision-language models across various tasks, image types, and input lengths, revealing that single-task performance is insufficient for gauging overall capability.
paper_ai_keywords: long-context vision-language models LCVLMs Visual RAG Many-Shot ICL cross-modal tokenization vision patches reasoning ability