Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of
  Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have shown significant promise in
various applications, leading to broad interest from researchers and
practitioners alike. However, a comprehensive evaluation of their long-context
capabilities remains underexplored. To address these gaps, we introduce the
MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to
assess the long-context capabilities of MLLMs. Besides multi-image input, we
employ image stitching to further increase the input context length, and
develop a protocol to automatically generate labels for sub-image level
retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their
capability to locate a target sub-image (needle) within a set of images
(haystack) based on textual instructions and descriptions of image contents.
This setup necessitates an advanced understanding of extensive visual contexts
and effective information retrieval within long-context image inputs. With this
benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and
open-source models. The findings reveal that GPT-4o consistently surpasses
other models in long-context scenarios, but suffers from hallucination problems
in negative samples, i.e., when needles are not in the haystacks. Our
comprehensive long-context evaluation of MLLMs also sheds lights on the
considerable performance gap between API-based and open-source models. All the
code, data, and instructions required to reproduce the main results are
available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.
6673b574c7356550f6db7c92
Hengyi Wang
extracted_confirmed
2024-06-20T05:05:42.747Z
658de969a33889b27f1da90a
/avatars/12da03b995742415ab446ff8d5b0439e.svg
Hengyi Wang
user
aaronwhy
6673b574c7356550f6db7c93
Haizhou Shi
admin_assigned
2024-06-20T07:35:54.475Z
658ceaae79c9c9b95f4714a6
/avatars/1e9db134e57afabc6d2e9ee32f0929d0.svg
Haizhou Shi
user
n1h111sm
6673b574c7356550f6db7c94
Shiwei Tan
admin_assigned
2024-06-20T07:36:00.177Z
630d80d7c169245d7808c242
/avatars/a6441b90ba67373bc49a34a101a29722.svg
Shiwei Tan
user
SKFE
6673b574c7356550f6db7c95
Weiyi Qin
claimed_verified
2024-06-20T07:27:29.222Z
65541465fc2c1019ab332492
/avatars/ed377e2d2d6652dda9fd165d2c8a9fa7.svg
Weiyi Qin
user
W8Yi
6673b574c7356550f6db7c96
Wenyuan Wang
claimed_verified
2024-06-21T07:06:55.142Z
6674290bf0bfd6fc217d8bb2
/avatars/458779d15c9ae97ef61b6fd48612b56d.svg
Wenyuan Wang
user
wenyuan99
6673b574c7356550f6db7c97
Tunyu Zhang
claimed_verified
2025-05-02T06:41:19.145Z
66b7a545c5a90c1aa84ca807
/avatars/6bfda13d57474cabd2165b170f67ee51.svg
Tunyu Zhang
user
Tyrion279
6673b574c7356550f6db7c98
Akshay Nambi
admin_assigned
2024-06-20T07:39:56.518Z
64aba383fddf117e6e5ba818
/avatars/ee7d25d865b34be5902872d060ad9153.svg
Akshay  Nambi
user
akshaynambi
6673b574c7356550f6db7c99
Tanuja Ganu
6673b574c7356550f6db7c9a
Hao Wang
claimed_verified
2024-06-20T07:27:23.699Z
630ebfa278be96577258b67a
/avatars/dbff77354763bd934639aa3740e62ef1.svg
Hao Wang
user
js05212
Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of
  Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have shown significant promise in
various applications, leading to broad interest from researchers and
practitioners alike. However, a comprehensive evaluation of their long-context
capabilities remains underexplored. To address these gaps, we introduce the
MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to
assess the long-context capabilities of MLLMs. Besides multi-image input, we
employ image stitching to further increase the input context length, and
develop a protocol to automatically generate labels for sub-image level
retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their
capability to locate a target sub-image (needle) within a set of images
(haystack) based on textual instructions and descriptions of image contents.
This setup necessitates an advanced understanding of extensive visual contexts
and effective information retrieval within long-context image inputs. With this
benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and
open-source models. The findings reveal that GPT-4o consistently surpasses
other models in long-context scenarios, but suffers from hallucination problems
in negative samples, i.e., when needles are not in the haystacks. Our
comprehensive long-context evaluation of MLLMs also sheds lights on the
considerable performance gap between API-based and open-source models. All the
code, data, and instructions required to reproduce the main results are
available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.
The MultiModal Needle-in-a-haystack benchmark assesses the long-context capabilities of Multimodal Large Language Models by evaluating their performance in locating sub-images within a set of images based on textual instructions.
Multimodal Large Language Models MLLMs MultiModal Needle-in-a-haystack MMNeedle long-context capabilities image stitching image inpainting sub-image level retrieval GPT-4o hallucination problems performance gap