title: Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of
    Multimodal Large Language Models
summary: Multimodal Large Language Models (MLLMs) have shown significant promise in
  various applications, leading to broad interest from researchers and
  practitioners alike. However, a comprehensive evaluation of their long-context
  capabilities remains underexplored. To address these gaps, we introduce the
  MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to
  assess the long-context capabilities of MLLMs. Besides multi-image input, we
  employ image stitching to further increase the input context length, and
  develop a protocol to automatically generate labels for sub-image level
  retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their
  capability to locate a target sub-image (needle) within a set of images
  (haystack) based on textual instructions and descriptions of image contents.
  This setup necessitates an advanced understanding of extensive visual contexts
  and effective information retrieval within long-context image inputs. With this
  benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and
  open-source models. The findings reveal that GPT-4o consistently surpasses
  other models in long-context scenarios, but suffers from hallucination problems
  in negative samples, i.e., when needles are not in the haystacks. Our
  comprehensive long-context evaluation of MLLMs also sheds lights on the
  considerable performance gap between API-based and open-source models. All the
  code, data, and instructions required to reproduce the main results are
  available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.
paper_authors._id: 6673b574c7356550f6db7c92
paper_authors.name: Hengyi Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-20T05:05:42.747Z
paper_authors._id: 658de969a33889b27f1da90a
paper_authors.avatarUrl: /avatars/12da03b995742415ab446ff8d5b0439e.svg
paper_authors.fullname: Hengyi Wang
paper_authors.type: user
paper_authors.user: aaronwhy
paper_authors._id: 6673b574c7356550f6db7c93
paper_authors.name: Haizhou Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:35:54.475Z
paper_authors._id: 658ceaae79c9c9b95f4714a6
paper_authors.avatarUrl: /avatars/1e9db134e57afabc6d2e9ee32f0929d0.svg
paper_authors.fullname: Haizhou Shi
paper_authors.type: user
paper_authors.user: n1h111sm
paper_authors._id: 6673b574c7356550f6db7c94
paper_authors.name: Shiwei Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:36:00.177Z
paper_authors._id: 630d80d7c169245d7808c242
paper_authors.avatarUrl: /avatars/a6441b90ba67373bc49a34a101a29722.svg
paper_authors.fullname: Shiwei Tan
paper_authors.type: user
paper_authors.user: SKFE
paper_authors._id: 6673b574c7356550f6db7c95
paper_authors.name: Weiyi Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:27:29.222Z
paper_authors._id: 65541465fc2c1019ab332492
paper_authors.avatarUrl: /avatars/ed377e2d2d6652dda9fd165d2c8a9fa7.svg
paper_authors.fullname: Weiyi Qin
paper_authors.type: user
paper_authors.user: W8Yi
paper_authors._id: 6673b574c7356550f6db7c96
paper_authors.name: Wenyuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:06:55.142Z
paper_authors._id: 6674290bf0bfd6fc217d8bb2
paper_authors.avatarUrl: /avatars/458779d15c9ae97ef61b6fd48612b56d.svg
paper_authors.fullname: Wenyuan Wang
paper_authors.type: user
paper_authors.user: wenyuan99
paper_authors._id: 6673b574c7356550f6db7c97
paper_authors.name: Tunyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:41:19.145Z
paper_authors._id: 66b7a545c5a90c1aa84ca807
paper_authors.avatarUrl: /avatars/6bfda13d57474cabd2165b170f67ee51.svg
paper_authors.fullname: Tunyu Zhang
paper_authors.type: user
paper_authors.user: Tyrion279
paper_authors._id: 6673b574c7356550f6db7c98
paper_authors.name: Akshay Nambi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:39:56.518Z
paper_authors._id: 64aba383fddf117e6e5ba818
paper_authors.avatarUrl: /avatars/ee7d25d865b34be5902872d060ad9153.svg
paper_authors.fullname: Akshay  Nambi
paper_authors.type: user
paper_authors.user: akshaynambi
paper_authors._id: 6673b574c7356550f6db7c99
paper_authors.name: Tanuja Ganu
paper_authors._id: 6673b574c7356550f6db7c9a
paper_authors.name: Hao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:27:23.699Z
paper_authors._id: 630ebfa278be96577258b67a
paper_authors.avatarUrl: /avatars/dbff77354763bd934639aa3740e62ef1.svg
paper_authors.fullname: Hao Wang
paper_authors.type: user
paper_authors.user: js05212
paper_title: Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of
    Multimodal Large Language Models
paper_summary: Multimodal Large Language Models (MLLMs) have shown significant promise in
  various applications, leading to broad interest from researchers and
  practitioners alike. However, a comprehensive evaluation of their long-context
  capabilities remains underexplored. To address these gaps, we introduce the
  MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to
  assess the long-context capabilities of MLLMs. Besides multi-image input, we
  employ image stitching to further increase the input context length, and
  develop a protocol to automatically generate labels for sub-image level
  retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their
  capability to locate a target sub-image (needle) within a set of images
  (haystack) based on textual instructions and descriptions of image contents.
  This setup necessitates an advanced understanding of extensive visual contexts
  and effective information retrieval within long-context image inputs. With this
  benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and
  open-source models. The findings reveal that GPT-4o consistently surpasses
  other models in long-context scenarios, but suffers from hallucination problems
  in negative samples, i.e., when needles are not in the haystacks. Our
  comprehensive long-context evaluation of MLLMs also sheds lights on the
  considerable performance gap between API-based and open-source models. All the
  code, data, and instructions required to reproduce the main results are
  available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.
paper_ai_summary: The MultiModal Needle-in-a-haystack benchmark assesses the long-context capabilities of Multimodal Large Language Models by evaluating their performance in locating sub-images within a set of images based on textual instructions.
paper_ai_keywords: Multimodal Large Language Models MLLMs MultiModal Needle-in-a-haystack MMNeedle long-context capabilities image stitching image inpainting sub-image level retrieval GPT-4o hallucination problems performance gap