title: Needle In A Multimodal Haystack
summary: With the rapid advancement of multimodal large language models (MLLMs), their
  evaluation has become increasingly comprehensive. However, understanding long
  multimodal content, as a foundational ability for real-world applications,
  remains underexplored. In this work, we present Needle In A Multimodal Haystack
  (MM-NIAH), the first benchmark specifically designed to systematically evaluate
  the capability of existing MLLMs to comprehend long multimodal documents. Our
  benchmark includes three types of evaluation tasks: multimodal retrieval,
  counting, and reasoning. In each task, the model is required to answer the
  questions according to different key information scattered throughout the given
  multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that
  existing models still have significant room for improvement on these tasks,
  especially on vision-centric evaluation. We hope this work can provide a
  platform for further research on long multimodal document comprehension and
  contribute to the advancement of MLLMs. Code and benchmark are released at
  https://github.com/OpenGVLab/MM-NIAH.
paper_authors._id: 6669285a759649b9b0a567c5
paper_authors.name: Weiyun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:52:35.011Z
paper_authors._id: 619507e7b74b6c591f794340
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
paper_authors.fullname: Weiyun Wang
paper_authors.type: user
paper_authors.user: Weiyun1025
paper_authors._id: 6669285a759649b9b0a567c6
paper_authors.name: Shuibo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:50:50.497Z
paper_authors._id: 666ff0e9251af54f960e9870
paper_authors.avatarUrl: /avatars/c6b727821e29f9d40afdf0ef6935cd0b.svg
paper_authors.fullname: Shuibai Zhang
paper_authors.type: user
paper_authors.user: Shuibai12138
paper_authors._id: 6669285a759649b9b0a567c7
paper_authors.name: Yiming Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:52:50.031Z
paper_authors._id: 659f996896e2038d1aa2a77c
paper_authors.avatarUrl: /avatars/6f6204f9d9a7aca63c2b4599b3065977.svg
paper_authors.fullname: yiming ren
paper_authors.type: user
paper_authors.user: Flash128
paper_authors._id: 6669285a759649b9b0a567c8
paper_authors.name: Yuchen Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T13:43:24.553Z
paper_authors._id: 6495a4d6d1cec0e2c2c96cdd
paper_authors.avatarUrl: /avatars/e8b42301514094c8af6fa1e5fcb53119.svg
paper_authors.fullname: Duan Yuchen
paper_authors.type: user
paper_authors.user: duanyuchen
paper_authors._id: 6669285a759649b9b0a567c9
paper_authors.name: Tiantong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:52:55.709Z
paper_authors._id: 645efb6fb5c9a8666d105401
paper_authors.avatarUrl: /avatars/39d6c51cd1f15319b6c4efe9c72f1df2.svg
paper_authors.fullname: Tiantong Li
paper_authors.type: user
paper_authors.user: casually5101
paper_authors._id: 6669285a759649b9b0a567ca
paper_authors.name: Shuo Liu
paper_authors._id: 6669285a759649b9b0a567cb
paper_authors.name: Mengkang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:07:29.396Z
paper_authors._id: 6237df4a5ab9df625fb70c1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6237df4a5ab9df625fb70c1a/rmtSgp-_WvHZBXnzPrXL0.jpeg
paper_authors.fullname: Mengkang Hu
paper_authors.type: user
paper_authors.user: MengkangHu
paper_authors._id: 6669285a759649b9b0a567cc
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:49:33.003Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 6669285a759649b9b0a567cd
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:33:57.728Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 6669285a759649b9b0a567ce
paper_authors.name: Lewei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:53:50.340Z
paper_authors._id: 65ead3ea908526a39082e641
paper_authors.avatarUrl: /avatars/dcf870695fd56b06ca03d82f831e9019.svg
paper_authors.fullname: Lewei Lu
paper_authors.type: user
paper_authors.user: luotto
paper_authors._id: 6669285a759649b9b0a567cf
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:53:56.048Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 6669285a759649b9b0a567d0
paper_authors.name: Ping Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:54:07.526Z
paper_authors._id: 666bd450d528e9911600b302
paper_authors.avatarUrl: /avatars/30fd6b03b70af8018e8d5f99b89c68a5.svg
paper_authors.fullname: luoping1121
paper_authors.type: user
paper_authors.user: luoping
paper_authors._id: 6669285a759649b9b0a567d1
paper_authors.name: Yu Qiao
paper_authors._id: 6669285a759649b9b0a567d2
paper_authors.name: Jifeng Dai
paper_authors._id: 6669285a759649b9b0a567d3
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:54:17.560Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 6669285a759649b9b0a567d4
paper_authors.name: Wenhai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T13:54:39.048Z
paper_authors._id: 64d1c560c0c627dfa71bdbe0
paper_authors.avatarUrl: /avatars/f42794fe25bffcd870a1bcee69b95298.svg
paper_authors.fullname: wenhai.wang
paper_authors.type: user
paper_authors.user: wangwhcore
paper_title: Needle In A Multimodal Haystack
paper_summary: With the rapid advancement of multimodal large language models (MLLMs), their
  evaluation has become increasingly comprehensive. However, understanding long
  multimodal content, as a foundational ability for real-world applications,
  remains underexplored. In this work, we present Needle In A Multimodal Haystack
  (MM-NIAH), the first benchmark specifically designed to systematically evaluate
  the capability of existing MLLMs to comprehend long multimodal documents. Our
  benchmark includes three types of evaluation tasks: multimodal retrieval,
  counting, and reasoning. In each task, the model is required to answer the
  questions according to different key information scattered throughout the given
  multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that
  existing models still have significant room for improvement on these tasks,
  especially on vision-centric evaluation. We hope this work can provide a
  platform for further research on long multimodal document comprehension and
  contribute to the advancement of MLLMs. Code and benchmark are released at
  https://github.com/OpenGVLab/MM-NIAH.
paper_ai_summary: NN-NIAH is a benchmark that evaluates MLLMs' comprehension of long multimodal documents across retrieval, counting, and reasoning tasks.
paper_ai_keywords: multimodal large language models MLLMs Needle In A Multimodal Haystack NN-NIAH multimodal retrieval multimodal document comprehension vision-centric evaluation