title: IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning
    in Multimodal LLMs
summary: Existing evaluation frameworks for Multimodal Large Language Models (MLLMs)
  primarily focus on image reasoning or general video understanding tasks,
  largely overlooking the significant role of image context in video
  comprehension. To bridge this gap, we propose IV-Bench, the first comprehensive
  benchmark for evaluating Image-Grounded Video Perception and Reasoning.
  IV-Bench consists of 967 videos paired with 2,585 meticulously annotated
  image-text queries across 13 tasks (7 perception and 6 reasoning tasks) and 5
  representative categories. Extensive evaluations of state-of-the-art
  open-source (e.g., InternVL2.5, Qwen2.5-VL) and closed-source (e.g., GPT-4o,
  Gemini2-Flash and Gemini2-Pro) MLLMs demonstrate that current models
  substantially underperform in image-grounded video Perception and Reasoning,
  merely achieving at most 28.9% accuracy. Further analysis reveals key factors
  influencing model performance on IV-Bench, including inference pattern, frame
  number, and resolution. Additionally, through a simple data synthesis approach,
  we demonstratethe challenges of IV- Bench extend beyond merely aligning the
  data format in the training proecss. These findings collectively provide
  valuable insights for future research. Our codes and data are released in
  https://github.com/multimodal-art-projection/IV-Bench.
paper_authors._id: 68084b04ba1dd0e6a077e09f
paper_authors.name: David Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:27:56.437Z
paper_authors._id: 64c910233d5a0dfed5ce5abb
paper_authors.avatarUrl: /avatars/8c73f380219c05ae7e7c2fad75a570d8.svg
paper_authors.fullname: dma
paper_authors.type: user
paper_authors.user: mdh98
paper_authors._id: 68084b04ba1dd0e6a077e0a0
paper_authors.name: Yuanxing Zhang
paper_authors._id: 68084b04ba1dd0e6a077e0a1
paper_authors.name: Jincheng Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:36:46.274Z
paper_authors._id: 6704ee27386892c420db1938
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6704ee27386892c420db1938/lb5mtEwYhn47RawkynYPs.jpeg
paper_authors.fullname: JinCheng Ren
paper_authors.type: user
paper_authors.user: JinChengRen
paper_authors._id: 68084b04ba1dd0e6a077e0a2
paper_authors.name: Jarvis Guo
paper_authors._id: 68084b04ba1dd0e6a077e0a3
paper_authors.name: Yifan Yao
paper_authors._id: 68084b04ba1dd0e6a077e0a4
paper_authors.name: Zhenlin Wei
paper_authors._id: 68084b04ba1dd0e6a077e0a5
paper_authors.name: Zhenzhu Yang
paper_authors._id: 68084b04ba1dd0e6a077e0a6
paper_authors.name: Zhongyuan Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:48:40.455Z
paper_authors._id: 63299f93688ad82b783aaf20
paper_authors.avatarUrl: /avatars/e68e6f5add62edddfbdd3795f3a72347.svg
paper_authors.fullname: zhongyuan peng
paper_authors.type: user
paper_authors.user: happzy2633
paper_authors._id: 68084b04ba1dd0e6a077e0a7
paper_authors.name: Boyu Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:48:34.343Z
paper_authors._id: 64337ff82bfb2b0ec75d5951
paper_authors.avatarUrl: /avatars/61c7189e759e8cd8aa59701ac86249a7.svg
paper_authors.fullname: Boyu Feng
paper_authors.type: user
paper_authors.user: fbyfby576
paper_authors._id: 68084b04ba1dd0e6a077e0a8
paper_authors.name: Jun Ma
paper_authors._id: 68084b04ba1dd0e6a077e0a9
paper_authors.name: Xiao Gu
paper_authors._id: 68084b04ba1dd0e6a077e0aa
paper_authors.name: Zhoufutu Wen
paper_authors._id: 68084b04ba1dd0e6a077e0ab
paper_authors.name: King Zhu
paper_authors._id: 68084b04ba1dd0e6a077e0ac
paper_authors.name: Yancheng He
paper_authors._id: 68084b04ba1dd0e6a077e0ad
paper_authors.name: Meng Cao
paper_authors._id: 68084b04ba1dd0e6a077e0ae
paper_authors.name: Shiwen Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:48:11.824Z
paper_authors._id: 641944acf9d6f1d772ede902
paper_authors.avatarUrl: /avatars/a2f88017f7ab0cbea6862201e35fe747.svg
paper_authors.fullname: ShiwenNi
paper_authors.type: user
paper_authors.user: ShiwenNi
paper_authors._id: 68084b04ba1dd0e6a077e0af
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:48:02.522Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 68084b04ba1dd0e6a077e0b0
paper_authors.name: Wenhao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:47:54.964Z
paper_authors._id: 641e5bf65f274a0a92c2f6a2
paper_authors.avatarUrl: /avatars/c15a54c51998c0e6367685e8e1737ec9.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: EZ-hwh
paper_authors._id: 68084b04ba1dd0e6a077e0b1
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:47:48.795Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 68084b04ba1dd0e6a077e0b2
paper_authors.name: Xiaojie Jin
paper_title: IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning
    in Multimodal LLMs
paper_summary: Existing evaluation frameworks for Multimodal Large Language Models (MLLMs)
  primarily focus on image reasoning or general video understanding tasks,
  largely overlooking the significant role of image context in video
  comprehension. To bridge this gap, we propose IV-Bench, the first comprehensive
  benchmark for evaluating Image-Grounded Video Perception and Reasoning.
  IV-Bench consists of 967 videos paired with 2,585 meticulously annotated
  image-text queries across 13 tasks (7 perception and 6 reasoning tasks) and 5
  representative categories. Extensive evaluations of state-of-the-art
  open-source (e.g., InternVL2.5, Qwen2.5-VL) and closed-source (e.g., GPT-4o,
  Gemini2-Flash and Gemini2-Pro) MLLMs demonstrate that current models
  substantially underperform in image-grounded video Perception and Reasoning,
  merely achieving at most 28.9% accuracy. Further analysis reveals key factors
  influencing model performance on IV-Bench, including inference pattern, frame
  number, and resolution. Additionally, through a simple data synthesis approach,
  we demonstratethe challenges of IV- Bench extend beyond merely aligning the
  data format in the training proecss. These findings collectively provide
  valuable insights for future research. Our codes and data are released in
  https://github.com/multimodal-art-projection/IV-Bench.
paper_ai_summary: IV-Bench evaluates Image-Grounded Video Perception and Reasoning in MLLMs, revealing substantial underperformance across multiple tasks and factors influencing model accuracy.
paper_ai_keywords: Multimodal Large Language Models MLLMs Image-Grounded Video Perception and Reasoning benchmark image-text queries inference pattern frame number resolution data synthesis