IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning
  in Multimodal LLMs
Existing evaluation frameworks for Multimodal Large Language Models (MLLMs)
primarily focus on image reasoning or general video understanding tasks,
largely overlooking the significant role of image context in video
comprehension. To bridge this gap, we propose IV-Bench, the first comprehensive
benchmark for evaluating Image-Grounded Video Perception and Reasoning.
IV-Bench consists of 967 videos paired with 2,585 meticulously annotated
image-text queries across 13 tasks (7 perception and 6 reasoning tasks) and 5
representative categories. Extensive evaluations of state-of-the-art
open-source (e.g., InternVL2.5, Qwen2.5-VL) and closed-source (e.g., GPT-4o,
Gemini2-Flash and Gemini2-Pro) MLLMs demonstrate that current models
substantially underperform in image-grounded video Perception and Reasoning,
merely achieving at most 28.9% accuracy. Further analysis reveals key factors
influencing model performance on IV-Bench, including inference pattern, frame
number, and resolution. Additionally, through a simple data synthesis approach,
we demonstratethe challenges of IV- Bench extend beyond merely aligning the
data format in the training proecss. These findings collectively provide
valuable insights for future research. Our codes and data are released in
https://github.com/multimodal-art-projection/IV-Bench.
68084b04ba1dd0e6a077e09f
David Ma
claimed_verified
2025-04-23T08:27:56.437Z
64c910233d5a0dfed5ce5abb
/avatars/8c73f380219c05ae7e7c2fad75a570d8.svg
madehua
user
mdh98
68084b04ba1dd0e6a077e0a0
Yuanxing Zhang
68084b04ba1dd0e6a077e0a1
Jincheng Ren
claimed_verified
2025-04-23T08:36:46.274Z
6704ee27386892c420db1938
https://cdn-avatars.huggingface.co/v1/production/uploads/6704ee27386892c420db1938/lb5mtEwYhn47RawkynYPs.jpeg
JinCheng Ren
user
JinChengRen
68084b04ba1dd0e6a077e0a2
Jarvis Guo
68084b04ba1dd0e6a077e0a3
Yifan Yao
68084b04ba1dd0e6a077e0a4
Zhenlin Wei
68084b04ba1dd0e6a077e0a5
Zhenzhu Yang
68084b04ba1dd0e6a077e0a6
Zhongyuan Peng
admin_assigned
2025-04-23T13:48:40.455Z
63299f93688ad82b783aaf20
/avatars/7c11e60e551ef1c62aa2862529e357f5.svg
zhongyuan peng
user
happzy2633
68084b04ba1dd0e6a077e0a7
Boyu Feng
admin_assigned
2025-04-23T13:48:34.343Z
64337ff82bfb2b0ec75d5951
/avatars/61c7189e759e8cd8aa59701ac86249a7.svg
Boyu Feng
user
fbyfby576
68084b04ba1dd0e6a077e0a8
Jun Ma
68084b04ba1dd0e6a077e0a9
Xiao Gu
68084b04ba1dd0e6a077e0aa
Zhoufutu Wen
68084b04ba1dd0e6a077e0ab
King Zhu
68084b04ba1dd0e6a077e0ac
Yancheng He
68084b04ba1dd0e6a077e0ad
Meng Cao
68084b04ba1dd0e6a077e0ae
Shiwen Ni
admin_assigned
2025-04-23T13:48:11.824Z
641944acf9d6f1d772ede902
/avatars/a2f88017f7ab0cbea6862201e35fe747.svg
ShiwenNi
user
ShiwenNi
68084b04ba1dd0e6a077e0af
Jiaheng Liu
admin_assigned
2025-04-23T13:48:02.522Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
68084b04ba1dd0e6a077e0b0
Wenhao Huang
admin_assigned
2025-04-23T13:47:54.964Z
641e5bf65f274a0a92c2f6a2
/avatars/c15a54c51998c0e6367685e8e1737ec9.svg
Wenhao Huang
user
EZ-hwh
68084b04ba1dd0e6a077e0b1
Ge Zhang
admin_assigned
2025-04-23T13:47:48.795Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
68084b04ba1dd0e6a077e0b2
Xiaojie Jin
IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning
  in Multimodal LLMs
Existing evaluation frameworks for Multimodal Large Language Models (MLLMs)
primarily focus on image reasoning or general video understanding tasks,
largely overlooking the significant role of image context in video
comprehension. To bridge this gap, we propose IV-Bench, the first comprehensive
benchmark for evaluating Image-Grounded Video Perception and Reasoning.
IV-Bench consists of 967 videos paired with 2,585 meticulously annotated
image-text queries across 13 tasks (7 perception and 6 reasoning tasks) and 5
representative categories. Extensive evaluations of state-of-the-art
open-source (e.g., InternVL2.5, Qwen2.5-VL) and closed-source (e.g., GPT-4o,
Gemini2-Flash and Gemini2-Pro) MLLMs demonstrate that current models
substantially underperform in image-grounded video Perception and Reasoning,
merely achieving at most 28.9% accuracy. Further analysis reveals key factors
influencing model performance on IV-Bench, including inference pattern, frame
number, and resolution. Additionally, through a simple data synthesis approach,
we demonstratethe challenges of IV- Bench extend beyond merely aligning the
data format in the training proecss. These findings collectively provide
valuable insights for future research. Our codes and data are released in
https://github.com/multimodal-art-projection/IV-Bench.
IV-Bench evaluates Image-Grounded Video Perception and Reasoning in MLLMs, revealing substantial underperformance across multiple tasks and factors influencing model accuracy.
Multimodal Large Language Models MLLMs Image-Grounded Video Perception and Reasoning benchmark image-text queries inference pattern frame number resolution data synthesis