title: MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video
    Understanding
summary: The advent of large vision-language models (LVLMs) has spurred research into
  their applications in multi-modal contexts, particularly in video
  understanding. Traditional VideoQA benchmarks, despite providing quantitative
  metrics, often fail to encompass the full spectrum of video content and
  inadequately assess models' temporal comprehension. To address these
  limitations, we introduce MMBench-Video, a quantitative benchmark designed to
  rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video
  incorporates lengthy videos from YouTube and employs free-form questions,
  mirroring practical use cases. The benchmark is meticulously crafted to probe
  the models' temporal reasoning skills, with all questions human-annotated
  according to a carefully constructed ability taxonomy. We employ GPT-4 for
  automated assessment, demonstrating superior accuracy and robustness over
  earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted
  comprehensive evaluations that include both proprietary and open-source LVLMs
  for images and videos. MMBench-Video stands as a valuable resource for the
  research community, facilitating improved evaluation of LVLMs and catalyzing
  progress in the field of video understanding. The evalutation code of
  MMBench-Video will be integrated into VLMEvalKit:
  https://github.com/open-compass/VLMEvalKit.
paper_authors._id: 667514ebdac349236b1d31d4
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:00:13.950Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 667514ebdac349236b1d31d5
paper_authors.name: Kangrui Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:55:57.855Z
paper_authors._id: 64ba19a63269cb9386db003d
paper_authors.avatarUrl: /avatars/86001ca10b50b0a1f8794521509348f7.svg
paper_authors.fullname: Kangrui Mao
paper_authors.type: user
paper_authors.user: karrymao
paper_authors._id: 667514ebdac349236b1d31d6
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:56:04.905Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 667514ebdac349236b1d31d7
paper_authors.name: Xiangyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:00:16.135Z
paper_authors._id: 6530e62f536dbca918e71c3e
paper_authors.avatarUrl: /avatars/efc93bc767e561c6c6d429f65c23382d.svg
paper_authors.fullname: Xiangyu Z
paper_authors.type: user
paper_authors.user: PhoenixZ
paper_authors._id: 667514ebdac349236b1d31d8
paper_authors.name: Yining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:56:24.200Z
paper_authors._id: 6570542c91da380dfbdd2d36
paper_authors.avatarUrl: /avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ANTiSOC1AL
paper_authors._id: 667514ebdac349236b1d31d9
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:56:30.981Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 667514ebdac349236b1d31da
paper_authors.name: Kai Chen
paper_title: MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video
    Understanding
paper_summary: The advent of large vision-language models (LVLMs) has spurred research into
  their applications in multi-modal contexts, particularly in video
  understanding. Traditional VideoQA benchmarks, despite providing quantitative
  metrics, often fail to encompass the full spectrum of video content and
  inadequately assess models' temporal comprehension. To address these
  limitations, we introduce MMBench-Video, a quantitative benchmark designed to
  rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video
  incorporates lengthy videos from YouTube and employs free-form questions,
  mirroring practical use cases. The benchmark is meticulously crafted to probe
  the models' temporal reasoning skills, with all questions human-annotated
  according to a carefully constructed ability taxonomy. We employ GPT-4 for
  automated assessment, demonstrating superior accuracy and robustness over
  earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted
  comprehensive evaluations that include both proprietary and open-source LVLMs
  for images and videos. MMBench-Video stands as a valuable resource for the
  research community, facilitating improved evaluation of LVLMs and catalyzing
  progress in the field of video understanding. The evalutation code of
  MMBench-Video will be integrated into VLMEvalKit:
  https://github.com/open-compass/VLMEvalKit.
paper_ai_keywords: large vision-language models (LVLMs) multi-modal contexts video understanding VideoQA benchmarks temporal comprehension MMBench-Video lengthy videos free-form questions temporal reasoning skills ability taxonomy GPT-4 LLM-based evaluations VLMEvalKit