MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video
  Understanding
The advent of large vision-language models (LVLMs) has spurred research into
their applications in multi-modal contexts, particularly in video
understanding. Traditional VideoQA benchmarks, despite providing quantitative
metrics, often fail to encompass the full spectrum of video content and
inadequately assess models' temporal comprehension. To address these
limitations, we introduce MMBench-Video, a quantitative benchmark designed to
rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video
incorporates lengthy videos from YouTube and employs free-form questions,
mirroring practical use cases. The benchmark is meticulously crafted to probe
the models' temporal reasoning skills, with all questions human-annotated
according to a carefully constructed ability taxonomy. We employ GPT-4 for
automated assessment, demonstrating superior accuracy and robustness over
earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted
comprehensive evaluations that include both proprietary and open-source LVLMs
for images and videos. MMBench-Video stands as a valuable resource for the
research community, facilitating improved evaluation of LVLMs and catalyzing
progress in the field of video understanding. The evalutation code of
MMBench-Video will be integrated into VLMEvalKit:
https://github.com/open-compass/VLMEvalKit.
667514ebdac349236b1d31d4
Xinyu Fang
claimed_verified
2024-06-21T07:00:13.950Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
user
nebulae09
667514ebdac349236b1d31d5
Kangrui Mao
admin_assigned
2024-06-21T07:55:57.855Z
64ba19a63269cb9386db003d
/avatars/86001ca10b50b0a1f8794521509348f7.svg
Kangrui Mao
user
karrymao
667514ebdac349236b1d31d6
Haodong Duan
admin_assigned
2024-06-21T07:56:04.905Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
667514ebdac349236b1d31d7
Xiangyu Zhao
claimed_verified
2024-06-21T07:00:16.135Z
6530e62f536dbca918e71c3e
/avatars/efc93bc767e561c6c6d429f65c23382d.svg
Xiangyu Z
user
PhoenixZ
667514ebdac349236b1d31d8
Yining Li
admin_assigned
2024-06-21T07:56:24.200Z
6570542c91da380dfbdd2d36
/avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
Yining Li
user
ANTiSOC1AL
667514ebdac349236b1d31d9
Dahua Lin
admin_assigned
2024-06-21T07:56:30.981Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
667514ebdac349236b1d31da
Kai Chen
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video
  Understanding
The advent of large vision-language models (LVLMs) has spurred research into
their applications in multi-modal contexts, particularly in video
understanding. Traditional VideoQA benchmarks, despite providing quantitative
metrics, often fail to encompass the full spectrum of video content and
inadequately assess models' temporal comprehension. To address these
limitations, we introduce MMBench-Video, a quantitative benchmark designed to
rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video
incorporates lengthy videos from YouTube and employs free-form questions,
mirroring practical use cases. The benchmark is meticulously crafted to probe
the models' temporal reasoning skills, with all questions human-annotated
according to a carefully constructed ability taxonomy. We employ GPT-4 for
automated assessment, demonstrating superior accuracy and robustness over
earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted
comprehensive evaluations that include both proprietary and open-source LVLMs
for images and videos. MMBench-Video stands as a valuable resource for the
research community, facilitating improved evaluation of LVLMs and catalyzing
progress in the field of video understanding. The evalutation code of
MMBench-Video will be integrated into VLMEvalKit:
https://github.com/open-compass/VLMEvalKit.
MMBench-Video assesses large vision-language models' video understanding capabilities through a quantitative benchmark using YouTube videos and human-annotated questions.
vision-language models multi-modal contexts video understanding VideoQA MMBench-Video temporal reasoning ability taxonomy GPT-4 VLMEvalKit