Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline
  Professional Videos
Humans acquire knowledge through three cognitive stages: perceiving
information, comprehending knowledge, and adapting knowledge to solve novel
problems. Videos serve as an effective medium for this learning process,
facilitating a progression through these cognitive stages. However, existing
video benchmarks fail to systematically evaluate the knowledge acquisition
capabilities in Large Multimodal Models (LMMs). To address this gap, we
introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to
assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU
features a curated collection of 300 expert-level videos and 900
human-annotated questions across six disciplines, evaluating knowledge
acquisition through stage-aligned question-answer pairs: Perception,
Comprehension, and Adaptation. A proposed knowledge gain metric,
{\Delta}knowledge, quantifies improvement in performance after video viewing.
Evaluation of LMMs reveals a steep decline in performance as cognitive demands
increase and highlights a significant gap between human and model knowledge
acquisition, underscoring the need for methods to enhance LMMs' capability to
learn and adapt from videos.
67934585e4e44e2866b644f2
Kairui Hu
claimed_verified
2025-01-24T09:07:46.937Z
6400ba2b261cfa61f3a00555
/avatars/1311e0b5e21b1c94d73fcaf455d3c7f7.svg
Kairui
user
KairuiHu
67934585e4e44e2866b644f3
Penghao Wu
admin_assigned
2025-01-24T10:21:29.750Z
64101f81b27543634e377fc1
/avatars/557dd9d4707e3b38e0805dfb87c08004.svg
Penghao Wu
user
craigwu
67934585e4e44e2866b644f4
Fanyi Pu
admin_assigned
2025-01-24T10:21:39.917Z
646e1ef5075bbcc48ddf21e8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
Pu Fanyi
user
pufanyi
67934585e4e44e2866b644f5
Wang Xiao
admin_assigned
2025-01-24T10:21:46.540Z
647efcc945baf21ad707e10c
/avatars/e2fab1c9031eb0eec9f015a8fc237f64.svg
Wang Xiao
user
wangxiao1208
67934585e4e44e2866b644f6
Yuanhan Zhang
admin_assigned
2025-01-24T10:26:42.528Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
67934585e4e44e2866b644f7
Xiang Yue
admin_assigned
2025-01-24T10:27:14.894Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
67934585e4e44e2866b644f8
Bo Li
67934585e4e44e2866b644f9
Ziwei Liu
admin_assigned
2025-01-24T10:27:30.310Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline
  Professional Videos
Humans acquire knowledge through three cognitive stages: perceiving
information, comprehending knowledge, and adapting knowledge to solve novel
problems. Videos serve as an effective medium for this learning process,
facilitating a progression through these cognitive stages. However, existing
video benchmarks fail to systematically evaluate the knowledge acquisition
capabilities in Large Multimodal Models (LMMs). To address this gap, we
introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to
assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU
features a curated collection of 300 expert-level videos and 900
human-annotated questions across six disciplines, evaluating knowledge
acquisition through stage-aligned question-answer pairs: Perception,
Comprehension, and Adaptation. A proposed knowledge gain metric,
{\Delta}knowledge, quantifies improvement in performance after video viewing.
Evaluation of LMMs reveals a steep decline in performance as cognitive demands
increase and highlights a significant gap between human and model knowledge
acquisition, underscoring the need for methods to enhance LMMs' capability to
learn and adapt from videos.
Video-MMMMU evaluates Large Multimodal Models' knowledge acquisition from videos through perception, comprehension, and adaptation stages, revealing performance gaps compared to humans.
Large Multimodal Models Video-MMMU perception comprehension adaptation knowledge gain metric {\Delta}knowledge