title: Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline
    Professional Videos
summary: Humans acquire knowledge through three cognitive stages: perceiving
  information, comprehending knowledge, and adapting knowledge to solve novel
  problems. Videos serve as an effective medium for this learning process,
  facilitating a progression through these cognitive stages. However, existing
  video benchmarks fail to systematically evaluate the knowledge acquisition
  capabilities in Large Multimodal Models (LMMs). To address this gap, we
  introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to
  assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU
  features a curated collection of 300 expert-level videos and 900
  human-annotated questions across six disciplines, evaluating knowledge
  acquisition through stage-aligned question-answer pairs: Perception,
  Comprehension, and Adaptation. A proposed knowledge gain metric,
  {\Delta}knowledge, quantifies improvement in performance after video viewing.
  Evaluation of LMMs reveals a steep decline in performance as cognitive demands
  increase and highlights a significant gap between human and model knowledge
  acquisition, underscoring the need for methods to enhance LMMs' capability to
  learn and adapt from videos.
paper_authors._id: 67934585e4e44e2866b644f2
paper_authors.name: Kairui Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:07:46.937Z
paper_authors._id: 6400ba2b261cfa61f3a00555
paper_authors.avatarUrl: /avatars/1311e0b5e21b1c94d73fcaf455d3c7f7.svg
paper_authors.fullname: Kairui
paper_authors.type: user
paper_authors.user: KairuiHu
paper_authors._id: 67934585e4e44e2866b644f3
paper_authors.name: Penghao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:21:29.750Z
paper_authors._id: 64101f81b27543634e377fc1
paper_authors.avatarUrl: /avatars/557dd9d4707e3b38e0805dfb87c08004.svg
paper_authors.fullname: Penghao Wu
paper_authors.type: user
paper_authors.user: craigwu
paper_authors._id: 67934585e4e44e2866b644f4
paper_authors.name: Fanyi Pu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:21:39.917Z
paper_authors._id: 646e1ef5075bbcc48ddf21e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
paper_authors.fullname: Pu Fanyi
paper_authors.type: user
paper_authors.user: pufanyi
paper_authors._id: 67934585e4e44e2866b644f5
paper_authors.name: Wang Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:21:46.540Z
paper_authors._id: 647efcc945baf21ad707e10c
paper_authors.avatarUrl: /avatars/e2fab1c9031eb0eec9f015a8fc237f64.svg
paper_authors.fullname: Wang Xiao
paper_authors.type: user
paper_authors.user: wangxiao1208
paper_authors._id: 67934585e4e44e2866b644f6
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:26:42.528Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 67934585e4e44e2866b644f7
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:27:14.894Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 67934585e4e44e2866b644f8
paper_authors.name: Bo Li
paper_authors._id: 67934585e4e44e2866b644f9
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:27:30.310Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline
    Professional Videos
paper_summary: Humans acquire knowledge through three cognitive stages: perceiving
  information, comprehending knowledge, and adapting knowledge to solve novel
  problems. Videos serve as an effective medium for this learning process,
  facilitating a progression through these cognitive stages. However, existing
  video benchmarks fail to systematically evaluate the knowledge acquisition
  capabilities in Large Multimodal Models (LMMs). To address this gap, we
  introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to
  assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU
  features a curated collection of 300 expert-level videos and 900
  human-annotated questions across six disciplines, evaluating knowledge
  acquisition through stage-aligned question-answer pairs: Perception,
  Comprehension, and Adaptation. A proposed knowledge gain metric,
  {\Delta}knowledge, quantifies improvement in performance after video viewing.
  Evaluation of LMMs reveals a steep decline in performance as cognitive demands
  increase and highlights a significant gap between human and model knowledge
  acquisition, underscoring the need for methods to enhance LMMs' capability to
  learn and adapt from videos.
paper_ai_summary: Video-MMMMU evaluates Large Multimodal Models' knowledge acquisition from videos through perception, comprehension, and adaptation stages, revealing performance gaps compared to humans.
paper_ai_keywords: Large Multimodal Models Video-MMMU perception comprehension adaptation knowledge gain metric {\Delta}knowledge