title: MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
summary: We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark
  for evaluating foundation models in video understanding. MMVU includes 3,000
  expert-annotated questions spanning 27 subjects across four core disciplines:
  Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to
  prior benchmarks, MMVU features three key advancements. First, it challenges
  models to apply domain-specific knowledge and perform expert-level reasoning to
  analyze specialized-domain videos, moving beyond the basic visual perception
  typically assessed in current video benchmarks. Second, each example is
  annotated by human experts from scratch. We implement strict data quality
  controls to ensure the high quality of the dataset. Finally, each example is
  enriched with expert-annotated reasoning rationals and relevant domain
  knowledge, facilitating in-depth analysis. We conduct an extensive evaluation
  of 32 frontier multimodal foundation models on MMVU. The latest
  System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest
  performance among the tested models. However, they still fall short of matching
  human expertise. Through in-depth error analyses and case studies, we offer
  actionable insights for future advancements in expert-level,
  knowledge-intensive video understanding for specialized domains.
paper_authors._id: 67906f432565fc5140d72dc3
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T20:49:53.483Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 67906f432565fc5140d72dc4
paper_authors.name: Lujing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:06:45.496Z
paper_authors._id: 64ffd83b96ec8a52185dfb54
paper_authors.avatarUrl: /avatars/a4fadc7e2f5c1125d5d455de4d5c9b8e.svg
paper_authors.fullname: Lujing Xie
paper_authors.type: user
paper_authors.user: leeroylucas
paper_authors._id: 67906f432565fc5140d72dc5
paper_authors.name: Haowei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T10:59:50.465Z
paper_authors._id: 637169557a5e5d8efdc3e58e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668515232215-637169557a5e5d8efdc3e58e.jpeg
paper_authors.fullname: Haowei Zhang
paper_authors.type: user
paper_authors.user: freesky
paper_authors._id: 67906f432565fc5140d72dc6
paper_authors.name: Guo Gan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:10:43.494Z
paper_authors._id: 65dfeee3d16fb170031df293
paper_authors.avatarUrl: /avatars/05e6fe0e61d4bb87536554c782385dac.svg
paper_authors.fullname: gan
paper_authors.type: user
paper_authors.user: guo9
paper_authors._id: 67906f432565fc5140d72dc7
paper_authors.name: Yitao Long
paper_authors._id: 67906f432565fc5140d72dc8
paper_authors.name: Zhiyuan Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:07:09.079Z
paper_authors._id: 65e9945f6bcbcae600b7e64f
paper_authors.avatarUrl: /avatars/8aa986a6c0e35c55d5d1461d1dc11ac3.svg
paper_authors.fullname: Zhiyuan Hu
paper_authors.type: user
paper_authors.user: zhiyhu
paper_authors._id: 67906f432565fc5140d72dc9
paper_authors.name: Tongyan Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T10:59:48.493Z
paper_authors._id: 66e83ec5deb449d8d856e78d
paper_authors.avatarUrl: /avatars/c5e56be65fcacb3192ce10ba6d8f48e2.svg
paper_authors.fullname: Tongyan Hu
paper_authors.type: user
paper_authors.user: entropyhu
paper_authors._id: 67906f432565fc5140d72dca
paper_authors.name: Weiyuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T14:04:09.444Z
paper_authors._id: 6652e84bff6ccc0ef5ac7055
paper_authors.avatarUrl: /avatars/70a8b6a0bae5f9e954fa15f56ab2ddc3.svg
paper_authors.fullname: Weiyuan Chen
paper_authors.type: user
paper_authors.user: RaidonShogun
paper_authors._id: 67906f432565fc5140d72dcb
paper_authors.name: Chuhan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:07:16.582Z
paper_authors._id: 65415f1d5168c4f3487a2103
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
paper_authors.fullname: Chuhan Li
paper_authors.type: user
paper_authors.user: ChuhanLi
paper_authors._id: 67906f432565fc5140d72dcc
paper_authors.name: Junyang Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:34:00.857Z
paper_authors._id: 67909add9551780939a950f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vS8OfOtpYr0nNXWixn1Er.png
paper_authors.fullname: Junyang Song
paper_authors.type: user
paper_authors.user: UndetectedAtom
paper_authors._id: 67906f432565fc5140d72dcd
paper_authors.name: Zhijian Xu
paper_authors._id: 67906f432565fc5140d72dce
paper_authors.name: Chengye Wang
paper_authors._id: 67906f432565fc5140d72dcf
paper_authors.name: Weifeng Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T10:08:18.785Z
paper_authors._id: 67907fdb6146e0f96241dcc3
paper_authors.avatarUrl: /avatars/ba16cb87dce1e3ccbbe8091a3fe553fc.svg
paper_authors.fullname: Wf Pan
paper_authors.type: user
paper_authors.user: Phil-01
paper_authors._id: 67906f432565fc5140d72dd0
paper_authors.name: Ziyao Shangguan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:06:39.157Z
paper_authors._id: 65dea56779f827c045b1df96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
paper_authors.fullname: Ziyao Shangguan
paper_authors.type: user
paper_authors.user: ziyaosg
paper_authors._id: 67906f432565fc5140d72dd1
paper_authors.name: Xiangru Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:07:56.034Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 67906f432565fc5140d72dd2
paper_authors.name: Zhenwen Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:08:01.981Z
paper_authors._id: 62ffa3f8311cad266f9af236
paper_authors.avatarUrl: /avatars/4c88cb518e000a475f8381573f21aa7f.svg
paper_authors.fullname: Zhenwen Liang
paper_authors.type: user
paper_authors.user: invokerliang
paper_authors._id: 67906f432565fc5140d72dd3
paper_authors.name: Yixin Liu
paper_authors._id: 67906f432565fc5140d72dd4
paper_authors.name: Chen Zhao
paper_authors._id: 67906f432565fc5140d72dd5
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T14:09:26.876Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_title: MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
paper_summary: We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark
  for evaluating foundation models in video understanding. MMVU includes 3,000
  expert-annotated questions spanning 27 subjects across four core disciplines:
  Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to
  prior benchmarks, MMVU features three key advancements. First, it challenges
  models to apply domain-specific knowledge and perform expert-level reasoning to
  analyze specialized-domain videos, moving beyond the basic visual perception
  typically assessed in current video benchmarks. Second, each example is
  annotated by human experts from scratch. We implement strict data quality
  controls to ensure the high quality of the dataset. Finally, each example is
  enriched with expert-annotated reasoning rationals and relevant domain
  knowledge, facilitating in-depth analysis. We conduct an extensive evaluation
  of 32 frontier multimodal foundation models on MMVU. The latest
  System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest
  performance among the tested models. However, they still fall short of matching
  human expertise. Through in-depth error analyses and case studies, we offer
  actionable insights for future advancements in expert-level,
  knowledge-intensive video understanding for specialized domains.
paper_ai_summary: A comprehensive benchmark for evaluating foundation models in video understanding includes domain-specific questions and expert evaluations, highlighting the gap between current models and human expertise.
paper_ai_keywords: multimodal foundation models expert-level reasoning domain-specific knowledge System-2-capable models MMVU video understanding