title: MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
    Benchmark
summary: This paper introduces MMMU-Pro, a robust version of the Massive
  Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
  MMMU-Pro rigorously assesses multimodal models' true understanding and
  reasoning capabilities through a three-step process based on MMMU: (1)
  filtering out questions answerable by text-only models, (2) augmenting
  candidate options, and (3) introducing a vision-only input setting where
  questions are embedded within images. This setting challenges AI to truly "see"
  and "read" simultaneously, testing a fundamental human cognitive skill of
  seamlessly integrating visual and textual information. Results show that model
  performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
  to 26.9% across models. We explore the impact of OCR prompts and Chain of
  Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
  generally improves performance. MMMU-Pro provides a more rigorous evaluation
  tool, closely mimicking real-world scenarios and offering valuable directions
  for future research in multimodal AI.
paper_authors._id: 66d93c930d5d2187a20c381d
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:45:25.393Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 66d93c930d5d2187a20c381e
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:23:55.953Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 66d93c930d5d2187a20c381f
paper_authors.name: Yuansheng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:44:59.359Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 66d93c930d5d2187a20c3820
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:41:28.084Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: /avatars/d9c5cf3491243d1f2b1c5df1873ee8e7.svg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 66d93c930d5d2187a20c3821
paper_authors.name: Kai Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T15:31:15.542Z
paper_authors._id: 63e0a50242591dda0b9dca5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: drogozhang
paper_authors._id: 66d93c930d5d2187a20c3822
paper_authors.name: Shengbang Tong
paper_authors._id: 66d93c930d5d2187a20c3823
paper_authors.name: Yuxuan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:46:44.822Z
paper_authors._id: 63745f7c0938c07542400172
paper_authors.avatarUrl: /avatars/df88a0b7f0693f25633240da56ce3aec.svg
paper_authors.fullname: yuxuan sun
paper_authors.type: user
paper_authors.user: jamessyx
paper_authors._id: 66d93c930d5d2187a20c3824
paper_authors.name: Ming Yin
paper_authors._id: 66d93c930d5d2187a20c3825
paper_authors.name: Botao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:47:50.432Z
paper_authors._id: 6528d0b07b99d91baa242ca8
paper_authors.avatarUrl: /avatars/56061bf1b3aca8fc49f73cbbbe38b19f.svg
paper_authors.fullname: Botao Yu
paper_authors.type: user
paper_authors.user: btyu
paper_authors._id: 66d93c930d5d2187a20c3826
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:47:43.316Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66d93c930d5d2187a20c3827
paper_authors.name: Huan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:47:18.892Z
paper_authors._id: 6127b327699a5d2cde0b44dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
paper_authors.fullname: Huan Sun
paper_authors.type: user
paper_authors.user: huansun
paper_authors._id: 66d93c930d5d2187a20c3828
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:21:55.013Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/grnmnxO1foz6aRRJ_A2NU.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 66d93c930d5d2187a20c3829
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:47:07.238Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 66d93c930d5d2187a20c382a
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:47:00.965Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
    Benchmark
paper_summary: This paper introduces MMMU-Pro, a robust version of the Massive
  Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
  MMMU-Pro rigorously assesses multimodal models' true understanding and
  reasoning capabilities through a three-step process based on MMMU: (1)
  filtering out questions answerable by text-only models, (2) augmenting
  candidate options, and (3) introducing a vision-only input setting where
  questions are embedded within images. This setting challenges AI to truly "see"
  and "read" simultaneously, testing a fundamental human cognitive skill of
  seamlessly integrating visual and textual information. Results show that model
  performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
  to 26.9% across models. We explore the impact of OCR prompts and Chain of
  Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
  generally improves performance. MMMU-Pro provides a more rigorous evaluation
  tool, closely mimicking real-world scenarios and offering valuable directions
  for future research in multimodal AI.
paper_ai_keywords: multimodal models MMMU benchmark text-only models candidate options vision-only input setting seamless integrating visual and textual information OCR prompts Chain of Thought (CoT) reasoning