MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
<|reference_start|>MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark: This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.<|reference_end|>
@article{yue2024mmmu-pro:,
    title={MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
  Benchmark},
    author={Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang
  Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham
  Neubig},
    journal={arXiv preprint arXiv:2409.02813},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.02813},
    primaryClass={cs.CL cs.CV}
}