MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks
Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled
unified processing of language, vision, and structured inputs, opening the door
to complex tasks such as logical deduction, spatial reasoning, and scientific
analysis. Despite their promise, the reasoning capabilities of MLLMs,
particularly those augmented with intermediate thinking traces (MLLMs-T),
remain poorly understood and lack standardized evaluation benchmarks. Existing
work focuses primarily on perception or final answer correctness, offering
limited insight into how models reason or fail across modalities. To address
this gap, we introduce the MMMR, a new benchmark designed to rigorously
evaluate multi-modal reasoning with explicit thinking. The MMMR comprises 1) a
high-difficulty dataset of 1,083 questions spanning six diverse reasoning types
with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace
Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy
through metrics like relevance, consistency, and structured error annotations.
Empirical results show that MLLMs-T overall outperform non-thinking
counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro
suffer from reasoning pathologies such as inconsistency and overthinking. This
benchmark reveals persistent gaps between accuracy and reasoning quality and
provides an actionable evaluation pipeline for future model development.
Overall, the MMMR offers a scalable foundation for evaluating, comparing, and
improving the next generation of multi-modal reasoning systems.
68355f94c682e155a8c766d4
Guiyao Tie
claimed_verified
2025-05-29T07:44:00.966Z
66e3f5b5df718255ccb5385e
/avatars/cd29f0eeb0f97e2facb1a0373478c452.svg
2024
user
tgy2024
68355f94c682e155a8c766d5
Xueyang Zhou
claimed_verified
2025-05-28T09:00:26.752Z
657157dc971de7383e01ebc9
/avatars/70a58d41bd4f86191205e916e4f6373e.svg
Zhou Xueyang
user
zhouxueyang
68355f94c682e155a8c766d6
Tianhe Gu
68355f94c682e155a8c766d7
Ruihang Zhang
claimed_verified
2025-05-28T12:48:10.929Z
67188e8f11e250d58d6a5305
/avatars/f9d7e68d1204f2f3f577b4651ebc0da3.svg
RheaZhang
user
RHEAZ
68355f94c682e155a8c766d8
Chaoran Hu
admin_assigned
2025-05-28T12:19:07.618Z
67a96673b4fa67845b049fb4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OeQASAuGeJRO5Ooicjs_T.png
Chaoran Hu
user
Cpmores
68355f94c682e155a8c766d9
Sizhe Zhang
68355f94c682e155a8c766da
Mengqu Sun
admin_assigned
2025-05-28T12:19:21.976Z
67af942763fee231f47d590b
/avatars/f6cecbf6ac30d791f91f9aedc29d1b20.svg
Mengqu Sun
user
MengquSun
68355f94c682e155a8c766db
Yan Zhang
68355f94c682e155a8c766dc
Pan Zhou
68355f94c682e155a8c766dd
Lichao Sun
admin_assigned
2025-05-28T12:19:29.418Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks
Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled
unified processing of language, vision, and structured inputs, opening the door
to complex tasks such as logical deduction, spatial reasoning, and scientific
analysis. Despite their promise, the reasoning capabilities of MLLMs,
particularly those augmented with intermediate thinking traces (MLLMs-T),
remain poorly understood and lack standardized evaluation benchmarks. Existing
work focuses primarily on perception or final answer correctness, offering
limited insight into how models reason or fail across modalities. To address
this gap, we introduce the MMMR, a new benchmark designed to rigorously
evaluate multi-modal reasoning with explicit thinking. The MMMR comprises 1) a
high-difficulty dataset of 1,083 questions spanning six diverse reasoning types
with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace
Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy
through metrics like relevance, consistency, and structured error annotations.
Empirical results show that MLLMs-T overall outperform non-thinking
counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro
suffer from reasoning pathologies such as inconsistency and overthinking. This
benchmark reveals persistent gaps between accuracy and reasoning quality and
provides an actionable evaluation pipeline for future model development.
Overall, the MMMR offers a scalable foundation for evaluating, comparing, and
improving the next generation of multi-modal reasoning systems.
The MMMR benchmark evaluates multi-modal reasoning in MLLMs by assessing thinking quality through diverse reasoning types and a modular evaluation pipeline.
Multi-Modal Large Language Models MLLMs reasoning traces MLLMs-T MMMR benchmark high-difficulty dataset six diverse reasoning types Reasoning Trace Evaluation Pipeline RTEP relevance consistency structured error annotations Claude-3.7-Sonnet Gemini-2.5 Pro