title: MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks
summary: Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled
  unified processing of language, vision, and structured inputs, opening the door
  to complex tasks such as logical deduction, spatial reasoning, and scientific
  analysis. Despite their promise, the reasoning capabilities of MLLMs,
  particularly those augmented with intermediate thinking traces (MLLMs-T),
  remain poorly understood and lack standardized evaluation benchmarks. Existing
  work focuses primarily on perception or final answer correctness, offering
  limited insight into how models reason or fail across modalities. To address
  this gap, we introduce the MMMR, a new benchmark designed to rigorously
  evaluate multi-modal reasoning with explicit thinking. The MMMR comprises 1) a
  high-difficulty dataset of 1,083 questions spanning six diverse reasoning types
  with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace
  Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy
  through metrics like relevance, consistency, and structured error annotations.
  Empirical results show that MLLMs-T overall outperform non-thinking
  counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro
  suffer from reasoning pathologies such as inconsistency and overthinking. This
  benchmark reveals persistent gaps between accuracy and reasoning quality and
  provides an actionable evaluation pipeline for future model development.
  Overall, the MMMR offers a scalable foundation for evaluating, comparing, and
  improving the next generation of multi-modal reasoning systems.
paper_authors._id: 68355f94c682e155a8c766d4
paper_authors.name: Guiyao Tie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:44:00.966Z
paper_authors._id: 66e3f5b5df718255ccb5385e
paper_authors.avatarUrl: /avatars/cd29f0eeb0f97e2facb1a0373478c452.svg
paper_authors.fullname: 2024
paper_authors.type: user
paper_authors.user: tgy2024
paper_authors._id: 68355f94c682e155a8c766d5
paper_authors.name: Xueyang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:26.752Z
paper_authors._id: 657157dc971de7383e01ebc9
paper_authors.avatarUrl: /avatars/70a58d41bd4f86191205e916e4f6373e.svg
paper_authors.fullname: Zhou Xueyang
paper_authors.type: user
paper_authors.user: zhouxueyang
paper_authors._id: 68355f94c682e155a8c766d6
paper_authors.name: Tianhe Gu
paper_authors._id: 68355f94c682e155a8c766d7
paper_authors.name: Ruihang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T12:48:10.929Z
paper_authors._id: 67188e8f11e250d58d6a5305
paper_authors.avatarUrl: /avatars/f9d7e68d1204f2f3f577b4651ebc0da3.svg
paper_authors.fullname: RheaZhang
paper_authors.type: user
paper_authors.user: RHEAZ
paper_authors._id: 68355f94c682e155a8c766d8
paper_authors.name: Chaoran Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:19:07.618Z
paper_authors._id: 67a96673b4fa67845b049fb4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OeQASAuGeJRO5Ooicjs_T.png
paper_authors.fullname: Chaoran Hu
paper_authors.type: user
paper_authors.user: Cpmores
paper_authors._id: 68355f94c682e155a8c766d9
paper_authors.name: Sizhe Zhang
paper_authors._id: 68355f94c682e155a8c766da
paper_authors.name: Mengqu Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:19:21.976Z
paper_authors._id: 67af942763fee231f47d590b
paper_authors.avatarUrl: /avatars/f6cecbf6ac30d791f91f9aedc29d1b20.svg
paper_authors.fullname: Mengqu Sun
paper_authors.type: user
paper_authors.user: MengquSun
paper_authors._id: 68355f94c682e155a8c766db
paper_authors.name: Yan Zhang
paper_authors._id: 68355f94c682e155a8c766dc
paper_authors.name: Pan Zhou
paper_authors._id: 68355f94c682e155a8c766dd
paper_authors.name: Lichao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:19:29.418Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_title: MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks
paper_summary: Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled
  unified processing of language, vision, and structured inputs, opening the door
  to complex tasks such as logical deduction, spatial reasoning, and scientific
  analysis. Despite their promise, the reasoning capabilities of MLLMs,
  particularly those augmented with intermediate thinking traces (MLLMs-T),
  remain poorly understood and lack standardized evaluation benchmarks. Existing
  work focuses primarily on perception or final answer correctness, offering
  limited insight into how models reason or fail across modalities. To address
  this gap, we introduce the MMMR, a new benchmark designed to rigorously
  evaluate multi-modal reasoning with explicit thinking. The MMMR comprises 1) a
  high-difficulty dataset of 1,083 questions spanning six diverse reasoning types
  with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace
  Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy
  through metrics like relevance, consistency, and structured error annotations.
  Empirical results show that MLLMs-T overall outperform non-thinking
  counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro
  suffer from reasoning pathologies such as inconsistency and overthinking. This
  benchmark reveals persistent gaps between accuracy and reasoning quality and
  provides an actionable evaluation pipeline for future model development.
  Overall, the MMMR offers a scalable foundation for evaluating, comparing, and
  improving the next generation of multi-modal reasoning systems.
paper_ai_summary: The MMMR benchmark evaluates multi-modal reasoning in MLLMs by assessing thinking quality through diverse reasoning types and a modular evaluation pipeline.
paper_ai_keywords: Multi-Modal Large Language Models MLLMs reasoning traces MLLMs-T MMMR benchmark high-difficulty dataset six diverse reasoning types Reasoning Trace Evaluation Pipeline RTEP relevance consistency structured error annotations Claude-3.7-Sonnet Gemini-2.5 Pro