MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
Logical reasoning is a fundamental aspect of human intelligence and an
essential capability for multimodal large language models (MLLMs). Despite the
significant advancement in multimodal reasoning, existing benchmarks fail to
comprehensively evaluate their reasoning abilities due to the lack of explicit
categorization for logical reasoning types and an unclear understanding of
reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive
benchmark designed to evaluate the reasoning ability of MLLMs, which covers all
three types of reasoning (i.e., inductive, deductive, and abductive) in its
questions. We carefully curate the data to ensure that each question
effectively evaluates reasoning ability rather than perceptual skills or
knowledge breadth, and extend the evaluation protocols to cover the evaluation
of diverse questions. Our evaluation reveals substantial limitations of
state-of-the-art MLLMs when subjected to holistic assessments of logical
reasoning capabilities. Even the most advanced MLLMs show limited performance
in comprehensive logical reasoning, with notable performance imbalances across
reasoning types. In addition, we conducted an in-depth analysis of approaches
such as ``thinking mode'' and Rule-based RL, which are commonly believed to
enhance reasoning abilities. These findings highlight the critical limitations
and performance imbalances of current MLLMs in diverse logical reasoning
scenarios, providing comprehensive and systematic insights into the
understanding and evaluation of reasoning capabilities.
6836799db9b35de1c4a90d73
Jiakang Yuan
admin_assigned
2025-05-28T12:10:39.171Z
64a3d1ddb3239f3e3892b24b
/avatars/7ce585f5fc1d077fb1d70cc18c4da2c1.svg
Jiakang Yuan
user
JiakangYuan
6836799db9b35de1c4a90d74
Tianshuo Peng
admin_assigned
2025-05-28T12:10:47.745Z
6538dd471ad9b3ba7c2df861
https://cdn-avatars.huggingface.co/v1/production/uploads/6538dd471ad9b3ba7c2df861/MbEa7KHAK6u7PRb7WiPUC.jpeg
Tianshuo Peng
user
Potentialts
6836799db9b35de1c4a90d75
Yilei Jiang
admin_assigned
2025-05-28T12:11:01.744Z
66838e043f885cb431c48fd9
/avatars/b14e238dabb99a692ce2189f1c7ec9c7.svg
JiangYilei
user
Yilei-Jiang
6836799db9b35de1c4a90d76
Yiting Lu
6836799db9b35de1c4a90d77
Renrui Zhang
6836799db9b35de1c4a90d78
Kaituo Feng
admin_assigned
2025-05-28T12:12:45.346Z
67079840a9bcb7459b8d2a46
/avatars/32466863c5554f20cb2775b138832ac3.svg
Kaituo Feng
user
KaituoFeng
6836799db9b35de1c4a90d79
Chaoyou Fu
6836799db9b35de1c4a90d7a
Tao Chen
6836799db9b35de1c4a90d7b
Lei Bai
6836799db9b35de1c4a90d7c
Bo Zhang
admin_assigned
2025-05-28T12:13:12.150Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
6836799db9b35de1c4a90d7d
Xiangyu Yue
admin_assigned
2025-05-28T12:12:59.396Z
666a8f24e2990b0cb16b7bf9
/avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
Xiangyu Yue
user
xyyue
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
Logical reasoning is a fundamental aspect of human intelligence and an
essential capability for multimodal large language models (MLLMs). Despite the
significant advancement in multimodal reasoning, existing benchmarks fail to
comprehensively evaluate their reasoning abilities due to the lack of explicit
categorization for logical reasoning types and an unclear understanding of
reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive
benchmark designed to evaluate the reasoning ability of MLLMs, which covers all
three types of reasoning (i.e., inductive, deductive, and abductive) in its
questions. We carefully curate the data to ensure that each question
effectively evaluates reasoning ability rather than perceptual skills or
knowledge breadth, and extend the evaluation protocols to cover the evaluation
of diverse questions. Our evaluation reveals substantial limitations of
state-of-the-art MLLMs when subjected to holistic assessments of logical
reasoning capabilities. Even the most advanced MLLMs show limited performance
in comprehensive logical reasoning, with notable performance imbalances across
reasoning types. In addition, we conducted an in-depth analysis of approaches
such as ``thinking mode'' and Rule-based RL, which are commonly believed to
enhance reasoning abilities. These findings highlight the critical limitations
and performance imbalances of current MLLMs in diverse logical reasoning
scenarios, providing comprehensive and systematic insights into the
understanding and evaluation of reasoning capabilities.
MME-Reasoning evaluates the logical reasoning capabilities of multimodal large language models, revealing significant limitations and performance imbalances across inductive, deductive, and abductive reasoning types.
multimodal large language models MME-Reasoning logical reasoning inductive reasoning deductive reasoning abductive reasoning reasoning ability thinking mode Rule-based RL