Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for
  Multimodal Reasoning Models
Existing Multimodal Large Language Models (MLLMs) are predominantly trained
and tested on consistent visual-textual inputs, leaving open the question of
whether they can handle inconsistencies in real-world, layout-rich content. To
bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR)
benchmark to assess MLLMs' ability to detect and reason about semantic
mismatches in artifacts such as webpages, presentation slides, and posters.
MMIR comprises 534 challenging samples, each containing synthetically injected
errors across five reasoning-heavy categories: Factual Contradiction, Identity
Misattribution, Contextual Mismatch, Quantitative Discrepancy, and
Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing
that models with dedicated multimodal reasoning capabilities, such as o1,
substantially outperform their counterparts while open-source models remain
particularly vulnerable to inconsistency errors. Detailed error analyses
further show that models excel in detecting inconsistencies confined to a
single modality, particularly in text, but struggle with cross-modal conflicts
and complex layouts. Probing experiments reveal that single-modality prompting,
including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal
gains, revealing a key bottleneck in cross-modal reasoning. Our findings
highlight the need for advanced multimodal reasoning and point to future
research on multimodal inconsistency.
67bd31d0d055a27740b16a30
Qianqi Yan
67bd31d0d055a27740b16a31
Yue Fan
67bd31d0d055a27740b16a32
Hongquan Li
67bd31d0d055a27740b16a33
Shan Jiang
67bd31d0d055a27740b16a34
Yang Zhao
67bd31d0d055a27740b16a35
Xinze Guan
67bd31d0d055a27740b16a36
Ching-Chen Kuo
67bd31d0d055a27740b16a37
Xin Eric Wang
admin_assigned
2025-02-25T16:24:13.717Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for
  Multimodal Reasoning Models
Existing Multimodal Large Language Models (MLLMs) are predominantly trained
and tested on consistent visual-textual inputs, leaving open the question of
whether they can handle inconsistencies in real-world, layout-rich content. To
bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR)
benchmark to assess MLLMs' ability to detect and reason about semantic
mismatches in artifacts such as webpages, presentation slides, and posters.
MMIR comprises 534 challenging samples, each containing synthetically injected
errors across five reasoning-heavy categories: Factual Contradiction, Identity
Misattribution, Contextual Mismatch, Quantitative Discrepancy, and
Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing
that models with dedicated multimodal reasoning capabilities, such as o1,
substantially outperform their counterparts while open-source models remain
particularly vulnerable to inconsistency errors. Detailed error analyses
further show that models excel in detecting inconsistencies confined to a
single modality, particularly in text, but struggle with cross-modal conflicts
and complex layouts. Probing experiments reveal that single-modality prompting,
including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal
gains, revealing a key bottleneck in cross-modal reasoning. Our findings
highlight the need for advanced multimodal reasoning and point to future
research on multimodal inconsistency.
A benchmark assesses state-of-the-art multimodal large language models' ability to detect and reason about semantic inconsistencies in real-world content, revealing gaps in cross-modal reasoning.
Multimodal Large Language Models MMIR benchmark Factual Contradiction Identity Misattribution Contextual Mismatch Quantitative Discrepancy Temporal/Spatial Incoherence o1 single-modality prompting Chain-of-Thought Set-of-Mark multimodal reasoning