title: Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for
    Multimodal Reasoning Models
summary: Existing Multimodal Large Language Models (MLLMs) are predominantly trained
  and tested on consistent visual-textual inputs, leaving open the question of
  whether they can handle inconsistencies in real-world, layout-rich content. To
  bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR)
  benchmark to assess MLLMs' ability to detect and reason about semantic
  mismatches in artifacts such as webpages, presentation slides, and posters.
  MMIR comprises 534 challenging samples, each containing synthetically injected
  errors across five reasoning-heavy categories: Factual Contradiction, Identity
  Misattribution, Contextual Mismatch, Quantitative Discrepancy, and
  Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing
  that models with dedicated multimodal reasoning capabilities, such as o1,
  substantially outperform their counterparts while open-source models remain
  particularly vulnerable to inconsistency errors. Detailed error analyses
  further show that models excel in detecting inconsistencies confined to a
  single modality, particularly in text, but struggle with cross-modal conflicts
  and complex layouts. Probing experiments reveal that single-modality prompting,
  including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal
  gains, revealing a key bottleneck in cross-modal reasoning. Our findings
  highlight the need for advanced multimodal reasoning and point to future
  research on multimodal inconsistency.
paper_authors._id: 67bd31d0d055a27740b16a30
paper_authors.name: Qianqi Yan
paper_authors._id: 67bd31d0d055a27740b16a31
paper_authors.name: Yue Fan
paper_authors._id: 67bd31d0d055a27740b16a32
paper_authors.name: Hongquan Li
paper_authors._id: 67bd31d0d055a27740b16a33
paper_authors.name: Shan Jiang
paper_authors._id: 67bd31d0d055a27740b16a34
paper_authors.name: Yang Zhao
paper_authors._id: 67bd31d0d055a27740b16a35
paper_authors.name: Xinze Guan
paper_authors._id: 67bd31d0d055a27740b16a36
paper_authors.name: Ching-Chen Kuo
paper_authors._id: 67bd31d0d055a27740b16a37
paper_authors.name: Xin Eric Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:24:13.717Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for
    Multimodal Reasoning Models
paper_summary: Existing Multimodal Large Language Models (MLLMs) are predominantly trained
  and tested on consistent visual-textual inputs, leaving open the question of
  whether they can handle inconsistencies in real-world, layout-rich content. To
  bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR)
  benchmark to assess MLLMs' ability to detect and reason about semantic
  mismatches in artifacts such as webpages, presentation slides, and posters.
  MMIR comprises 534 challenging samples, each containing synthetically injected
  errors across five reasoning-heavy categories: Factual Contradiction, Identity
  Misattribution, Contextual Mismatch, Quantitative Discrepancy, and
  Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing
  that models with dedicated multimodal reasoning capabilities, such as o1,
  substantially outperform their counterparts while open-source models remain
  particularly vulnerable to inconsistency errors. Detailed error analyses
  further show that models excel in detecting inconsistencies confined to a
  single modality, particularly in text, but struggle with cross-modal conflicts
  and complex layouts. Probing experiments reveal that single-modality prompting,
  including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal
  gains, revealing a key bottleneck in cross-modal reasoning. Our findings
  highlight the need for advanced multimodal reasoning and point to future
  research on multimodal inconsistency.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Multimodal Inconsistency Reasoning (MMIR) Multimodal reasoning capabilities Chain-of-Thought (CoT) Set-of-Mark (SoM) Cross-modal conflicts Cross-modal reasoning