title: MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for
    Reasoning Quality, Robustness, and Efficiency
summary: Answering questions with Chain-of-Thought (CoT) has significantly enhanced
  the reasoning capabilities of Large Language Models (LLMs), yet its impact on
  Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth
  investigation. In this paper, we introduce MME-CoT, a specialized benchmark
  evaluating the CoT reasoning performance of LMMs, spanning six domains: math,
  science, OCR, logic, space-time, and general scenes. As the first comprehensive
  study in this area, we propose a thorough evaluation suite incorporating three
  novel metrics that assess the reasoning quality, robustness, and efficiency at
  a fine-grained level. Leveraging curated high-quality data and a unique
  evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs,
  uncovering several key insights: 1) Models with reflection mechanism
  demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and
  demonstrating the highest quality results; 2) CoT prompting often degrades LMM
  performance on perception-heavy tasks, suggesting a potentially harmful
  overthinking behavior; and 3) Although the CoT quality is high, LMMs with
  reflection exhibit significant inefficiency in both normal response and
  self-correction phases. We hope MME-CoT serves as a foundation for advancing
  multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
paper_authors._id: 67aee0229e69670f49533146
paper_authors.name: Dongzhi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:05.736Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 67aee0229e69670f49533147
paper_authors.name: Renrui Zhang
paper_authors._id: 67aee0229e69670f49533148
paper_authors.name: Ziyu Guo
paper_authors._id: 67aee0229e69670f49533149
paper_authors.name: Yanwei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:50:20.626Z
paper_authors._id: 643ff78cdc984afcbbbc3b1a
paper_authors.avatarUrl: /avatars/eec5198ce88aaf8156840bec0d190a7f.svg
paper_authors.fullname: Yanwei Li
paper_authors.type: user
paper_authors.user: YanweiLi
paper_authors._id: 67aee0229e69670f4953314a
paper_authors.name: Yu Qi
paper_authors._id: 67aee0229e69670f4953314b
paper_authors.name: Xinyan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:50:07.239Z
paper_authors._id: 647c7a4ed412b3b376572a00
paper_authors.avatarUrl: /avatars/9cc310fd3f9e3f211475816ed9b0cdaa.svg
paper_authors.fullname: Xinyan Chen
paper_authors.type: user
paper_authors.user: xy06
paper_authors._id: 67aee0229e69670f4953314c
paper_authors.name: Liuhui Wang
paper_authors._id: 67aee0229e69670f4953314d
paper_authors.name: Jianhan Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:49:46.156Z
paper_authors._id: 67aeecf31f297f2bdabd91bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mZBgh3oudPtx_rgSt8PJ7.png
paper_authors.fullname: Jianhan Jin
paper_authors.type: user
paper_authors.user: Pala718
paper_authors._id: 67aee0229e69670f4953314e
paper_authors.name: Claire Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:49:39.014Z
paper_authors._id: 64862c6e19b74d6d646baa88
paper_authors.avatarUrl: /avatars/5ca91ff2b7dddb36378edf74067cd9e2.svg
paper_authors.fullname: Claire Guo
paper_authors.type: user
paper_authors.user: clairerg
paper_authors._id: 67aee0229e69670f4953314f
paper_authors.name: Shen Yan
paper_authors._id: 67aee0229e69670f49533150
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:53:22.348Z
paper_authors._id: 643dfd235aafbdca3a5792c0
paper_authors.avatarUrl: /avatars/ce8553cf5936012c692e08054ee27937.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: BoZhang
paper_authors._id: 67aee0229e69670f49533151
paper_authors.name: Chaoyou Fu
paper_authors._id: 67aee0229e69670f49533152
paper_authors.name: Peng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:49:22.182Z
paper_authors._id: 6759af3eccbc8817f9169179
paper_authors.avatarUrl: /avatars/49e64c7ccf71b8f25c52783b6ae93620.svg
paper_authors.fullname: Peng Gao
paper_authors.type: user
paper_authors.user: gaopenghigh
paper_authors._id: 67aee0229e69670f49533153
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:49:00.794Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for
    Reasoning Quality, Robustness, and Efficiency
paper_summary: Answering questions with Chain-of-Thought (CoT) has significantly enhanced
  the reasoning capabilities of Large Language Models (LLMs), yet its impact on
  Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth
  investigation. In this paper, we introduce MME-CoT, a specialized benchmark
  evaluating the CoT reasoning performance of LMMs, spanning six domains: math,
  science, OCR, logic, space-time, and general scenes. As the first comprehensive
  study in this area, we propose a thorough evaluation suite incorporating three
  novel metrics that assess the reasoning quality, robustness, and efficiency at
  a fine-grained level. Leveraging curated high-quality data and a unique
  evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs,
  uncovering several key insights: 1) Models with reflection mechanism
  demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and
  demonstrating the highest quality results; 2) CoT prompting often degrades LMM
  performance on perception-heavy tasks, suggesting a potentially harmful
  overthinking behavior; and 3) Although the CoT quality is high, LMMs with
  reflection exhibit significant inefficiency in both normal response and
  self-correction phases. We hope MME-CoT serves as a foundation for advancing
  multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
paper_ai_summary: MME-CoT provides a comprehensive evaluation of Chain-of-Thought reasoning in Large Multimodal Models across multiple domains, revealing insights into model performance, efficiency, and potential pitfalls.
paper_ai_keywords: Chain-of-Thought Large Language Models Large Multimodal Models MME-CoT benchmark evaluation suite reasoning quality robustness efficiency perception-heavy tasks reflection mechanism self-correction