title: MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
    Efficient Evaluation
summary: Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
  and reasoning abilities, often assessed through multiple-choice questions
  (MCQs) that include an image, a question, and several options. However, many
  benchmarks used for such evaluations suffer from systematic biases. Remarkably,
  Large Language Models (LLMs) without any visual perception capabilities achieve
  non-trivial performance, undermining the credibility of these evaluations. To
  address this issue while maintaining the efficiency of MCQ evaluations, we
  propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
  trilogy evaluation pipeline and more rigorous metrics. For each original
  question from existing benchmarks, human annotators augment it by creating one
  perception question and one knowledge anchor question through a meticulous
  annotation process. MMEvalPro comprises 2,138 question triplets, totaling
  6,414 distinct questions. Two-thirds of these questions are manually labeled
  by human experts, while the rest are sourced from existing benchmarks (MMMU,
  ScienceQA, and MathVista). Compared with the existing benchmarks, our
  experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
  challenging (the best LMM lags behind human performance by 31.73%, compared
  to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
  best LLM trails the best LMM by 23.09%, whereas the gap for previous
  benchmarks is just 14.64%). Our in-depth analysis explains the reason for
  the large performance gap and justifies the trustworthiness of evaluation,
  underscoring its significant potential for advancing future research.
paper_authors._id: 668352541d6b7ca699ee18ff
paper_authors.name: Jinsheng Huang
paper_authors._id: 668352541d6b7ca699ee1900
paper_authors.name: Liang Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-02T01:06:02.744Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 668352541d6b7ca699ee1901
paper_authors.name: Taian Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-02T12:00:58.055Z
paper_authors._id: 628f8ec70ea7e76254bd3c7a
paper_authors.avatarUrl: /avatars/873c25eb9af052a436106a4d0618ff0b.svg
paper_authors.fullname: Taian Guo
paper_authors.type: user
paper_authors.user: ChazzyGordon
paper_authors._id: 668352541d6b7ca699ee1902
paper_authors.name: Fu Zeng
paper_authors._id: 668352541d6b7ca699ee1903
paper_authors.name: Yusheng Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:39:11.213Z
paper_authors._id: 668388cb549c1b932c9fe699
paper_authors.avatarUrl: /avatars/aa7523fbde4c2a8508cff13c74291e6a.svg
paper_authors.fullname: Yusheng Zhao
paper_authors.type: user
paper_authors.user: yszhao
paper_authors._id: 668352541d6b7ca699ee1904
paper_authors.name: Bohan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:51:05.892Z
paper_authors._id: 6688e31d24f96af5297860c0
paper_authors.avatarUrl: /avatars/21254cbba9dd3f99df98b4a3ffc394eb.svg
paper_authors.fullname: Bohan Wu
paper_authors.type: user
paper_authors.user: WXT3710
paper_authors._id: 668352541d6b7ca699ee1905
paper_authors.name: Ye Yuan
paper_authors._id: 668352541d6b7ca699ee1906
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:39.664Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 668352541d6b7ca699ee1907
paper_authors.name: Zhihui Guo
paper_authors._id: 668352541d6b7ca699ee1908
paper_authors.name: Yichi Zhang
paper_authors._id: 668352541d6b7ca699ee1909
paper_authors.name: Jingyang Yuan
paper_authors._id: 668352541d6b7ca699ee190a
paper_authors.name: Wei Ju
paper_authors._id: 668352541d6b7ca699ee190b
paper_authors.name: Luchen Liu
paper_authors._id: 668352541d6b7ca699ee190c
paper_authors.name: Tianyu Liu
paper_authors._id: 668352541d6b7ca699ee190d
paper_authors.name: Baobao Chang
paper_authors._id: 668352541d6b7ca699ee190e
paper_authors.name: Ming Zhang
paper_title: MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
    Efficient Evaluation
paper_summary: Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
  and reasoning abilities, often assessed through multiple-choice questions
  (MCQs) that include an image, a question, and several options. However, many
  benchmarks used for such evaluations suffer from systematic biases. Remarkably,
  Large Language Models (LLMs) without any visual perception capabilities achieve
  non-trivial performance, undermining the credibility of these evaluations. To
  address this issue while maintaining the efficiency of MCQ evaluations, we
  propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
  trilogy evaluation pipeline and more rigorous metrics. For each original
  question from existing benchmarks, human annotators augment it by creating one
  perception question and one knowledge anchor question through a meticulous
  annotation process. MMEvalPro comprises 2,138 question triplets, totaling
  6,414 distinct questions. Two-thirds of these questions are manually labeled
  by human experts, while the rest are sourced from existing benchmarks (MMMU,
  ScienceQA, and MathVista). Compared with the existing benchmarks, our
  experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
  challenging (the best LMM lags behind human performance by 31.73%, compared
  to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
  best LLM trails the best LMM by 23.09%, whereas the gap for previous
  benchmarks is just 14.64%). Our in-depth analysis explains the reason for
  the large performance gap and justifies the trustworthiness of evaluation,
  underscoring its significant potential for advancing future research.
paper_ai_summary: MMEvalPro is a benchmark designed to mitigate biases in multimodal evaluations through augmented questions and rigorous metrics, demonstrating greater challenge and trustworthiness for LMM and LLM performance assessments.
paper_ai_keywords: Large Multimodal Models LMMs multiple-choice questions MCQs Large Language Models LLMs perception question knowledge anchor question MMEvalPro Type-I errors triplets benchmark evaluation human performance performance gap trustworthiness