title: MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
summary: As a prominent direction of Artificial General Intelligence (AGI), Multimodal
  Large Language Models (MLLMs) have garnered increased attention from both
  industry and academia. Building upon pre-trained LLMs, this family of models
  further develops multimodal perception and reasoning capabilities that are
  impressive, such as writing code given a flow chart or creating stories based
  on an image. In the development process, evaluation is critical since it
  provides intuitive feedback and guidance on improving models. Distinct from the
  traditional train-eval-test paradigm that only favors a single task like image
  classification, the versatility of MLLMs has spurred the rise of various new
  benchmarks and evaluation methods. In this paper, we aim to present a
  comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
  summarised benchmarks types divided by the evaluation capabilities, including
  foundation capabilities, model self-analysis, and extented applications; 2) the
  typical process of benchmark counstruction, consisting of data collection,
  annotation, and precautions; 3) the systematic evaluation manner composed of
  judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
  aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
  according to different needs and to inspire better evaluation methods, thereby
  driving the progress of MLLM research.
paper_authors._id: 67456b5485003a3689267f80
paper_authors.name: Chaoyou Fu
paper_authors._id: 67456b5485003a3689267f81
paper_authors.name: Yi-Fan Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-20T03:09:47.616Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 67456b5485003a3689267f82
paper_authors.name: Shukang Yin
paper_authors._id: 67456b5485003a3689267f83
paper_authors.name: Bo Li
paper_authors._id: 67456b5485003a3689267f84
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T10:08:17.693Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 67456b5485003a3689267f85
paper_authors.name: Sirui Zhao
paper_authors._id: 67456b5485003a3689267f86
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:48.871Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67456b5485003a3689267f87
paper_authors.name: Xing Sun
paper_authors._id: 67456b5485003a3689267f88
paper_authors.name: Ziwei Liu
paper_authors._id: 67456b5485003a3689267f89
paper_authors.name: Liang Wang
paper_authors._id: 67456b5485003a3689267f8a
paper_authors.name: Caifeng Shan
paper_authors._id: 67456b5485003a3689267f8b
paper_authors.name: Ran He
paper_title: MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
paper_summary: As a prominent direction of Artificial General Intelligence (AGI), Multimodal
  Large Language Models (MLLMs) have garnered increased attention from both
  industry and academia. Building upon pre-trained LLMs, this family of models
  further develops multimodal perception and reasoning capabilities that are
  impressive, such as writing code given a flow chart or creating stories based
  on an image. In the development process, evaluation is critical since it
  provides intuitive feedback and guidance on improving models. Distinct from the
  traditional train-eval-test paradigm that only favors a single task like image
  classification, the versatility of MLLMs has spurred the rise of various new
  benchmarks and evaluation methods. In this paper, we aim to present a
  comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
  summarised benchmarks types divided by the evaluation capabilities, including
  foundation capabilities, model self-analysis, and extented applications; 2) the
  typical process of benchmark counstruction, consisting of data collection,
  annotation, and precautions; 3) the systematic evaluation manner composed of
  judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
  aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
  according to different needs and to inspire better evaluation methods, thereby
  driving the progress of MLLM research.
paper_ai_summary: A comprehensive survey of evaluation methods and benchmarks for Multimodal Large Language Models (MLLMs) to improve model performance and drive research progress.
paper_ai_keywords: multimodal large language models evaluation benchmarks evaluation capabilities model self-analysis extended applications data collection annotation evaluation metrics evaluation toolkits