MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
As a prominent direction of Artificial General Intelligence (AGI), Multimodal
Large Language Models (MLLMs) have garnered increased attention from both
industry and academia. Building upon pre-trained LLMs, this family of models
further develops multimodal perception and reasoning capabilities that are
impressive, such as writing code given a flow chart or creating stories based
on an image. In the development process, evaluation is critical since it
provides intuitive feedback and guidance on improving models. Distinct from the
traditional train-eval-test paradigm that only favors a single task like image
classification, the versatility of MLLMs has spurred the rise of various new
benchmarks and evaluation methods. In this paper, we aim to present a
comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
summarised benchmarks types divided by the evaluation capabilities, including
foundation capabilities, model self-analysis, and extented applications; 2) the
typical process of benchmark counstruction, consisting of data collection,
annotation, and precautions; 3) the systematic evaluation manner composed of
judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
according to different needs and to inspire better evaluation methods, thereby
driving the progress of MLLM research.
67456b5485003a3689267f80
Chaoyou Fu
67456b5485003a3689267f81
Yi-Fan Zhang
extracted_confirmed
2025-03-20T03:09:47.616Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
67456b5485003a3689267f82
Shukang Yin
67456b5485003a3689267f83
Bo Li
67456b5485003a3689267f84
Xinyu Fang
claimed_verified
2024-11-29T10:08:17.693Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
user
nebulae09
67456b5485003a3689267f85
Sirui Zhao
67456b5485003a3689267f86
Haodong Duan
claimed_verified
2024-11-27T22:20:48.871Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67456b5485003a3689267f87
Xing Sun
67456b5485003a3689267f88
Ziwei Liu
67456b5485003a3689267f89
Liang Wang
67456b5485003a3689267f8a
Caifeng Shan
67456b5485003a3689267f8b
Ran He
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
As a prominent direction of Artificial General Intelligence (AGI), Multimodal
Large Language Models (MLLMs) have garnered increased attention from both
industry and academia. Building upon pre-trained LLMs, this family of models
further develops multimodal perception and reasoning capabilities that are
impressive, such as writing code given a flow chart or creating stories based
on an image. In the development process, evaluation is critical since it
provides intuitive feedback and guidance on improving models. Distinct from the
traditional train-eval-test paradigm that only favors a single task like image
classification, the versatility of MLLMs has spurred the rise of various new
benchmarks and evaluation methods. In this paper, we aim to present a
comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
summarised benchmarks types divided by the evaluation capabilities, including
foundation capabilities, model self-analysis, and extented applications; 2) the
typical process of benchmark counstruction, consisting of data collection,
annotation, and precautions; 3) the systematic evaluation manner composed of
judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
according to different needs and to inspire better evaluation methods, thereby
driving the progress of MLLM research.
A comprehensive survey of evaluation methods and benchmarks for Multimodal Large Language Models (MLLMs) to improve model performance and drive research progress.
multimodal large language models evaluation benchmarks evaluation capabilities model self-analysis extended applications data collection annotation evaluation metrics evaluation toolkits