LIME: Less Is More for MLLM Evaluation
<|reference_start|>LIME: Less Is More for MLLM Evaluation: Multimodal Large Language Models (MLLMs) are measured on numerous benchmarks like image captioning, visual question answering, and reasoning. However, these benchmarks often include overly simple or uninformative samples, making it difficult to effectively distinguish the performance of different MLLMs. Additionally, evaluating models across many benchmarks creates a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated using a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. Our experiments show that LIME reduces the number of samples by 76% and evaluation time by 77%, while more effectively distinguishing between models. Notably, we find that traditional automatic metrics like CIDEr are insufficient for evaluating MLLMs' captioning performance, and excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME<|reference_end|>
@article{zhu2024lime:,
    title={LIME: Less Is More for MLLM Evaluation},
    author={King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li,
  Shawn Gavin, Tuney Zheng, Jiawei Guo, Bo Li, Haoning Wu, Xingwei Qu, Jian
  Yang, Zachary Liu, Xiang Yue, J.H. Liu, Chenghua Lin, Min Yang, Shiwen Ni,
  Wenhao Huang, Ge Zhang},
    journal={arXiv preprint arXiv:2409.06851},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.06851},
    primaryClass={cs.CV cs.AI}
}