TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing
  Prompts
Recently, multimodal large language models (MLLMs) have received much
attention for their impressive capabilities. The evaluation of MLLMs is
becoming critical to analyzing attributes of MLLMs and providing valuable
insights. However, current benchmarks overlook the problem of prompt
sensitivity - minor prompt variations may lead to significant performance
fluctuations. Thus, inappropriate prompts may obscure the models' capabilities,
underestimating the models' performance. Moreover, different models have
different preferences for different prompts, and thus, using the same prompt
for all models will cause evaluation bias. This paper analyzes this deficiency
in existing benchmarks and further introduces a new evaluation framework named
TP-Eval, which introduces a prompt customization method to reduce evaluation
biases and tap models' potential. TP-Eval will rewrite the original prompts to
different customized prompts for different models. In particular, we propose
some well-designed modules for prompt customization tailored to the scenario of
MLLM evaluation. Extensive experiments demonstrate the effectiveness of our
approach to uncovering models' capabilities, and TP-Eval should benefit the
community in developing more comprehensive and convincing MLLM evaluation
benchmarks.
6719f8a4defdd700b137bd79
Yuxuan Xie
6719f8a4defdd700b137bd7a
Tianhua Li
6719f8a4defdd700b137bd7b
Wenqi Shao
admin_assigned
2024-10-24T08:36:16.397Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
6719f8a4defdd700b137bd7c
Kaipeng Zhang
admin_assigned
2024-10-24T08:36:10.188Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing
  Prompts
Recently, multimodal large language models (MLLMs) have received much
attention for their impressive capabilities. The evaluation of MLLMs is
becoming critical to analyzing attributes of MLLMs and providing valuable
insights. However, current benchmarks overlook the problem of prompt
sensitivity - minor prompt variations may lead to significant performance
fluctuations. Thus, inappropriate prompts may obscure the models' capabilities,
underestimating the models' performance. Moreover, different models have
different preferences for different prompts, and thus, using the same prompt
for all models will cause evaluation bias. This paper analyzes this deficiency
in existing benchmarks and further introduces a new evaluation framework named
TP-Eval, which introduces a prompt customization method to reduce evaluation
biases and tap models' potential. TP-Eval will rewrite the original prompts to
different customized prompts for different models. In particular, we propose
some well-designed modules for prompt customization tailored to the scenario of
MLLM evaluation. Extensive experiments demonstrate the effectiveness of our
approach to uncovering models' capabilities, and TP-Eval should benefit the
community in developing more comprehensive and convincing MLLM evaluation
benchmarks.
A new evaluation framework named TP-Eval addresses prompt sensitivity in multimodal large language models, enhancing performance assessment by customizing prompts for different models.
multimodal large language models MLLMs prompt sensitivity evaluation framework TP-Eval prompt customization evaluation biases