title: TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing
    Prompts
summary: Recently, multimodal large language models (MLLMs) have received much
  attention for their impressive capabilities. The evaluation of MLLMs is
  becoming critical to analyzing attributes of MLLMs and providing valuable
  insights. However, current benchmarks overlook the problem of prompt
  sensitivity - minor prompt variations may lead to significant performance
  fluctuations. Thus, inappropriate prompts may obscure the models' capabilities,
  underestimating the models' performance. Moreover, different models have
  different preferences for different prompts, and thus, using the same prompt
  for all models will cause evaluation bias. This paper analyzes this deficiency
  in existing benchmarks and further introduces a new evaluation framework named
  TP-Eval, which introduces a prompt customization method to reduce evaluation
  biases and tap models' potential. TP-Eval will rewrite the original prompts to
  different customized prompts for different models. In particular, we propose
  some well-designed modules for prompt customization tailored to the scenario of
  MLLM evaluation. Extensive experiments demonstrate the effectiveness of our
  approach to uncovering models' capabilities, and TP-Eval should benefit the
  community in developing more comprehensive and convincing MLLM evaluation
  benchmarks.
paper_authors._id: 6719f8a4defdd700b137bd79
paper_authors.name: Yuxuan Xie
paper_authors._id: 6719f8a4defdd700b137bd7a
paper_authors.name: Tianhua Li
paper_authors._id: 6719f8a4defdd700b137bd7b
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:36:16.397Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 6719f8a4defdd700b137bd7c
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:36:10.188Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_title: TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing
    Prompts
paper_summary: Recently, multimodal large language models (MLLMs) have received much
  attention for their impressive capabilities. The evaluation of MLLMs is
  becoming critical to analyzing attributes of MLLMs and providing valuable
  insights. However, current benchmarks overlook the problem of prompt
  sensitivity - minor prompt variations may lead to significant performance
  fluctuations. Thus, inappropriate prompts may obscure the models' capabilities,
  underestimating the models' performance. Moreover, different models have
  different preferences for different prompts, and thus, using the same prompt
  for all models will cause evaluation bias. This paper analyzes this deficiency
  in existing benchmarks and further introduces a new evaluation framework named
  TP-Eval, which introduces a prompt customization method to reduce evaluation
  biases and tap models' potential. TP-Eval will rewrite the original prompts to
  different customized prompts for different models. In particular, we propose
  some well-designed modules for prompt customization tailored to the scenario of
  MLLM evaluation. Extensive experiments demonstrate the effectiveness of our
  approach to uncovering models' capabilities, and TP-Eval should benefit the
  community in developing more comprehensive and convincing MLLM evaluation
  benchmarks.
paper_ai_keywords: multimodal large language models (MLLMs) evaluation framework prompt sensitivity prompt customization TP-Eval evaluation biases prompt customization methods modules for prompt customization MLLM evaluation