title: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
summary: We propose MM-Vet, an evaluation benchmark that examines large multimodal
  models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various
  intriguing abilities, such as solving math problems written on the blackboard,
  reasoning about events and celebrities in news images, and explaining visual
  jokes. Rapid model advancements pose challenges to evaluation benchmark
  development. Problems include: (1) How to systematically structure and evaluate
  the complicated multimodal tasks; (2) How to design evaluation metrics that
  work well across question and answer types; and (3) How to give model insights
  beyond a simple performance ranking. To this end, we present MM-Vet, designed
  based on the insight that the intriguing ability to solve complicated tasks is
  often achieved by a generalist model being able to integrate different core
  vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and
  examines the 16 integrations of interest derived from the capability
  combination. For evaluation metrics, we propose an LLM-based evaluator for
  open-ended outputs. The evaluator enables the evaluation across different
  question types and answer styles, resulting in a unified scoring metric. We
  evaluate representative LMMs on MM-Vet, providing insights into the
  capabilities of different LMM system paradigms and models. Code and data are
  available at https://github.com/yuweihao/MM-Vet.
paper_authors._id: 64d06f239e9ca8123dc5ab45
paper_authors.name: Weihao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:50:19.998Z
paper_authors._id: 5df833bdda6d0311fd3d5403
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
paper_authors.fullname: Weihao Yu
paper_authors.type: user
paper_authors.user: whyu
paper_authors._id: 64d06f239e9ca8123dc5ab46
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:50:47.078Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 64d06f239e9ca8123dc5ab47
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:51:04.437Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 64d06f239e9ca8123dc5ab48
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:51:28.557Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 64d06f239e9ca8123dc5ab49
paper_authors.name: Kevin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-08T01:19:37.040Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 64d06f239e9ca8123dc5ab4a
paper_authors.name: Zicheng Liu
paper_authors._id: 64d06f239e9ca8123dc5ab4b
paper_authors.name: Xinchao Wang
paper_authors._id: 64d06f239e9ca8123dc5ab4c
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:52:29.085Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
paper_summary: We propose MM-Vet, an evaluation benchmark that examines large multimodal
  models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various
  intriguing abilities, such as solving math problems written on the blackboard,
  reasoning about events and celebrities in news images, and explaining visual
  jokes. Rapid model advancements pose challenges to evaluation benchmark
  development. Problems include: (1) How to systematically structure and evaluate
  the complicated multimodal tasks; (2) How to design evaluation metrics that
  work well across question and answer types; and (3) How to give model insights
  beyond a simple performance ranking. To this end, we present MM-Vet, designed
  based on the insight that the intriguing ability to solve complicated tasks is
  often achieved by a generalist model being able to integrate different core
  vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and
  examines the 16 integrations of interest derived from the capability
  combination. For evaluation metrics, we propose an LLM-based evaluator for
  open-ended outputs. The evaluator enables the evaluation across different
  question types and answer styles, resulting in a unified scoring metric. We
  evaluate representative LMMs on MM-Vet, providing insights into the
  capabilities of different LMM system paradigms and models. Code and data are
  available at https://github.com/yuweihao/MM-Vet.
paper_ai_summary: MM-Vet evaluates large multimodal models across complicated tasks using defined vision-language capabilities and a unified LLM-based evaluator.
paper_ai_keywords: multimodal models VL capabilities LLM-based evaluator vision-language