title: MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models
    for Integrated Capabilities
summary: MM-Vet, with open-ended vision-language questions targeting at evaluating
  integrated capabilities, has become one of the most popular benchmarks for
  large multimodal model evaluation. MM-Vet assesses six core vision-language
  (VL) capabilities: recognition, knowledge, spatial awareness, language
  generation, OCR, and math. However, its question format is restricted to single
  image-text pairs, lacking the interleaved image and text sequences prevalent in
  real-world scenarios. To address this limitation, we introduce MM-Vet v2, which
  includes a new VL capability called "image-text sequence understanding",
  evaluating models' ability to process VL sequences. Furthermore, we maintain
  the high quality of evaluation samples while further expanding the evaluation
  set size. Using MM-Vet v2 to benchmark large multimodal models, we found that
  Claude 3.5 Sonnet is the best model with a score of 71.8, slightly
  outperforming GPT-4o which scored 71.0. Among open-weight models,
  InternVL2-Llama3-76B leads with a score of 68.4.
paper_authors._id: 66ac4a1b6c84fba4552abfde
paper_authors.name: Weihao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:24:01.048Z
paper_authors._id: 5df833bdda6d0311fd3d5403
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
paper_authors.fullname: Weihao Yu
paper_authors.type: user
paper_authors.user: whyu
paper_authors._id: 66ac4a1b6c84fba4552abfdf
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:24:12.708Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 66ac4a1b6c84fba4552abfe0
paper_authors.name: Linfeng Ren
paper_authors._id: 66ac4a1b6c84fba4552abfe1
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:24:42.499Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 66ac4a1b6c84fba4552abfe2
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:24:54.370Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 66ac4a1b6c84fba4552abfe3
paper_authors.name: Kevin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:25:08.200Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 66ac4a1b6c84fba4552abfe4
paper_authors.name: Chung-Ching Lin
paper_authors._id: 66ac4a1b6c84fba4552abfe5
paper_authors.name: Zicheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:25:27.461Z
paper_authors._id: 6569f6cb44ce94a70187f407
paper_authors.avatarUrl: /avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
paper_authors.fullname: Zicheng Liu
paper_authors.type: user
paper_authors.user: ZCLiu35
paper_authors._id: 66ac4a1b6c84fba4552abfe6
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:25:50.777Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 66ac4a1b6c84fba4552abfe7
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:26:36.131Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models
    for Integrated Capabilities
paper_summary: MM-Vet, with open-ended vision-language questions targeting at evaluating
  integrated capabilities, has become one of the most popular benchmarks for
  large multimodal model evaluation. MM-Vet assesses six core vision-language
  (VL) capabilities: recognition, knowledge, spatial awareness, language
  generation, OCR, and math. However, its question format is restricted to single
  image-text pairs, lacking the interleaved image and text sequences prevalent in
  real-world scenarios. To address this limitation, we introduce MM-Vet v2, which
  includes a new VL capability called "image-text sequence understanding",
  evaluating models' ability to process VL sequences. Furthermore, we maintain
  the high quality of evaluation samples while further expanding the evaluation
  set size. Using MM-Vet v2 to benchmark large multimodal models, we found that
  Claude 3.5 Sonnet is the best model with a score of 71.8, slightly
  outperforming GPT-4o which scored 71.0. Among open-weight models,
  InternVL2-Llama3-76B leads with a score of 68.4.
paper_ai_keywords: vision-language (VL) capabilities image-text pairs image-text sequence understanding multimodal models Claude 3.5 Sonnet GPT-4o InternVL2-Llama3-76B