title: LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
summary: The advances of large foundation models necessitate wide-coverage, low-cost,
  and zero-contamination benchmarks. Despite continuous exploration of language
  model evaluations, comprehensive studies on the evaluation of Large Multi-modal
  Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified
  and standardized multimodal benchmark framework with over 50 tasks and more
  than 10 models to promote transparent and reproducible evaluations. Although
  LMMS-EVAL offers comprehensive coverage, we find it still falls short in
  achieving low cost and zero contamination. To approach this evaluation
  trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that
  emphasizes both coverage and efficiency. Additionally, we present Multimodal
  LIVEBENCH that utilizes continuously updating news and online forums to assess
  models' generalization abilities in the wild, featuring a low-cost and
  zero-contamination evaluation approach. In summary, our work highlights the
  importance of considering the evaluation trilemma and provides practical
  solutions to navigate the trade-offs in evaluating large multi-modal models,
  paving the way for more effective and reliable benchmarking of LMMs. We
  opensource our codebase and maintain leaderboard of LIVEBENCH at
  https://github.com/EvolvingLMMs-Lab/lmms-eval and
  https://huggingface.co/spaces/lmms-lab/LiveBench.
paper_authors._id: 66987d94a029d7f9e39da94f
paper_authors.name: Kaichen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:28.420Z
paper_authors._id: 64bb77e786e7fb5b8a317a43
paper_authors.avatarUrl: /avatars/4c1720db323082e644babeb827c6f5d4.svg
paper_authors.fullname: kcz
paper_authors.type: user
paper_authors.user: kcz358
paper_authors._id: 66987d94a029d7f9e39da950
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:40.807Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 66987d94a029d7f9e39da951
paper_authors.name: Peiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:25.903Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 66987d94a029d7f9e39da952
paper_authors.name: Fanyi Pu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:47.248Z
paper_authors._id: 646e1ef5075bbcc48ddf21e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
paper_authors.fullname: Pu Fanyi
paper_authors.type: user
paper_authors.user: pufanyi
paper_authors._id: 66987d94a029d7f9e39da953
paper_authors.name: Joshua Adrian Cahyono
paper_authors._id: 66987d94a029d7f9e39da954
paper_authors.name: Kairui Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:08:32.574Z
paper_authors._id: 6400ba2b261cfa61f3a00555
paper_authors.avatarUrl: /avatars/1311e0b5e21b1c94d73fcaf455d3c7f7.svg
paper_authors.fullname: Kairui
paper_authors.type: user
paper_authors.user: KairuiHu
paper_authors._id: 66987d94a029d7f9e39da955
paper_authors.name: Shuai Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:38:17.038Z
paper_authors._id: 64f7f5b54101c731ca84ae05
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f7f5b54101c731ca84ae05/13DwdxOo3tWbxKDLd44B9.jpeg
paper_authors.fullname: Shuai Liu
paper_authors.type: user
paper_authors.user: Choiszt
paper_authors._id: 66987d94a029d7f9e39da956
paper_authors.name: Yuanhan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:46.860Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 66987d94a029d7f9e39da957
paper_authors.name: Jingkang Yang
paper_authors._id: 66987d94a029d7f9e39da958
paper_authors.name: Chunyuan Li
paper_authors._id: 66987d94a029d7f9e39da959
paper_authors.name: Ziwei Liu
paper_title: LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
paper_summary: The advances of large foundation models necessitate wide-coverage, low-cost,
  and zero-contamination benchmarks. Despite continuous exploration of language
  model evaluations, comprehensive studies on the evaluation of Large Multi-modal
  Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified
  and standardized multimodal benchmark framework with over 50 tasks and more
  than 10 models to promote transparent and reproducible evaluations. Although
  LMMS-EVAL offers comprehensive coverage, we find it still falls short in
  achieving low cost and zero contamination. To approach this evaluation
  trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that
  emphasizes both coverage and efficiency. Additionally, we present Multimodal
  LIVEBENCH that utilizes continuously updating news and online forums to assess
  models' generalization abilities in the wild, featuring a low-cost and
  zero-contamination evaluation approach. In summary, our work highlights the
  importance of considering the evaluation trilemma and provides practical
  solutions to navigate the trade-offs in evaluating large multi-modal models,
  paving the way for more effective and reliable benchmarking of LMMs. We
  opensource our codebase and maintain leaderboard of LIVEBENCH at
  https://github.com/EvolvingLMMs-Lab/lmms-eval and
  https://huggingface.co/spaces/lmms-lab/LiveBench.
paper_ai_keywords: Large Multi-modal Models (LMMs) LMMS-EVAL multimodal benchmark framework LMMS-EVAL LITE Multimodal LIVEBENCH generalization abilities