title: VHELM: A Holistic Evaluation of Vision Language Models
summary: Current benchmarks for assessing vision-language models (VLMs) often focus on
  their perception or problem-solving capabilities and neglect other critical
  aspects such as fairness, multilinguality, or toxicity. Furthermore, they
  differ in their evaluation procedures and the scope of the evaluation, making
  it difficult to compare models. To address these issues, we extend the HELM
  framework to VLMs to present the Holistic Evaluation of Vision Language Models
  (VHELM). VHELM aggregates various datasets to cover one or more of the 9
  aspects: visual perception, knowledge, reasoning, bias, fairness,
  multilinguality, robustness, toxicity, and safety. In doing so, we produce a
  comprehensive, multi-dimensional view of the capabilities of the VLMs across
  these important factors. In addition, we standardize the standard inference
  parameters, methods of prompting, and evaluation metrics to enable fair
  comparisons across models. Our framework is designed to be lightweight and
  automatic so that evaluation runs are cheap and fast. Our initial run evaluates
  22 VLMs on 21 existing datasets to provide a holistic snapshot of the models.
  We uncover new key findings, such as the fact that efficiency-focused models
  (e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than
  their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark
  but not when evaluated on the other aspects. For transparency, we release the
  raw model generations and complete results on our website
  (https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living
  benchmark, and we hope to continue adding new datasets and models over time.
paper_authors._id: 6708205234300d85742d61b1
paper_authors.name: Tony Lee
paper_authors._id: 6708205234300d85742d61b2
paper_authors.name: Haoqin Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:40.117Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 6708205234300d85742d61b3
paper_authors.name: Chi Heem Wong
paper_authors._id: 6708205234300d85742d61b4
paper_authors.name: Wenhao Zheng
paper_authors._id: 6708205234300d85742d61b5
paper_authors.name: Yiyang Zhou
paper_authors._id: 6708205234300d85742d61b6
paper_authors.name: Yifan Mai
paper_authors._id: 6708205234300d85742d61b7
paper_authors.name: Josselin Somerville Roberts
paper_authors._id: 6708205234300d85742d61b8
paper_authors.name: Michihiro Yasunaga
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T16:39:14.717Z
paper_authors._id: 621e9388345a1d9ab65391c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e9388345a1d9ab65391c3/RxurNzyAWJOUdgeSHQi1R.jpeg
paper_authors.fullname: Michihiro Yasunaga
paper_authors.type: user
paper_authors.user: michiyasunaga
paper_authors._id: 6708205234300d85742d61b9
paper_authors.name: Huaxiu Yao
paper_authors._id: 6708205234300d85742d61ba
paper_authors.name: Cihang Xie
paper_authors._id: 6708205234300d85742d61bb
paper_authors.name: Percy Liang
paper_title: VHELM: A Holistic Evaluation of Vision Language Models
paper_summary: Current benchmarks for assessing vision-language models (VLMs) often focus on
  their perception or problem-solving capabilities and neglect other critical
  aspects such as fairness, multilinguality, or toxicity. Furthermore, they
  differ in their evaluation procedures and the scope of the evaluation, making
  it difficult to compare models. To address these issues, we extend the HELM
  framework to VLMs to present the Holistic Evaluation of Vision Language Models
  (VHELM). VHELM aggregates various datasets to cover one or more of the 9
  aspects: visual perception, knowledge, reasoning, bias, fairness,
  multilinguality, robustness, toxicity, and safety. In doing so, we produce a
  comprehensive, multi-dimensional view of the capabilities of the VLMs across
  these important factors. In addition, we standardize the standard inference
  parameters, methods of prompting, and evaluation metrics to enable fair
  comparisons across models. Our framework is designed to be lightweight and
  automatic so that evaluation runs are cheap and fast. Our initial run evaluates
  22 VLMs on 21 existing datasets to provide a holistic snapshot of the models.
  We uncover new key findings, such as the fact that efficiency-focused models
  (e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than
  their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark
  but not when evaluated on the other aspects. For transparency, we release the
  raw model generations and complete results on our website
  (https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living
  benchmark, and we hope to continue adding new datasets and models over time.
paper_ai_keywords: HELM framework Holistic Evaluation of Vision Language Models (VHELM) visual perception knowledge reasoning bias fairness multilinguality robustness toxicity safety standard inference parameters prompting methods evaluation metrics efficiency-focused models