VHELM: A Holistic Evaluation of Vision Language Models
Current benchmarks for assessing vision-language models (VLMs) often focus on
their perception or problem-solving capabilities and neglect other critical
aspects such as fairness, multilinguality, or toxicity. Furthermore, they
differ in their evaluation procedures and the scope of the evaluation, making
it difficult to compare models. To address these issues, we extend the HELM
framework to VLMs to present the Holistic Evaluation of Vision Language Models
(VHELM). VHELM aggregates various datasets to cover one or more of the 9
aspects: visual perception, knowledge, reasoning, bias, fairness,
multilinguality, robustness, toxicity, and safety. In doing so, we produce a
comprehensive, multi-dimensional view of the capabilities of the VLMs across
these important factors. In addition, we standardize the standard inference
parameters, methods of prompting, and evaluation metrics to enable fair
comparisons across models. Our framework is designed to be lightweight and
automatic so that evaluation runs are cheap and fast. Our initial run evaluates
22 VLMs on 21 existing datasets to provide a holistic snapshot of the models.
We uncover new key findings, such as the fact that efficiency-focused models
(e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than
their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark
but not when evaluated on the other aspects. For transparency, we release the
raw model generations and complete results on our website
(https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living
benchmark, and we hope to continue adding new datasets and models over time.
6708205234300d85742d61b1
Tony Lee
6708205234300d85742d61b2
Haoqin Tu
claimed_verified
2024-12-02T08:51:40.117Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
6708205234300d85742d61b3
Chi Heem Wong
6708205234300d85742d61b4
Wenhao Zheng
6708205234300d85742d61b5
Yiyang Zhou
6708205234300d85742d61b6
Yifan Mai
6708205234300d85742d61b7
Josselin Somerville Roberts
6708205234300d85742d61b8
Michihiro Yasunaga
claimed_verified
2025-02-21T16:39:14.717Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
6708205234300d85742d61b9
Huaxiu Yao
6708205234300d85742d61ba
Cihang Xie
claimed_verified
2025-07-31T08:17:16.010Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
6708205234300d85742d61bb
Percy Liang
VHELM: A Holistic Evaluation of Vision Language Models
Current benchmarks for assessing vision-language models (VLMs) often focus on
their perception or problem-solving capabilities and neglect other critical
aspects such as fairness, multilinguality, or toxicity. Furthermore, they
differ in their evaluation procedures and the scope of the evaluation, making
it difficult to compare models. To address these issues, we extend the HELM
framework to VLMs to present the Holistic Evaluation of Vision Language Models
(VHELM). VHELM aggregates various datasets to cover one or more of the 9
aspects: visual perception, knowledge, reasoning, bias, fairness,
multilinguality, robustness, toxicity, and safety. In doing so, we produce a
comprehensive, multi-dimensional view of the capabilities of the VLMs across
these important factors. In addition, we standardize the standard inference
parameters, methods of prompting, and evaluation metrics to enable fair
comparisons across models. Our framework is designed to be lightweight and
automatic so that evaluation runs are cheap and fast. Our initial run evaluates
22 VLMs on 21 existing datasets to provide a holistic snapshot of the models.
We uncover new key findings, such as the fact that efficiency-focused models
(e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than
their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark
but not when evaluated on the other aspects. For transparency, we release the
raw model generations and complete results on our website
(https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living
benchmark, and we hope to continue adding new datasets and models over time.
VHELM extends the HELM framework to evaluate vision-language models across multiple critical aspects, including fairness, multilinguality, bias, and toxicity, using standardized inference parameters and evaluation metrics.
vision-language models HELM framework Holistic Evaluation of Vision Language Models VHELM visual perception knowledge reasoning bias fairness multilinguality robustness toxicity safety datasets standard inference parameters prompting methods evaluation metrics