WildVision: Evaluating Vision-Language Models in the Wild with Human
  Preferences
Recent breakthroughs in vision-language models (VLMs) emphasize the necessity
of benchmarking human preferences in real-world multimodal interactions. To
address this gap, we launched WildVision-Arena (WV-Arena), an online platform
that collects human preferences to evaluate VLMs. We curated WV-Bench by
selecting 500 high-quality samples from 8,000 user submissions in WV-Arena.
WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet,
achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This
significantly outperforms other benchmarks like MMVet, MMMU, and MMStar.
  Our comprehensive analysis of 20K real-world interactions reveals important
insights into the failure cases of top-performing VLMs. For example, we find
that although GPT-4V surpasses many other models like Reka-Flash, Opus, and
Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces
challenges with subtle contextual cues, spatial reasoning, visual imagination,
and expert domain knowledge. Additionally, current VLMs exhibit issues with
hallucinations and safety when intentionally provoked. We are releasing our
chat and feedback data to further advance research in the field of VLMs.
667115bcc4148dbf42826c03
Yujie Lu
extracted_pending
2024-06-18T05:06:07.210Z
633cfb6ad5935998f7582b7b
https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
Yujie Lu
user
FunCube
667115bcc4148dbf42826c04
Dongfu Jiang
claimed_verified
2024-06-18T07:17:48.606Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
667115bcc4148dbf42826c05
Wenhu Chen
admin_assigned
2024-06-18T15:29:35.280Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
667115bcc4148dbf42826c06
William Yang Wang
admin_assigned
2024-06-18T15:29:42.006Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
667115bcc4148dbf42826c07
Yejin Choi
admin_assigned
2024-06-18T15:29:48.493Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
667115bcc4148dbf42826c08
Bill Yuchen Lin
admin_assigned
2024-06-18T07:10:17.789Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
WildVision: Evaluating Vision-Language Models in the Wild with Human
  Preferences
Recent breakthroughs in vision-language models (VLMs) emphasize the necessity
of benchmarking human preferences in real-world multimodal interactions. To
address this gap, we launched WildVision-Arena (WV-Arena), an online platform
that collects human preferences to evaluate VLMs. We curated WV-Bench by
selecting 500 high-quality samples from 8,000 user submissions in WV-Arena.
WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet,
achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This
significantly outperforms other benchmarks like MMVet, MMMU, and MMStar.
  Our comprehensive analysis of 20K real-world interactions reveals important
insights into the failure cases of top-performing VLMs. For example, we find
that although GPT-4V surpasses many other models like Reka-Flash, Opus, and
Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces
challenges with subtle contextual cues, spatial reasoning, visual imagination,
and expert domain knowledge. Additionally, current VLMs exhibit issues with
hallucinations and safety when intentionally provoked. We are releasing our
chat and feedback data to further advance research in the field of VLMs.
WildVision-Arena evaluates vision-language models using human preferences collected from real-world interactions, providing insights into their performance and identifying areas for improvement.
vision-language models WildVision-Arena WV-Arena WV-Bench GPT-4 Claude-3-Sonnet Spearman correlation Elo rating MMVet MMMU MMStar visual recognition reasoning tasks contextual cues spatial reasoning visual imagination expert domain knowledge hallucinations safety