title: WildVision: Evaluating Vision-Language Models in the Wild with Human
    Preferences
summary: Recent breakthroughs in vision-language models (VLMs) emphasize the necessity
  of benchmarking human preferences in real-world multimodal interactions. To
  address this gap, we launched WildVision-Arena (WV-Arena), an online platform
  that collects human preferences to evaluate VLMs. We curated WV-Bench by
  selecting 500 high-quality samples from 8,000 user submissions in WV-Arena.
  WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet,
  achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This
  significantly outperforms other benchmarks like MMVet, MMMU, and MMStar.
    Our comprehensive analysis of 20K real-world interactions reveals important
  insights into the failure cases of top-performing VLMs. For example, we find
  that although GPT-4V surpasses many other models like Reka-Flash, Opus, and
  Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces
  challenges with subtle contextual cues, spatial reasoning, visual imagination,
  and expert domain knowledge. Additionally, current VLMs exhibit issues with
  hallucinations and safety when intentionally provoked. We are releasing our
  chat and feedback data to further advance research in the field of VLMs.
paper_authors._id: 667115bcc4148dbf42826c03
paper_authors.name: Yujie Lu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-18T05:06:07.210Z
paper_authors._id: 633cfb6ad5935998f7582b7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633cfb6ad5935998f7582b7b/mzU4lI_GWPsYOFZHTBsYA.jpeg
paper_authors.fullname: Yujie Lu
paper_authors.type: user
paper_authors.user: FunCube
paper_authors._id: 667115bcc4148dbf42826c04
paper_authors.name: Dongfu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:17:48.606Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 667115bcc4148dbf42826c05
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:29:35.280Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 667115bcc4148dbf42826c06
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:29:42.006Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_authors._id: 667115bcc4148dbf42826c07
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:29:48.493Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 667115bcc4148dbf42826c08
paper_authors.name: Bill Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T07:10:17.789Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_title: WildVision: Evaluating Vision-Language Models in the Wild with Human
    Preferences
paper_summary: Recent breakthroughs in vision-language models (VLMs) emphasize the necessity
  of benchmarking human preferences in real-world multimodal interactions. To
  address this gap, we launched WildVision-Arena (WV-Arena), an online platform
  that collects human preferences to evaluate VLMs. We curated WV-Bench by
  selecting 500 high-quality samples from 8,000 user submissions in WV-Arena.
  WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet,
  achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This
  significantly outperforms other benchmarks like MMVet, MMMU, and MMStar.
    Our comprehensive analysis of 20K real-world interactions reveals important
  insights into the failure cases of top-performing VLMs. For example, we find
  that although GPT-4V surpasses many other models like Reka-Flash, Opus, and
  Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces
  challenges with subtle contextual cues, spatial reasoning, visual imagination,
  and expert domain knowledge. Additionally, current VLMs exhibit issues with
  hallucinations and safety when intentionally provoked. We are releasing our
  chat and feedback data to further advance research in the field of VLMs.
paper_ai_summary: WildVision-Arena evaluates vision-language models using human preferences collected from real-world interactions, providing insights into their performance and identifying areas for improvement.
paper_ai_keywords: vision-language models WildVision-Arena WV-Arena WV-Bench GPT-4 Claude-3-Sonnet Spearman correlation Elo rating MMVet MMMU MMStar visual recognition reasoning tasks contextual cues spatial reasoning visual imagination expert domain knowledge hallucinations safety