title: VisionArena: 230K Real World User-VLM Conversations with Preference
    Labels
summary: With the growing adoption and capabilities of vision-language models (VLMs)
  comes the need for benchmarks that capture authentic user-VLM interactions. In
  response, we create VisionArena, a dataset of 230K real-world conversations
  between users and VLMs. Collected from Chatbot Arena - an open-source platform
  where users interact with VLMs and submit preference votes - VisionArena spans
  73K unique users, 45 VLMs, and 138 languages. Our dataset contains three
  subsets: VisionArena-Chat, 200k single and multi-turn conversations between a
  user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous
  VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark
  of 500 diverse user prompts that efficiently approximate the live Chatbot Arena
  model rankings. Additionally, we highlight the types of question asked by
  users, the influence of response style on preference, and areas where models
  often fail. We find open-ended tasks like captioning and humor are highly
  style-dependent, and current VLMs struggle with spatial reasoning and planning
  tasks. Lastly, we show finetuning the same base model on VisionArena-Chat
  outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point
  gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai
paper_authors._id: 675c7167458594642e5526e6
paper_authors.name: Christopher Chou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:04.363Z
paper_authors._id: 62fb0555e8c9c532aa7c6e5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fb0555e8c9c532aa7c6e5d/AUds_M-PA7TCrj3gQ15IN.jpeg
paper_authors.fullname: Christopher Chou
paper_authors.type: user
paper_authors.user: BabyChou
paper_authors._id: 675c7167458594642e5526e7
paper_authors.name: Lisa Dunlap
paper_authors._id: 675c7167458594642e5526e8
paper_authors.name: Koki Mashita
paper_authors._id: 675c7167458594642e5526e9
paper_authors.name: Krishna Mandal
paper_authors._id: 675c7167458594642e5526ea
paper_authors.name: Trevor Darrell
paper_authors._id: 675c7167458594642e5526eb
paper_authors.name: Ion Stoica
paper_authors._id: 675c7167458594642e5526ec
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 675c7167458594642e5526ed
paper_authors.name: Wei-Lin Chiang
paper_title: VisionArena: 230K Real World User-VLM Conversations with Preference
    Labels
paper_summary: With the growing adoption and capabilities of vision-language models (VLMs)
  comes the need for benchmarks that capture authentic user-VLM interactions. In
  response, we create VisionArena, a dataset of 230K real-world conversations
  between users and VLMs. Collected from Chatbot Arena - an open-source platform
  where users interact with VLMs and submit preference votes - VisionArena spans
  73K unique users, 45 VLMs, and 138 languages. Our dataset contains three
  subsets: VisionArena-Chat, 200k single and multi-turn conversations between a
  user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous
  VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark
  of 500 diverse user prompts that efficiently approximate the live Chatbot Arena
  model rankings. Additionally, we highlight the types of question asked by
  users, the influence of response style on preference, and areas where models
  often fail. We find open-ended tasks like captioning and humor are highly
  style-dependent, and current VLMs struggle with spatial reasoning and planning
  tasks. Lastly, we show finetuning the same base model on VisionArena-Chat
  outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point
  gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai
paper_ai_summary: VisionArena is a comprehensive dataset of real-world user-VLM interactions that benchmarks model performance across diverse tasks and preferences.
paper_ai_keywords: vision-language models VLMs Chatbot Arena VisionArena-Chat VisionArena-Battle VisionArena-Bench captioning humor spatial reasoning planning tasks finetuning MMMU WildVision