title: Automated Generation of Challenging Multiple-Choice Questions for Vision
    Language Model Evaluation
summary: The rapid development of vision language models (VLMs) demands rigorous and
  reliable evaluation. However, current visual question answering (VQA)
  benchmarks often depend on open-ended questions, making accurate evaluation
  difficult due to the variability in natural language responses. To address
  this, we introduce AutoConverter, an agentic framework that automatically
  converts these open-ended questions into multiple-choice format, enabling
  objective evaluation while reducing the costly question creation process. Our
  experiments demonstrate that AutoConverter can generate correct and challenging
  multiple-choice questions, with VLMs demonstrating consistently similar or
  lower accuracy on these questions compared to human-created ones. Using
  AutoConverter, we construct VMCBench, a benchmark created by transforming 20
  existing VQA datasets into a unified multiple-choice format, totaling 9,018
  questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
  setting a new standard for scalable, consistent, and reproducible VLM
  evaluation.
paper_authors._id: 677ca00e34a1a97687b77c65
paper_authors.name: Yuhui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:14.449Z
paper_authors._id: 62da55164398e21bf7f0e292
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
paper_authors.fullname: Yuhui Zhang
paper_authors.type: user
paper_authors.user: yuhuizhang
paper_authors._id: 677ca00e34a1a97687b77c66
paper_authors.name: Yuchang Su
paper_authors._id: 677ca00e34a1a97687b77c67
paper_authors.name: Yiming Liu
paper_authors._id: 677ca00e34a1a97687b77c68
paper_authors.name: Xiaohan Wang
paper_authors._id: 677ca00e34a1a97687b77c69
paper_authors.name: James Burgess
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:51:12.432Z
paper_authors._id: 650871aeb44445e9b3625c7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
paper_authors.fullname: James Burgess
paper_authors.type: user
paper_authors.user: jmhb
paper_authors._id: 677ca00e34a1a97687b77c6a
paper_authors.name: Elaine Sui
paper_authors._id: 677ca00e34a1a97687b77c6b
paper_authors.name: Chenyu Wang
paper_authors._id: 677ca00e34a1a97687b77c6c
paper_authors.name: Josiah Aklilu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T21:55:48.814Z
paper_authors._id: 66844ab799dbd7c30a19a0fe
paper_authors.avatarUrl: /avatars/5a5f8fa530e127105812bd1d0e515db8.svg
paper_authors.fullname: Josiah Aklilu
paper_authors.type: user
paper_authors.user: josaklil-ai
paper_authors._id: 677ca00e34a1a97687b77c6d
paper_authors.name: Alejandro Lozano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:12.182Z
paper_authors._id: 630e56d6f6f6d700f50be105
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e56d6f6f6d700f50be105/_gzPXZeL2LP8oalMCriNN.jpeg
paper_authors.fullname: Alejandro Lozano
paper_authors.type: user
paper_authors.user: Alejandro98
paper_authors._id: 677ca00e34a1a97687b77c6e
paper_authors.name: Anjiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:48:08.230Z
paper_authors._id: 674286496efe2b931f7ce354
paper_authors.avatarUrl: /avatars/8f920618b777dbff5f2a117ebd9e9caa.svg
paper_authors.fullname: Anjiang Wei
paper_authors.type: user
paper_authors.user: anjiangwei
paper_authors._id: 677ca00e34a1a97687b77c6f
paper_authors.name: Ludwig Schmidt
paper_authors._id: 677ca00e34a1a97687b77c70
paper_authors.name: Serena Yeung-Levy
paper_title: Automated Generation of Challenging Multiple-Choice Questions for Vision
    Language Model Evaluation
paper_summary: The rapid development of vision language models (VLMs) demands rigorous and
  reliable evaluation. However, current visual question answering (VQA)
  benchmarks often depend on open-ended questions, making accurate evaluation
  difficult due to the variability in natural language responses. To address
  this, we introduce AutoConverter, an agentic framework that automatically
  converts these open-ended questions into multiple-choice format, enabling
  objective evaluation while reducing the costly question creation process. Our
  experiments demonstrate that AutoConverter can generate correct and challenging
  multiple-choice questions, with VLMs demonstrating consistently similar or
  lower accuracy on these questions compared to human-created ones. Using
  AutoConverter, we construct VMCBench, a benchmark created by transforming 20
  existing VQA datasets into a unified multiple-choice format, totaling 9,018
  questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
  setting a new standard for scalable, consistent, and reproducible VLM
  evaluation.
paper_ai_summary: AutoConverter converts open-ended VQA questions to multiple-choice format, enhancing evaluation of vision language models.
paper_ai_keywords: vision language models VLMs visual question answering VQA multiple-choice format AutoConverter VMCBench scalable evaluation reproducible evaluation