title: Guiding Vision-Language Model Selection for Visual Question-Answering
    Across Tasks, Domains, and Knowledge Types
summary: Visual Question-Answering (VQA) has become a key use-case in several
  applications to aid user experience, particularly after Vision-Language Models
  (VLMs) achieving good results in zero-shot inference. But evaluating different
  VLMs for an application requirement using a standardized framework in practical
  settings is still challenging. This paper introduces a comprehensive framework
  for evaluating VLMs tailored to VQA tasks in practical settings. We present a
  novel dataset derived from established VQA benchmarks, annotated with task
  types, application domains, and knowledge types, three key practical aspects on
  which tasks can vary. We also introduce GoEval, a multimodal evaluation metric
  developed using GPT-4o, achieving a correlation factor of 56.71% with human
  judgments. Our experiments with ten state-of-the-art VLMs reveals that no
  single model excelling universally, making appropriate selection a key design
  decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally
  outperform others, though open-source models like InternVL-2-8B and
  CogVLM-2-Llama-3-19B demonstrate competitive strengths in specific contexts,
  while providing additional advantages. This study guides the selection of VLMs
  based on specific task requirements and resource constraints, and can also be
  extended to other vision-language tasks.
paper_authors._id: 66e9108fcfd2f509914b1e94
paper_authors.name: Neelabh Sinha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:45:52.171Z
paper_authors._id: 6639a17d7256327c68a0841e
paper_authors.avatarUrl: /avatars/28fcd935149c2d89661c6f642842c310.svg
paper_authors.fullname: Neelabh Sinha
paper_authors.type: user
paper_authors.user: neelabhsinha
paper_authors._id: 66e9108fcfd2f509914b1e95
paper_authors.name: Vinija Jain
paper_authors._id: 66e9108fcfd2f509914b1e96
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T07:11:18.674Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_title: Guiding Vision-Language Model Selection for Visual Question-Answering
    Across Tasks, Domains, and Knowledge Types
paper_summary: Visual Question-Answering (VQA) has become a key use-case in several
  applications to aid user experience, particularly after Vision-Language Models
  (VLMs) achieving good results in zero-shot inference. But evaluating different
  VLMs for an application requirement using a standardized framework in practical
  settings is still challenging. This paper introduces a comprehensive framework
  for evaluating VLMs tailored to VQA tasks in practical settings. We present a
  novel dataset derived from established VQA benchmarks, annotated with task
  types, application domains, and knowledge types, three key practical aspects on
  which tasks can vary. We also introduce GoEval, a multimodal evaluation metric
  developed using GPT-4o, achieving a correlation factor of 56.71% with human
  judgments. Our experiments with ten state-of-the-art VLMs reveals that no
  single model excelling universally, making appropriate selection a key design
  decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally
  outperform others, though open-source models like InternVL-2-8B and
  CogVLM-2-Llama-3-19B demonstrate competitive strengths in specific contexts,
  while providing additional advantages. This study guides the selection of VLMs
  based on specific task requirements and resource constraints, and can also be
  extended to other vision-language tasks.
paper_ai_summary: A framework for evaluating Vision-Language Models in VQA tasks using a novel dataset and GoEval metric, highlighting performance variation across models and guiding selection based on task and resource constraints.
paper_ai_keywords: Vision-Language Models VQA zero-shot inference evaluation framework multimodal evaluation metric GoEval GPT-4o task types application domains knowledge types Gemini-1.5-Pro GPT-4o-mini InternVL-2-8B CogVLM-2-Llama-3-19B