title: Benchmarking Multi-Image Understanding in Vision and Language Models:
    Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
summary: The advancement of large language models (LLMs) has significantly broadened
  the scope of applications in natural language processing, with multi-modal LLMs
  extending these capabilities to integrate and interpret visual data. However,
  existing benchmarks for visual language models (VLMs) predominantly focus on
  single-image inputs, neglecting the crucial aspect of multi-image
  understanding. In this paper, we introduce a Multi-Image Relational Benchmark
  MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
  multiple images. Our benchmark encompasses four categories: perception, visual
  world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
  evaluation of a wide range of open-source and closed-source models, we
  demonstrate that while open-source VLMs were shown to approach the performance
  of GPT-4V in single-image tasks, a significant performance gap remains in
  multi-image reasoning tasks. Our findings also reveal that even the
  state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
  need for further research and development in this area. We believe our
  contribution of MIRB could serve as a testbed for developing the
  next-generation multi-modal models.
paper_authors._id: 66727eeef022d14aa15540ea
paper_authors.name: Bingchen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:37:20.887Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 66727eeef022d14aa15540eb
paper_authors.name: Yongshuo Zong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:58:49.432Z
paper_authors._id: 643d5de2c17bfb2256f14b4b
paper_authors.avatarUrl: /avatars/1124128ce69ae6868fd1a2c06420d72f.svg
paper_authors.fullname: Yongshuo Zong
paper_authors.type: user
paper_authors.user: ys-zong
paper_authors._id: 66727eeef022d14aa15540ec
paper_authors.name: Letian Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:58:56.282Z
paper_authors._id: 648efcad5adc64d0d982ff42
paper_authors.avatarUrl: /avatars/449723010f64a5c9e274434ebdc03150.svg
paper_authors.fullname: Letian Zhang
paper_authors.type: user
paper_authors.user: WonderThyme
paper_authors._id: 66727eeef022d14aa15540ed
paper_authors.name: Timothy Hospedales
paper_title: Benchmarking Multi-Image Understanding in Vision and Language Models:
    Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
paper_summary: The advancement of large language models (LLMs) has significantly broadened
  the scope of applications in natural language processing, with multi-modal LLMs
  extending these capabilities to integrate and interpret visual data. However,
  existing benchmarks for visual language models (VLMs) predominantly focus on
  single-image inputs, neglecting the crucial aspect of multi-image
  understanding. In this paper, we introduce a Multi-Image Relational Benchmark
  MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
  multiple images. Our benchmark encompasses four categories: perception, visual
  world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
  evaluation of a wide range of open-source and closed-source models, we
  demonstrate that while open-source VLMs were shown to approach the performance
  of GPT-4V in single-image tasks, a significant performance gap remains in
  multi-image reasoning tasks. Our findings also reveal that even the
  state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
  need for further research and development in this area. We believe our
  contribution of MIRB could serve as a testbed for developing the
  next-generation multi-modal models.
paper_ai_summary: A benchmark for visual language models (VLMs) evaluating their performance across multiple images, highlighting gaps in multi-image reasoning.
paper_ai_keywords: large language models multi-modal LLMs visual language models VLMs single-image inputs multi-image understanding Multi-Image Relational Benchmark MIRB perception visual world knowledge reasoning multi-hop reasoning GPT-4V multi-modal models