title: Visual Haystacks: Answering Harder Questions About Sets of Images
summary: Recent advancements in Large Multimodal Models (LMMs) have made significant
  progress in the field of single-image visual question answering. However, these
  models face substantial challenges when tasked with queries that span extensive
  collections of images, similar to real-world scenarios like searching through
  large photo albums, finding specific information across the internet, or
  monitoring environmental changes through satellite imagery. This paper explores
  the task of Multi-Image Visual Question Answering (MIQA): given a large set of
  images and a natural language query, the task is to generate a relevant and
  grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
  (VHs)," specifically designed to evaluate LMMs' capabilities in visual
  retrieval and reasoning over sets of unrelated images, where we perform
  comprehensive evaluations demonstrating that even robust closed-source models
  struggle significantly. Towards addressing these shortcomings, we introduce
  MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
  framework tailored for LMMs that confronts the challenges of MIQA with marked
  efficiency and accuracy improvements over baseline methods. Our evaluation
  shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
  benchmark and offers up to 3.4x improvements in efficiency over text-focused
  multi-stage approaches.
paper_authors._id: 669fc8149c7cc392d3a19489
paper_authors.name: Tsung-Han Wu
paper_authors._id: 669fc8149c7cc392d3a1948a
paper_authors.name: Giscard Biamby
paper_authors._id: 669fc8149c7cc392d3a1948b
paper_authors.name: Jerome Quenum
paper_authors._id: 669fc8149c7cc392d3a1948c
paper_authors.name: Ritwik Gupta
paper_authors._id: 669fc8149c7cc392d3a1948d
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 669fc8149c7cc392d3a1948e
paper_authors.name: Trevor Darrell
paper_authors._id: 669fc8149c7cc392d3a1948f
paper_authors.name: David M. Chan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T15:51:46.294Z
paper_authors._id: 6388f68c43d8b0797a09ff84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669920369989-noauth.jpeg
paper_authors.fullname: David Chan
paper_authors.type: user
paper_authors.user: davidchan
paper_title: Visual Haystacks: Answering Harder Questions About Sets of Images
paper_summary: Recent advancements in Large Multimodal Models (LMMs) have made significant
  progress in the field of single-image visual question answering. However, these
  models face substantial challenges when tasked with queries that span extensive
  collections of images, similar to real-world scenarios like searching through
  large photo albums, finding specific information across the internet, or
  monitoring environmental changes through satellite imagery. This paper explores
  the task of Multi-Image Visual Question Answering (MIQA): given a large set of
  images and a natural language query, the task is to generate a relevant and
  grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
  (VHs)," specifically designed to evaluate LMMs' capabilities in visual
  retrieval and reasoning over sets of unrelated images, where we perform
  comprehensive evaluations demonstrating that even robust closed-source models
  struggle significantly. Towards addressing these shortcomings, we introduce
  MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
  framework tailored for LMMs that confronts the challenges of MIQA with marked
  efficiency and accuracy improvements over baseline methods. Our evaluation
  shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
  benchmark and offers up to 3.4x improvements in efficiency over text-focused
  multi-stage approaches.
paper_ai_keywords: Large Multimodal Models (LMMs) Multi-Image Visual Question Answering (MIQA) Visual Haystacks (VHs) Multi-Image Retrieval Augmented Generation (MIRAGE)