title: Vision Search Assistant: Empower Vision-Language Models as Multimodal
    Search Engines
summary: Search engines enable the retrieval of unknown information with texts.
  However, traditional methods fall short when it comes to understanding
  unfamiliar visual content, such as identifying an object that the model has
  never seen before. This challenge is particularly pronounced for large
  vision-language models (VLMs): if the model has not been exposed to the object
  depicted in an image, it struggles to generate reliable answers to the user's
  question regarding that image. Moreover, as new objects and events continuously
  emerge, frequently updating VLMs is impractical due to heavy computational
  burdens. To address this limitation, we propose Vision Search Assistant, a
  novel framework that facilitates collaboration between VLMs and web agents.
  This approach leverages VLMs' visual understanding capabilities and web agents'
  real-time information access to perform open-world Retrieval-Augmented
  Generation via the web. By integrating visual and textual representations
  through this collaboration, the model can provide informed responses even when
  the image is novel to the system. Extensive experiments conducted on both
  open-set and closed-set QA benchmarks demonstrate that the Vision Search
  Assistant significantly outperforms the other models and can be widely applied
  to existing VLMs.
paper_authors._id: 67204b36cbdaf3111383d445
paper_authors.name: Zhixin Zhang
paper_authors._id: 67204b36cbdaf3111383d446
paper_authors.name: Yiyuan Zhang
paper_authors._id: 67204b36cbdaf3111383d447
paper_authors.name: Xiaohan Ding
paper_authors._id: 67204b36cbdaf3111383d448
paper_authors.name: Xiangyu Yue
paper_title: Vision Search Assistant: Empower Vision-Language Models as Multimodal
    Search Engines
paper_summary: Search engines enable the retrieval of unknown information with texts.
  However, traditional methods fall short when it comes to understanding
  unfamiliar visual content, such as identifying an object that the model has
  never seen before. This challenge is particularly pronounced for large
  vision-language models (VLMs): if the model has not been exposed to the object
  depicted in an image, it struggles to generate reliable answers to the user's
  question regarding that image. Moreover, as new objects and events continuously
  emerge, frequently updating VLMs is impractical due to heavy computational
  burdens. To address this limitation, we propose Vision Search Assistant, a
  novel framework that facilitates collaboration between VLMs and web agents.
  This approach leverages VLMs' visual understanding capabilities and web agents'
  real-time information access to perform open-world Retrieval-Augmented
  Generation via the web. By integrating visual and textual representations
  through this collaboration, the model can provide informed responses even when
  the image is novel to the system. Extensive experiments conducted on both
  open-set and closed-set QA benchmarks demonstrate that the Vision Search
  Assistant significantly outperforms the other models and can be widely applied
  to existing VLMs.
paper_ai_keywords: vision-language models (VLMs) web agents Retrieval-Augmented Generation visual and textual representations open-set QA benchmarks closed-set QA benchmarks