title: MMSearch: Benchmarking the Potential of Large Models as Multi-modal
    Search Engines
summary: The advent of Large Language Models (LLMs) has paved the way for AI search
  engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
  interaction. However, most current AI search engines are limited to text-only
  settings, neglecting the multimodal user queries and the text-image interleaved
  nature of website information. Recently, Large Multimodal Models (LMMs) have
  made impressive strides. Yet, whether they can function as AI search engines
  remains under-explored, leaving the potential of LMMs in multimodal search an
  open question. To this end, we first design a delicate pipeline,
  MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
  top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
  assess the multimodal search performance of LMMs. The curated dataset contains
  300 manually collected instances spanning 14 subfields, which involves no
  overlap with the current LMMs' training data, ensuring the correct answer can
  only be obtained within searching. By using MMSearch-Engine, the LMMs are
  evaluated by performing three individual tasks (requery, rerank, and
  summarization), and one challenging end-to-end task with a complete searching
  process. We conduct extensive experiments on closed-source and open-source
  LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
  results, which surpasses the commercial product, Perplexity Pro, in the
  end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
  further present error analysis to unveil current LMMs still struggle to fully
  grasp the multimodal search tasks, and conduct ablation study to indicate the
  potential of scaling test-time computation for AI search engine. We hope
  MMSearch may provide unique insights to guide the future development of
  multimodal AI search engine. Project Page: https://mmsearch.github.io
paper_authors._id: 66ecef9de719fa9f4aaf3a56
paper_authors.name: Dongzhi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:45:40.407Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 66ecef9de719fa9f4aaf3a57
paper_authors.name: Renrui Zhang
paper_authors._id: 66ecef9de719fa9f4aaf3a58
paper_authors.name: Ziyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:44:19.006Z
paper_authors._id: 647d9ab61a1fcad2fdbf2d3d
paper_authors.avatarUrl: /avatars/48c8aeae8979d2c87df8bde922437d62.svg
paper_authors.fullname: Ziyu Guo
paper_authors.type: user
paper_authors.user: ZiyuG
paper_authors._id: 66ecef9de719fa9f4aaf3a59
paper_authors.name: Yanmin Wu
paper_authors._id: 66ecef9de719fa9f4aaf3a5a
paper_authors.name: Jiayi Lei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:44:43.491Z
paper_authors._id: 64c3c72e8f31d1e6c664b052
paper_authors.avatarUrl: /avatars/af1ad5048eaa9dc417837ad02f927911.svg
paper_authors.fullname: jiayi lei
paper_authors.type: user
paper_authors.user: jyjyjyjy
paper_authors._id: 66ecef9de719fa9f4aaf3a5b
paper_authors.name: Pengshuo Qiu
paper_authors._id: 66ecef9de719fa9f4aaf3a5c
paper_authors.name: Pan Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:45:08.810Z
paper_authors._id: 60f5f68fa7fd83d025749234
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
paper_authors.fullname: Pan Lu
paper_authors.type: user
paper_authors.user: lupantech
paper_authors._id: 66ecef9de719fa9f4aaf3a5d
paper_authors.name: Zehui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:45:22.532Z
paper_authors._id: 64892d31cbda0d1cdb956897
paper_authors.avatarUrl: /avatars/3cdafe03a8295124636347d15a099aaf.svg
paper_authors.fullname: Zehui Chen
paper_authors.type: user
paper_authors.user: lovesnowbest
paper_authors._id: 66ecef9de719fa9f4aaf3a5e
paper_authors.name: Guanglu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:45:31.943Z
paper_authors._id: 636a2efffbc31ee68fee5c1c
paper_authors.avatarUrl: /avatars/21e119f23e1e6fe926567409e74d1a89.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Guanglu
paper_authors._id: 66ecef9de719fa9f4aaf3a5f
paper_authors.name: Peng Gao
paper_authors._id: 66ecef9de719fa9f4aaf3a60
paper_authors.name: Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:46:13.519Z
paper_authors._id: 636de85cc4a7a729c164d2b5
paper_authors.avatarUrl: /avatars/3e281e547e1697e1c06805e7e63f3918.svg
paper_authors.fullname: Yu Liu
paper_authors.type: user
paper_authors.user: YuLiu
paper_authors._id: 66ecef9de719fa9f4aaf3a61
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:46:19.307Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 66ecef9de719fa9f4aaf3a62
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:46:30.927Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: MMSearch: Benchmarking the Potential of Large Models as Multi-modal
    Search Engines
paper_summary: The advent of Large Language Models (LLMs) has paved the way for AI search
  engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
  interaction. However, most current AI search engines are limited to text-only
  settings, neglecting the multimodal user queries and the text-image interleaved
  nature of website information. Recently, Large Multimodal Models (LMMs) have
  made impressive strides. Yet, whether they can function as AI search engines
  remains under-explored, leaving the potential of LMMs in multimodal search an
  open question. To this end, we first design a delicate pipeline,
  MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
  top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
  assess the multimodal search performance of LMMs. The curated dataset contains
  300 manually collected instances spanning 14 subfields, which involves no
  overlap with the current LMMs' training data, ensuring the correct answer can
  only be obtained within searching. By using MMSearch-Engine, the LMMs are
  evaluated by performing three individual tasks (requery, rerank, and
  summarization), and one challenging end-to-end task with a complete searching
  process. We conduct extensive experiments on closed-source and open-source
  LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
  results, which surpasses the commercial product, Perplexity Pro, in the
  end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
  further present error analysis to unveil current LMMs still struggle to fully
  grasp the multimodal search tasks, and conduct ablation study to indicate the
  potential of scaling test-time computation for AI search engine. We hope
  MMSearch may provide unique insights to guide the future development of
  multimodal AI search engine. Project Page: https://mmsearch.github.io
paper_ai_summary: MMSearch-Engine enables multimodal search capabilities in Large Multimodal Models (LMMs) using a comprehensive benchmark and demonstrates improved performance over commercial products.
paper_ai_keywords: LLMs SearchGPT Large Multimodal Models LMMs multimodal search MMSearch-Engine MMSearch requery rerank summarization end-to-end task GPT-4o Perplexity Pro error analysis ablation study