MMSearch: Benchmarking the Potential of Large Models as Multi-modal
  Search Engines
The advent of Large Language Models (LLMs) has paved the way for AI search
engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
interaction. However, most current AI search engines are limited to text-only
settings, neglecting the multimodal user queries and the text-image interleaved
nature of website information. Recently, Large Multimodal Models (LMMs) have
made impressive strides. Yet, whether they can function as AI search engines
remains under-explored, leaving the potential of LMMs in multimodal search an
open question. To this end, we first design a delicate pipeline,
MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
assess the multimodal search performance of LMMs. The curated dataset contains
300 manually collected instances spanning 14 subfields, which involves no
overlap with the current LMMs' training data, ensuring the correct answer can
only be obtained within searching. By using MMSearch-Engine, the LMMs are
evaluated by performing three individual tasks (requery, rerank, and
summarization), and one challenging end-to-end task with a complete searching
process. We conduct extensive experiments on closed-source and open-source
LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
results, which surpasses the commercial product, Perplexity Pro, in the
end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
further present error analysis to unveil current LMMs still struggle to fully
grasp the multimodal search tasks, and conduct ablation study to indicate the
potential of scaling test-time computation for AI search engine. We hope
MMSearch may provide unique insights to guide the future development of
multimodal AI search engine. Project Page: https://mmsearch.github.io
66ecef9de719fa9f4aaf3a56
Dongzhi Jiang
claimed_verified
2024-09-20T07:45:40.407Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
66ecef9de719fa9f4aaf3a57
Renrui Zhang
66ecef9de719fa9f4aaf3a58
Ziyu Guo
admin_assigned
2024-09-20T08:44:19.006Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
66ecef9de719fa9f4aaf3a59
Yanmin Wu
66ecef9de719fa9f4aaf3a5a
Jiayi Lei
admin_assigned
2024-09-20T08:44:43.491Z
64c3c72e8f31d1e6c664b052
/avatars/af1ad5048eaa9dc417837ad02f927911.svg
jiayi lei
user
jyjyjyjy
66ecef9de719fa9f4aaf3a5b
Pengshuo Qiu
66ecef9de719fa9f4aaf3a5c
Pan Lu
admin_assigned
2024-09-20T08:45:08.810Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
66ecef9de719fa9f4aaf3a5d
Zehui Chen
admin_assigned
2024-09-20T08:45:22.532Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
user
lovesnowbest
66ecef9de719fa9f4aaf3a5e
Guanglu Song
admin_assigned
2024-09-20T08:45:31.943Z
636a2efffbc31ee68fee5c1c
/avatars/21e119f23e1e6fe926567409e74d1a89.svg
Song
user
Guanglu
66ecef9de719fa9f4aaf3a5f
Peng Gao
66ecef9de719fa9f4aaf3a60
Yu Liu
admin_assigned
2024-09-20T08:46:13.519Z
636de85cc4a7a729c164d2b5
/avatars/3e281e547e1697e1c06805e7e63f3918.svg
Yu Liu
user
YuLiu
66ecef9de719fa9f4aaf3a61
Chunyuan Li
admin_assigned
2024-09-20T08:46:19.307Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
66ecef9de719fa9f4aaf3a62
Hongsheng Li
admin_assigned
2024-09-20T08:46:30.927Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
MMSearch: Benchmarking the Potential of Large Models as Multi-modal
  Search Engines
The advent of Large Language Models (LLMs) has paved the way for AI search
engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
interaction. However, most current AI search engines are limited to text-only
settings, neglecting the multimodal user queries and the text-image interleaved
nature of website information. Recently, Large Multimodal Models (LMMs) have
made impressive strides. Yet, whether they can function as AI search engines
remains under-explored, leaving the potential of LMMs in multimodal search an
open question. To this end, we first design a delicate pipeline,
MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
assess the multimodal search performance of LMMs. The curated dataset contains
300 manually collected instances spanning 14 subfields, which involves no
overlap with the current LMMs' training data, ensuring the correct answer can
only be obtained within searching. By using MMSearch-Engine, the LMMs are
evaluated by performing three individual tasks (requery, rerank, and
summarization), and one challenging end-to-end task with a complete searching
process. We conduct extensive experiments on closed-source and open-source
LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
results, which surpasses the commercial product, Perplexity Pro, in the
end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
further present error analysis to unveil current LMMs still struggle to fully
grasp the multimodal search tasks, and conduct ablation study to indicate the
potential of scaling test-time computation for AI search engine. We hope
MMSearch may provide unique insights to guide the future development of
multimodal AI search engine. Project Page: https://mmsearch.github.io
MMSearch-Engine enables multimodal search capabilities in Large Multimodal Models (LMMs) using a comprehensive benchmark and demonstrates improved performance over commercial products.
LLMs SearchGPT Large Multimodal Models LMMs multimodal search MMSearch-Engine MMSearch requery rerank summarization end-to-end task GPT-4o Perplexity Pro error analysis ablation study