Qilin: A Multimodal Information Retrieval Dataset with APP-level User
  Sessions
User-generated content (UGC) communities, especially those featuring
multimodal content, improve user experiences by integrating visual and textual
information into results (or items). The challenge of improving user
experiences in complex systems with search and recommendation (S\&R) services
has drawn significant attention from both academia and industry these years.
However, the lack of high-quality datasets has limited the research progress on
multimodal S\&R. To address the growing need for developing better S\&R
services, we present a novel multimodal information retrieval dataset in this
paper, namely Qilin. The dataset is collected from Xiaohongshu, a popular
social platform with over 300 million monthly active users and an average
search penetration rate of over 70\%. In contrast to existing datasets,
Qilin offers a comprehensive collection of user sessions with
heterogeneous results like image-text notes, video notes, commercial notes, and
direct answers, facilitating the development of advanced multimodal neural
retrieval models across diverse task settings. To better model user
satisfaction and support the analysis of heterogeneous user behaviors, we also
collect extensive APP-level contextual signals and genuine user feedback.
Notably, Qilin contains user-favored answers and their referred results for
search requests triggering the Deep Query Answering (DQA) module. This allows
not only the training \& evaluation of a Retrieval-augmented Generation (RAG)
pipeline, but also the exploration of how such a module would affect users'
search behavior. Through comprehensive analysis and experiments, we provide
interesting findings and insights for further improving S\&R systems. We hope
that Qilin will significantly contribute to the advancement of
multimodal content platforms with S\&R services in the future.
67c6a343ad6b7c2fa29d5e7e
Jia Chen
claimed_verified
2025-03-04T16:08:10.744Z
67c03221aed8409476d39da8
https://cdn-avatars.huggingface.co/v1/production/uploads/67c03221aed8409476d39da8/eQIhOPRLNoiphsR145mfB.png
Jia Chen
user
Regulus309
67c6a343ad6b7c2fa29d5e7f
Qian Dong
claimed_verified
2025-03-04T08:34:51.762Z
60c0ed29d8bc072769d78f48
https://cdn-avatars.huggingface.co/v1/production/uploads/60c0ed29d8bc072769d78f48/V6q6Tn4kzB46NIbTYw9pQ.jpeg
Qian Dong
user
qian
67c6a343ad6b7c2fa29d5e80
Haitao Li
admin_assigned
2025-03-04T10:20:57.898Z
67b5d91558369f6b38c5b596
/avatars/18b08d5d9b05786cad34bc000c7606aa.svg
Haitao Li
user
haitaoli
67c6a343ad6b7c2fa29d5e81
Xiaohui He
67c6a343ad6b7c2fa29d5e82
Yan Gao
67c6a343ad6b7c2fa29d5e83
Shaosheng Cao
67c6a343ad6b7c2fa29d5e84
Yi Wu
67c6a343ad6b7c2fa29d5e85
Ping Yang
67c6a343ad6b7c2fa29d5e86
Chen Xu
67c6a343ad6b7c2fa29d5e87
Yao Hu
67c6a343ad6b7c2fa29d5e88
Qingyao Ai
admin_assigned
2025-03-04T10:21:22.100Z
6657e7045f6e35c7d541bdd8
/avatars/368e5cef6c93543b2b92fbca79a4e4b9.svg
Qingyao Ai
user
aiqy
67c6a343ad6b7c2fa29d5e89
Yiqun Liu
Qilin: A Multimodal Information Retrieval Dataset with APP-level User
  Sessions
User-generated content (UGC) communities, especially those featuring
multimodal content, improve user experiences by integrating visual and textual
information into results (or items). The challenge of improving user
experiences in complex systems with search and recommendation (S\&R) services
has drawn significant attention from both academia and industry these years.
However, the lack of high-quality datasets has limited the research progress on
multimodal S\&R. To address the growing need for developing better S\&R
services, we present a novel multimodal information retrieval dataset in this
paper, namely Qilin. The dataset is collected from Xiaohongshu, a popular
social platform with over 300 million monthly active users and an average
search penetration rate of over 70\%. In contrast to existing datasets,
Qilin offers a comprehensive collection of user sessions with
heterogeneous results like image-text notes, video notes, commercial notes, and
direct answers, facilitating the development of advanced multimodal neural
retrieval models across diverse task settings. To better model user
satisfaction and support the analysis of heterogeneous user behaviors, we also
collect extensive APP-level contextual signals and genuine user feedback.
Notably, Qilin contains user-favored answers and their referred results for
search requests triggering the Deep Query Answering (DQA) module. This allows
not only the training \& evaluation of a Retrieval-augmented Generation (RAG)
pipeline, but also the exploration of how such a module would affect users'
search behavior. Through comprehensive analysis and experiments, we provide
interesting findings and insights for further improving S\&R systems. We hope
that Qilin will significantly contribute to the advancement of
multimodal content platforms with S\&R services in the future.
A new multimodal dataset named Qilin, sourced from Xiaohongshu, supports the development of advanced retrieval models and examines the impact of deep query answering on user behavior.
multimodal information retrieval Deep Query Answering Retrieval-augmented Generation multimodal neural retrieval models