Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendation
<|reference_start|>Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendation: Multi-modal recommendation greatly enhances the performance of recommender systems by modeling the auxiliary information from multi-modality contents. Most existing multi-modal recommendation models primarily exploit multimedia information propagation processes to enrich item representations and directly utilize modal-specific embedding vectors independently obtained from upstream pre-trained models. However, this might be inappropriate since the abundant task-specific semantics remain unexplored, and the cross-modality semantic gap hinders the recommendation performance.   Inspired by the recent progress of the cross-modal alignment model CLIP, in this paper, we propose a novel \textbf{CLIP} \textbf{E}nhanced \textbf{R}ecommender (\textbf{CLIPER}) framework to bridge the semantic gap between modalities and extract fine-grained multi-view semantic information. Specifically, we introduce a multi-view modality-alignment approach for representation extraction and measure the semantic similarity between modalities. Furthermore, we integrate the multi-view multimedia representations into downstream recommendation models. Extensive experiments conducted on three public datasets demonstrate the consistent superiority of our model over state-of-the-art multi-modal recommendation models.<|reference_end|>
@article{wu2024towards,
    title={Towards Bridging the Cross-modal Semantic Gap for Multi-modal
  Recommendation},
    author={Xinglong Wu, Anfeng Huang, Hongwei Yang, Hui He, Yu Tai, Weizhe Zhang},
    journal={arXiv preprint arXiv:2407.05420},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.05420},
    primaryClass={cs.IR}
}