Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding
<|reference_start|>Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding: Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence. The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning. In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA). To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM). Specifically, we design three experts of soft prompts: a text prompt and an image prompt that extract modality-specific features to enrich the single-modal representation, and a unified prompt to assist multi-modal interaction. Additionally, we reorganize Transformer layers into several blocks and introduce cross-modal prompt attention between adjacent blocks, which smoothens the transition from single-modal representation to multi-modal fusion. On both MSD and MSA datasets in few-shot setting, our proposed model not only surpasses the 8.2B model InstructBLIP with merely 2% parameters (150M), but also significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.<|reference_end|>
@article{wu2024mixture-of-prompt-experts,
    title={Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding},
    author={Zichen Wu, Hsiu-Yuan Huang, Fanyi Qu and Yunfang Wu},
    journal={arXiv preprint arXiv:2403.11311},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11311},
    primaryClass={cs.CL cs.MM}
}