Turbo your multi-modal classification with contrastive learning
<|reference_start|>Turbo your multi-modal classification with contrastive learning: Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.<|reference_end|>
@article{zhang2024turbo,
    title={Turbo your multi-modal classification with contrastive learning},
    author={Zhiyu Zhang and Da Liu and Shengqiang Liu and Anna Wang and Jie Gao
  and Yali Li},
    journal={arXiv preprint arXiv:2409.09282},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09282},
    primaryClass={cs.LG cs.MM}
}