Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples
<|reference_start|>Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples: The Multimodal Emotion Recognition challenge MER2024 focuses on recognizing emotions using audio, language, and visual signals. In this paper, we present our submission solutions for the Semi-Supervised Learning Sub-Challenge (MER2024-SEMI), which tackles the issue of limited annotated data in emotion recognition. Firstly, to address the class imbalance, we adopt an oversampling strategy. Secondly, we propose a modality representation combinatorial contrastive learning (MR-CCL) framework on the trimodal input data to establish robust initial models. Thirdly, we explore a self-training approach to expand the training set. Finally, we enhance prediction robustness through a multi-classifier weighted soft voting strategy. Our proposed method is validated to be effective on the MER2024-SEMI Challenge, achieving a weighted average F-score of 88.25% and ranking 6th on the leaderboard. Our project is available at https://github.com/WooyoohL/MER2024-SEMI.<|reference_end|>
@article{fan2024leveraging,
    title={Leveraging Contrastive Learning and Self-Training for Multimodal Emotion
  Recognition with Limited Labeled Samples},
    author={Qi Fan, Yutong Li, Yi Xin, Xinyu Cheng, Guanglai Gao, Miao Ma},
    journal={arXiv preprint arXiv:2409.04447},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.04447},
    primaryClass={cs.SD cs.AI eess.AS}
}