Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers
<|reference_start|>Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers: Due to the complex nature of human emotions and the diversity of emotion representation methods in humans, emotion recognition is a challenging field. In this research, three input modalities, namely text, audio (speech), and video, are employed to generate multimodal feature vectors. For generating features for each of these modalities, pre-trained Transformer models with fine-tuning are utilized. In each modality, a Transformer model is used with transfer learning to extract feature and emotional structure. These features are then fused together, and emotion recognition is performed using a classifier. To select an appropriate fusion method and classifier, various feature-level and decision-level fusion techniques have been experimented with, and ultimately, the best model, which combines feature-level fusion by concatenating feature vectors and classification using a Support Vector Machine on the IEMOCAP multimodal dataset, achieves an accuracy of 75.42%. Keywords: Multimodal Emotion Recognition, IEMOCAP, Self-Supervised Learning, Transfer Learning, Transformer.<|reference_end|>
@article{shayaninasab2024multi-modal,
    title={Multi-Modal Emotion Recognition by Text, Speech and Video Using
  Pretrained Transformers},
    author={Minoo Shayaninasab, Bagher Babaali},
    journal={arXiv preprint arXiv:2402.07327},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07327},
    primaryClass={cs.AI}
}