MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
<|reference_start|>MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition: This paper presents a novel approach to processing multimodal data for dynamic emotion recognition, named as the Multimodal Masked Autoencoder for Dynamic Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the closely correlated representation information within spatiotemporal sequences across visual and audio modalities. By utilizing a pre-trained masked autoencoder model, the MultiMAEDER is accomplished through simple, straightforward finetuning. The performance of the MultiMAE-DER is enhanced by optimizing six fusion strategies for multimodal input sequences. These strategies address dynamic feature correlations within cross-domain data across spatial, temporal, and spatiotemporal sequences. In comparison to state-of-the-art multimodal supervised learning models for dynamic emotion recognition, MultiMAE-DER enhances the weighted average recall (WAR) by 4.41% on the RAVDESS dataset and by 2.06% on the CREMAD. Furthermore, when compared with the state-of-the-art model of multimodal self-supervised learning, MultiMAE-DER achieves a 1.86% higher WAR on the IEMOCAP dataset.<|reference_end|>
@article{xiang2024multimae-der:,
    title={MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion
  Recognition},
    author={Peihao Xiang, Chaohao Lin, Kaida Wu, Ou Bai},
    journal={arXiv preprint arXiv:2404.18327},
    year={2024},
    doi={10.1109/ICPRS62101.2024.10677820},
    archivePrefix={arXiv},
    eprint={2404.18327},
    primaryClass={cs.CV}
}