Joint Multimodal Transformer for Emotion Recognition in the Wild
<|reference_start|>Joint Multimodal Transformer for Emotion Recognition in the Wild: Multimodal emotion recognition (MMER) systems typically outperform unimodal systems by leveraging the inter- and intra-modal relationships between, e.g., visual, textual, physiological, and auditory modalities. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention. This framework can exploit the complementary nature of diverse modalities to improve predictive accuracy. Separate backbones capture intra-modal spatiotemporal dependencies within each modality over video sequences. Subsequently, our JMT fusion architecture integrates the individual modality embeddings, allowing the model to effectively capture inter- and intra-modal relationships. Extensive experiments on two challenging expression recognition tasks -- (1) dimensional emotion recognition on the Affwild2 dataset (with face and voice) and (2) pain estimation on the Biovid dataset (with face and biosensors) -- indicate that our JMT fusion can provide a cost-effective solution for MMER. Empirical results show that MMER systems with our proposed fusion allow us to outperform relevant baseline and state-of-the-art methods.<|reference_end|>
@article{waligora2024joint,
    title={Joint Multimodal Transformer for Emotion Recognition in the Wild},
    author={Paul Waligora, Haseeb Aslam, Osama Zeeshan, Soufiane Belharbi,
  Alessandro Lameiras Koerich, Marco Pedersoli, Simon Bacon, Eric Granger},
    journal={arXiv preprint arXiv:2403.10488},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10488},
    primaryClass={cs.CV cs.LG cs.SD eess.AS}
}