Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment
<|reference_start|>Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment: This paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance emotion recognition accuracy in challenging acoustic conditions. Our approach combines a modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-channel audio processing with an R(2+1)D Convolutional Neural Networks (CNN) model for video analysis. We evaluate our proposed method on a reverberated version of the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using synthetic and real-world Room Impulse Responsess (RIRs). Our results demonstrate that integrating audio and video modalities yields superior performance compared to uni-modal approaches, especially in challenging acoustic conditions. Moreover, we show that the multimodal (audiovisual) approach that utilizes multiple microphones outperforms its single-microphone counterpart.<|reference_end|>
@article{cohen2024multi-microphone,
    title={Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant
  Environment},
    author={Ohad Cohen, Gershon Hazan, Sharon Gannot},
    journal={arXiv preprint arXiv:2409.09545},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09545},
    primaryClass={cs.SD cs.LG eess.AS}
}