Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment
<|reference_start|>Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment: Speech emotion recognition (SER) systems often struggle in real-world environments, where ambient noise severely degrades their performance. This paper explores a novel approach that exploits prior knowledge of testing environments to maximize SER performance under noisy conditions. To address this task, we propose a text-guided, environment-aware training where an SER model is trained with contaminated speech samples and their paired noise description. We use a pre-trained text encoder to extract the text-based environment embedding and then fuse it to a transformer-based SER model during training and inference. We demonstrate the effectiveness of our approach through our experiment with the MSP-Podcast corpus and real-world additive noise samples collected from the Freesound repository. Our experiment indicates that the text-based environment descriptions processed by a large language model (LLM) produce representations that improve the noise-robustness of the SER system. In addition, our proposed approach with an LLM yields better performance than our environment-agnostic baselines, especially in low signal-to-noise ratio (SNR) conditions. When testing at -5dB SNR level, our proposed method shows better performance than our best baseline model by 31.8 % (arousal), 23.5% (dominance), and 9.5% (valence).<|reference_end|>
@article{leem2024describe,
    title={Describe Where You Are: Improving Noise-Robustness for Speech Emotion
  Recognition with Text Description of the Environment},
    author={Seong-Gyun Leem, Daniel Fulford, Jukka-Pekka Onnela, David Gard, and
  Carlos Busso},
    journal={arXiv preprint arXiv:2407.17716},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.17716},
    primaryClass={cs.SD cs.CL cs.LG eess.AS}
}