Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge
<|reference_start|>Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge: Speech Self-Supervised Learning (SSL) has demonstrated considerable efficacy in various downstream tasks. Nevertheless, prevailing self-supervised models often overlook the incorporation of emotion-related prior information, thereby neglecting the potential enhancement of emotion task comprehension through emotion prior knowledge in speech. In this paper, we propose an emotion-aware speech representation learning with intensity knowledge. Specifically, we extract frame-level emotion intensities using an established speech-emotion understanding model. Subsequently, we propose a novel emotional masking strategy (EMS) to incorporate emotion intensities into the masking process. We selected two representative models based on Transformer and CNN, namely MockingJay and Non-autoregressive Predictive Coding (NPC), and conducted experiments on IEMOCAP dataset. Experiments have demonstrated that the representations derived from our proposed method outperform the original model in SER task.<|reference_end|>
@article{liu2024emotion-aware,
    title={Emotion-Aware Speech Self-Supervised Representation Learning with
  Intensity Knowledge},
    author={Rui Liu, Zening Ma},
    journal={arXiv preprint arXiv:2406.06646},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06646},
    primaryClass={eess.AS cs.SD}
}