  The research and applications of multimodal emotion recognition have become
increasingly popular recently. However, multimodal emotion recognition faces
the challenge of lack of data. To solve this problem, we propose to use
transfer learning which leverages state-of-the-art pre-trained models including
wav2vec 2.0 and BERT for this task. Multi-level fusion approaches including
coattention-based early fusion and late fusion with the models trained on both
embeddings are explored. Also, a multi-granularity framework which extracts not
only frame-level speech embeddings but also segment-level embeddings including
phone, syllable and word-level speech embeddings is proposed to further boost
the performance. By combining our coattention-based early fusion model and late
fusion model with the multi-granularity feature extraction framework, we obtain
result that outperforms best baseline approaches by 1.3% unweighted accuracy
(UA) on the IEMOCAP dataset.

Computation and Language;Sound;Audio and Speech Processing