  Recent voice assistants are usually based on the cascade spoken language
understanding (SLU) solution, which consists of an automatic speech recognition
(ASR) engine and a natural language understanding (NLU) system. Because such
approach relies on the ASR output, it often suffers from the so-called ASR
error propagation. In this work, we investigate impacts of this ASR error
propagation on state-of-the-art NLU systems based on pre-trained language
models (PLM), such as BERT and RoBERTa. Moreover, a multimodal language
understanding (MLU) module is proposed to mitigate SLU performance degradation
caused by errors present in the ASR transcript. The MLU benefits from
self-supervised features learned from both audio and text modalities,
specifically Wav2Vec for speech and Bert/RoBERTa for language. Our MLU combines
an encoder network to embed the audio signal and a text encoder to process text
transcripts followed by a late fusion layer to fuse audio and text logits. We
found that the proposed MLU showed to be robust towards poor quality ASR
transcripts, while the performance of BERT and RoBERTa are severely
compromised. Our model is evaluated on five tasks from three SLU datasets and
robustness is tested using ASR transcripts from three ASR engines. Results show
that the proposed approach effectively mitigates the ASR error propagation
problem, surpassing the PLM models' performance across all datasets for the
academic ASR engine.

Computation and Language;Machine Learning;Audio and Speech Processing