Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer
<|reference_start|>Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer: Humans are adept at leveraging visual cues from lip movements for recognizing speech in adverse listening conditions. Audio-Visual Speech Recognition (AVSR) models follow similar approach to achieve robust speech recognition in noisy conditions. In this work, we present a multilingual AVSR model incorporating several enhancements to improve performance and audio noise robustness. Notably, we adapt the recently proposed Fast Conformer model to process both audio and visual modalities using a novel hybrid CTC/RNN-T architecture. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets (VoxCeleb2 and AVSpeech). Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%. On the recently introduced MuAViC benchmark, our model yields an absolute average-WER reduction of 11.9% in comparison to the original baseline. Finally, we demonstrate the ability of the proposed model to perform audio-only, visual-only, and audio-visual speech recognition at test time.<|reference_end|>
@article{burchi2024multilingual,
    title={Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast
  Conformer},
    author={Maxime Burchi, Krishna C. Puvvada, Jagadeesh Balam, Boris Ginsburg,
  Radu Timofte},
    journal={arXiv preprint arXiv:2405.12983},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.12983},
    primaryClass={eess.AS cs.AI cs.CV cs.MM cs.SD}
}