LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition
<|reference_start|>LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition: Visual cues, like lip motion, have been shown to improve the performance of Automatic Speech Recognition (ASR) systems in noisy environments. We propose LipGER (Lip Motion aided Generative Error Correction), a novel framework for leveraging visual cues for noise-robust ASR. Instead of learning the cross-modal correlation between the audio and visual modalities, we make an LLM learn the task of visually-conditioned (generative) ASR error correction. Specifically, we instruct an LLM to predict the transcription from the N-best hypotheses generated using ASR beam-search. This is further conditioned on lip motions. This approach addresses key challenges in traditional AVSR learning, such as the lack of large-scale paired datasets and difficulties in adapting to new domains. We experiment on 4 datasets in various settings and show that LipGER improves the Word Error Rate in the range of 1.1%-49.2%. We also release LipHyp, a large-scale dataset with hypothesis-transcription pairs that is additionally equipped with lip motion cues to promote further research in this space<|reference_end|>
@article{ghosh2024lipger:,
    title={LipGER: Visually-Conditioned Generative Error Correction for Robust
  Automatic Speech Recognition},
    author={Sreyan Ghosh and Sonal Kumar and Ashish Seth and Purva Chiniya and
  Utkarsh Tyagi and Ramani Duraiswami and Dinesh Manocha},
    journal={arXiv preprint arXiv:2406.04432},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.04432},
    primaryClass={eess.AS cs.AI cs.CL}
}