Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
<|reference_start|>Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation: Audio-Visual Speech Recognition (AVSR) uses lip-based video to improve performance in noise. Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours. In contrast, speech models such as Whisper are trained with hundreds of thousands of hours of data, and thus learn a better speech-to-text decoder. The huge training data difference motivates us to adapt Whisper to handle video inputs. Inspired by Flamingo which injects visual features into language models, we propose Whisper-Flamingo which integrates visual features into the Whisper speech recognition and translation model with gated cross attention. Our audio-visual Whisper-Flamingo outperforms audio-only Whisper on English speech recognition and En-X translation for 6 languages in noisy conditions. Moreover, Whisper-Flamingo is a versatile model and conducts all of these tasks using one set of parameters, while prior methods are trained separately on each language.<|reference_end|>
@article{rouditchenko2024whisper-flamingo:,
    title={Whisper-Flamingo: Integrating Visual Features into Whisper for
  Audio-Visual Speech Recognition and Translation},
    author={Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde
  Kuehne, Rogerio Feris, James Glass},
    journal={arXiv preprint arXiv:2406.10082},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10082},
    primaryClass={eess.AS cs.CV cs.SD}
}