RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement
<|reference_start|>RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement: In this paper, we aim to generate clean speech frame by frame from a live video stream and a noisy audio stream without relying on future inputs. To this end, we propose RT-LA-VocE, which completely re-designs every component of LA-VocE, a state-of-the-art non-causal audio-visual speech enhancement model, to perform causal real-time inference with a 40ms input frame. We do so by devising new visual and audio encoders that rely solely on past frames, replacing the Transformer encoder with the Emformer, and designing a new causal neural vocoder C-HiFi-GAN. On the popular AVSpeech dataset, we show that our algorithm achieves state-of-the-art results in all real-time scenarios. More importantly, each component is carefully tuned to minimize the algorithm latency to the theoretical minimum (40ms) while maintaining a low end-to-end processing latency of 28.15ms per frame, enabling real-time frame-by-frame enhancement with minimal delay.<|reference_end|>
@article{chen2024rt-la-voce:,
    title={RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement},
    author={Honglie Chen and Rodrigo Mira and Stavros Petridis and Maja Pantic},
    journal={arXiv preprint arXiv:2407.07825},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07825},
    primaryClass={cs.SD cs.CV cs.MM eess.AS}
}