Diffusion-based Unsupervised Audio-visual Speech Enhancement
<|reference_start|>Diffusion-based Unsupervised Audio-visual Speech Enhancement: This paper proposes a new unsupervised audiovisual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to iteratively estimate clean speech. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervisedgenerative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method.<|reference_end|>
@article{ayilo2024diffusion-based,
    title={Diffusion-based Unsupervised Audio-visual Speech Enhancement},
    author={Jean-Eudes Ayilo (MULTISPEECH), Mostafa Sadeghi (MULTISPEECH), Romain
  Serizel (MULTISPEECH), Xavier Alameda-Pineda (ROBOTLEARN)},
    journal={arXiv preprint arXiv:2410.05301},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05301},
    primaryClass={cs.SD cs.AI cs.CV cs.LG eess.AS eess.SP}
}