Sequence-to-Sequence Multi-Modal Speech In-Painting
<|reference_start|>Sequence-to-Sequence Multi-Modal Speech In-Painting: Speech in-painting is the task of regenerating missing audio contents using reliable context information. Despite various recent studies in multi-modal perception of audio in-painting, there is still a need for an effective infusion of visual and auditory information in speech in-painting. In this paper, we introduce a novel sequence-to-sequence model that leverages the visual information to in-paint audio signals via an encoder-decoder architecture. The encoder plays the role of a lip-reader for facial recordings and the decoder takes both encoder outputs as well as the distorted audio spectrograms to restore the original speech. Our model outperforms an audio-only speech in-painting model and has comparable results with a recent multi-modal speech in-painter in terms of speech quality and intelligibility metrics for distortions of 300 ms to 1500 ms duration, which proves the effectiveness of the introduced multi-modality in speech in-painting.<|reference_end|>
@article{elyaderani2024sequence-to-sequence,
    title={Sequence-to-Sequence Multi-Modal Speech In-Painting},
    author={Mahsa Kadkhodaei Elyaderani, Shahram Shirani},
    journal={arXiv preprint arXiv:2406.01321},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01321},
    primaryClass={cs.SD cs.AI cs.LG cs.MM eess.AS}
}