Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach
<|reference_start|>Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach: The process of reconstructing missing parts of speech audio from context is called speech in-painting. Human perception of speech is inherently multi-modal, involving both audio and visual (AV) cues. In this paper, we introduce and study a sequence-to-sequence (seq2seq) speech in-painting model that incorporates AV features. Our approach extends AV speech in-painting techniques to scenarios where both audio and visual data may be jointly corrupted. To achieve this, we employ a multi-modal training paradigm that boosts the robustness of our model across various conditions involving acoustic and visual distortions. This makes our distortion-aware model a plausible solution for real-world challenging environments. We compare our method with existing transformer-based and recurrent neural network-based models, which attempt to reconstruct missing speech gaps ranging from a few milliseconds to over a second. Our experimental results demonstrate that our novel seq2seq architecture outperforms the state-of-the-art transformer solution by 38.8% in terms of enhancing speech quality and 7.14% in terms of improving speech intelligibility. We exploit a multi-task learning framework that simultaneously performs lip-reading (transcribing video components to text) while reconstructing missing parts of the associated speech.<|reference_end|>
@article{elyaderani2024robust,
    title={Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach},
    author={Mahsa Kadkhodaei Elyaderani, and Shahram Shirani},
    journal={arXiv preprint arXiv:2406.00901},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.00901},
    primaryClass={cs.MM cs.AI cs.LG cs.SD eess.AS}
}