MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production
<|reference_start|>MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production: Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.<|reference_end|>
@article{ma2024ms2sl:,
    title={MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production},
    author={Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng},
    journal={arXiv preprint arXiv:2407.12842},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12842},
    primaryClass={cs.CL cs.AI}
}