Sign Stitching: A Novel Approach to Sign Language Production
<|reference_start|>Sign Stitching: A Novel Approach to Sign Language Production: Sign Language Production (SLP) is a challenging task, given the limited resources available and the inherent diversity within sign data. As a result, previous works have suffered from the problem of regression to the mean, leading to under-articulated and incomprehensible signing. In this paper, we propose using dictionary examples and a learnt codebook of facial expressions to create expressive sign language sequences. However, simply concatenating signs and adding the face creates robotic and unnatural sequences. To address this we present a 7-step approach to effectively stitch sequences together. First, by normalizing each sign into a canonical pose, cropping, and stitching we create a continuous sequence. Then, by applying filtering in the frequency domain and resampling each sign, we create cohesive natural sequences that mimic the prosody found in the original data. We leverage a SignGAN model to map the output to a photo-realistic signer and present a complete Text-to-Sign (T2S) SLP pipeline. Our evaluation demonstrates the effectiveness of the approach, showcasing state-of-the-art performance across all datasets. Finally, a user evaluation shows our approach outperforms the baseline model and is capable of producing realistic sign language sequences.<|reference_end|>
@article{walsh2024sign,
    title={Sign Stitching: A Novel Approach to Sign Language Production},
    author={Harry Walsh, Ben Saunders, Richard Bowden},
    journal={arXiv preprint arXiv:2405.07663},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07663},
    primaryClass={cs.CV cs.CL}
}