Improving Continuous Sign Language Recognition with Adapted Image Models
<|reference_start|>Improving Continuous Sign Language Recognition with Adapted Image Models: The increase of web-scale weakly labelled image-text pairs have greatly facilitated the development of large-scale vision-language models (e.g., CLIP), which have shown impressive generalization performance over a series of downstream tasks. However, the massive model size and scarcity of available data limit their applications to fine-tune the whole model in downstream tasks. Besides, fully fine-tuning the model easily forgets the generic essential knowledge acquired in the pretraining stage and overfits the downstream data. To enable high efficiency when adapting these large vision-language models (e.g., CLIP) to performing continuous sign language recognition (CSLR) while preserving their generalizability, we propose a novel strategy (AdaptSign). Especially, CLIP is adopted as the visual backbone to extract frame-wise features whose parameters are fixed, and a set of learnable modules are introduced to model spatial sign variations or capture temporal sign movements. The introduced additional modules are quite lightweight, only owning 3.2% extra computations with high efficiency. The generic knowledge acquired in the pretraining stage is well-preserved in the frozen CLIP backbone in this process. Extensive experiments show that despite being efficient, AdaptSign is able to demonstrate superior performance across a series of CSLR benchmarks including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL compared to existing methods. Visualizations show that AdaptSign could learn to dynamically pay major attention to the informative spatial regions and cross-frame trajectories in sign videos.<|reference_end|>
@article{hu2024improving,
    title={Improving Continuous Sign Language Recognition with Adapted Image Models},
    author={Lianyu Hu, Tongkai Shi, Liqing Gao, Zekang Liu, Wei Feng},
    journal={arXiv preprint arXiv:2404.08226},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.08226},
    primaryClass={cs.CV}
}