Improving Gloss-free Sign Language Translation by Reducing Representation Density
<|reference_start|>Improving Gloss-free Sign Language Translation by Reducing Representation Density: Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCL achieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at https://github.com/JinhuiYE/SignCL.<|reference_end|>
@article{ye2024improving,
    title={Improving Gloss-free Sign Language Translation by Reducing
  Representation Density},
    author={Jinhui Ye, Xing Wang, Wenxiang Jiao, Junwei Liang, Hui Xiong},
    journal={arXiv preprint arXiv:2405.14312},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14312},
    primaryClass={cs.CV cs.CL cs.MM}
}