TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions
<|reference_start|>TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions: A key challenge in continuous sign language recognition (CSLR) is to efficiently capture long-range spatial interactions over time from the video input. To address this challenge, we propose TCNet, a hybrid network that effectively models spatio-temporal information from Trajectories and Correlated regions. TCNet's trajectory module transforms frames into aligned trajectories composed of continuous visual tokens. In addition, for a query token, self-attention is learned along the trajectory. As such, our network can also focus on fine-grained spatio-temporal patterns, such as finger movements, of a specific region in motion. TCNet's correlation module uses a novel dynamic attention mechanism that filters out irrelevant frame regions. Additionally, it assigns dynamic key-value tokens from correlated regions to each query. Both innovations significantly reduce the computation cost and memory. We perform experiments on four large-scale datasets: PHOENIX14, PHOENIX14-T, CSL, and CSL-Daily, respectively. Our results demonstrate that TCNet consistently achieves state-of-the-art performance. For example, we improve over the previous state-of-the-art by 1.5% and 1.0% word error rate on PHOENIX14 and PHOENIX14-T, respectively.<|reference_end|>
@article{lu2024tcnet:,
    title={TCNet: Continuous Sign Language Recognition from Trajectories and
  Correlated Regions},
    author={Hui Lu, Albert Ali Salah, Ronald Poppe},
    journal={arXiv preprint arXiv:2403.11818},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11818},
    primaryClass={cs.CV}
}