  In this paper, we propose P3D, the human part-wise motion context learning
framework for sign language recognition. Our main contributions lie in two
dimensions: learning the part-wise motion context and employing the pose
ensemble to utilize 2D and 3D pose jointly. First, our empirical observation
implies that part-wise context encoding benefits the performance of sign
language recognition. While previous methods of sign language recognition
learned motion context from the sequence of the entire pose, we argue that such
methods cannot exploit part-specific motion context. In order to utilize
part-wise motion context, we propose the alternating combination of a part-wise
encoding Transformer (PET) and a whole-body encoding Transformer (WET). PET
encodes the motion contexts from a part sequence, while WET merges them into a
unified context. By learning part-wise motion context, our P3D achieves
superior performance on WLASL compared to previous state-of-the-art methods.
Second, our framework is the first to ensemble 2D and 3D poses for sign
language recognition. Since the 3D pose holds rich motion context and depth
information to distinguish the words, our P3D outperformed the previous
state-of-the-art methods employing a pose ensemble.

Computer Vision and Pattern Recognition