Abstract AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sentences. We introduce OpenHands 1 , a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition. First, we propose using pose extracted through pretrained models as the standard modality of data in this work to reduce training time and enable efficient inference, and we release standardized pose datasets for different existing sign language datasets. Second, we train and release checkpoints of 4 pose-based isolated sign language recognition models across 6 languages (American, Argentinian, Chinese, Greek, Indian, and Turkish), providing baselines and ready checkpoints for deployment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unlabelled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare different pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslingual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages reproducible and more accessible. According to the World Federation of the Deaf, there are approximately 72 million Deaf people worldwide. More than 80% of them live in developing countries. Collectively, they use more than  Equal contribution. 300 different sign languages varying across different nations (UN, 2021). Loss of hearing severely limits the ability of the Deaf to communicate and thereby adversely impacts their quality of life. In the current increasingly digital world, systems to ease digital communication between Deaf and hearing people are important accessibility aids. AI has a crucial role to play in enabling this accessibility with automated tools for Sign Language Recognition (SLR). Specifically, transcription of sign language as complete sentences is referred to as Continuous Sign Language Recognition (CSLR), while recognition of individual signs is referred to as Isolated Sign Language Recognition (ISLR). There have been various efforts to build datasets and models for ISLR and CLSR tasks (Adaloglou et al., 2021; Koller, 2020). But these results are often concentrated on a few sign languages (such as the American Sign Language) and are reported across different research communities with few standardized baselines. When compared against textand speech-based NLP research, the progress in AI research for sign languages is significantly lagging. This lag has been recently brought to notice of the wider NLP community (Yin et al., 2021). For most sign languages across the world, the amount of labelled data is very low and hence they can be considered low-resource languages . In the NLP literature, many successful templates have been proposed for such low-resource languages. In this work, we adopt and combine many of these ideas from NLP to sign language research. We implement these ideas and release several datasets and models in an open-source library OpenHands with the following key contributions: 1. Standardizing on pose as the modality: We consider using pose-extractor as an encoder, which processes raw RGB videos and extracts the framewise coordinates for few keypoints. Pose-extractors are useful across sign languages and also other 2114 tasks such as action recognition (Yan et al., 2018; Liu et al., 2020), and can be trained to high accuracy. Further, as we report, pose as a modality makes both training and inference for SLR tasks efficient. We release pose-based versions of existing datasets for 5 sign languages: American, Argentinian, Greek, Indian, and Turkish. 2. Standardized comparison of models across languages: The progress in NLP has been earmarked by the release of standard datasets, including multilingual datasets like XGLUE (Liang et al., 2020), on which various models are compared. As a step towards such standardization for ISLR, we train 4 different models spanning sequence models (LSTM and Transformer) and graph-based models (ST-GCN and SL-GCN) on 7 different datasets across 6 sign languages mentioned in Table 1, and compare them against models proposed in the literature. We release all 28 trained models along with scripts for efficient deployment which demonstrably achieve real-time performance on CPUs and GPUs. 3. Corpus for self-supervised training: A defining success in NLP has been the use of self-supervised training, for instance masked-language modelling (Devlin et al., 2018), on large corpora of natural language text. To apply this idea to SLR, we need similarly large corpora of sign language data. To this end, we curate 1,129 hours of video data on Indian Sign Language. We pre-process these videos with a custom pipeline and extract keypoints for all frames. We release this corpus which is the first such large-scale sign language corpus for self-supervised training. 4. Effectiveness of self-supervised training: Self-supervised training has been demonstrated to be effective for NLP: Pretrained models require small amounts of fine-tuning data (Devlin et al., 2018; Baevski et al., 2020) and multilingual pretraining allows crosslingual generalization (Hu et al., 2020b). To apply this for SLR, we evaluate multiple strategies for self-supervised pretraining of ISLR models and identify those that are effective. With the identified pretraining strategies, we demonstrate the significance of pretraining by showing improved fine-tuning performance, especially in very low-resource settings and also show high crosslingual transfer from Indian SL to other sign languages. This is the first and successful attempt that establishes the effectiveness of self-supervised learning in SLR. We release the pretrained model and the fine-tuned models for 4 different sign languages. Through these datasets, models, and experiments we make several observations. First, in comparing standardized models across different sign languages, we find that graph-based models working on pose modality define state-of-the-art results on most sign languages. RNN-based models lag on accuracy but are significantly faster and thus appropriate for constrained devices. Second, we establish that self-supervised pretraining helps as it improves on equivalent models trained from scratch on labelled ISLR data. The performance gap is particularly high if the labelled data contains fewer samples per label, i.e., for the many sign languages which have limited resources the value of self-supervised pretraining is particularly high. Third, we establish that self-supervision in one sign language (Indian SL) can be crosslingually transferred to improve SLR on other sign languages (Amer-ican, Chinese, and Argentinian). This is particularly encouraging for the long tail of over 300 sign languages that are used across the globe. Fourth, we establish that for real-time applications, pose-based modality is preferable over other modalities such as RGB, use of depth sensors, etc. due to reduced infrastructure requirements (only camera), and higher efficiency in self-supervised pretraining, fine-tuning on ISLR, and inference. We believe such standardization can help accelerate dataset collection and model benchmarking. Fifth, we observe that the trained checkpoints of the pose-based models can be directly integrated with pose estimation models to create a pipeline that can provide real-time inference even on CPUs. Such a pipeline can enable the deployment of these models in realtime video conferencing tools, perhaps even on smartphones. As mentioned all datasets and models are released with permissible licenses in OpenHands with the intention to make SLR research more accessible and standardized. We hope that others contribute datasets and models to the library, especially representing the diversity of sign languages used across the globe. The rest of the paper is organized as follows. In section 2 we present a brief overview of the existing work. In section 3 we describe our efforts in standardizing datasets and models across six different sign languages. In section 4 we explain our pretraining corpus and strategies for self-2115 supervised learning and detail results that establish its effectiveness. In section 5 we describe in brief the functionalities of the OpenHands library. In section 6, we summarize our work and also list potential follow-up work. Significant progress has been made in Isolated Sign Language Recognition (ISLR) due to the release of datasets (Li et al., 2020; Sincan and Keles, 2020; Chai et al., 2014; Huang et al., 2019) and recent deep learning architectures (Adaloglou et al., 2021). This section reviews this work, with a focus on pose-based models. A sign language (SL) is the visual language used by the Deaf and hard-of-hearing (DHH) individuals (and also by those who comunnicate with them), which involves usage of various bodily actions, like hand gestures and facial expressions, called signs to communicate. A sequence of signs constitutes a phrase or sentence in a SL. The signs can be transcribed into sign-words of any specific spoken language usually written completely in capital letters. Each such sign-word is technically called as a gloss and is the standardized basic atomic token of an SL transcript (Schembri and Crasborn, 2010). It is be noted that there is not (always) one-to-one relationships between glosses and spoken language words. The task of converting each visual sign communicated by a signer into a gloss is called isolated sign language recognition (ISLR). The task of converting a continuous sequence of visual signs into serialized glosses is referred as continuous sign language recognition (CSLR). CSLR can either be modeled as an end-to-end task, or as a combination of sign language segmentation and ISLR. The task of converting signs into spoken language text is referred as sign language translation (SLT), which can again either be end-to-end or a combination of CLSR and gloss-sequence to spoken phrase converter. In terms of real-world applications, eventhough CSLR is more practically useful than ISLR, it does not still undermine the value in studying and implementing ISLR. The applications of ISLR include building sign spotting systems (Albanie et al., 2020), building alignment networks (Albanie et al., 2021) to aid in building CSLR datasets (or evaluate CSLR output), building CSLR systems on top of an automatic SL segmentation model (Farag and Brock, 2019) which identifies the frame boundaries for signs in videos to divide them into approximate meaningful segments (glosses), etc. Although SL content is predominantly recorded as RGB (color) videos, it can also be captured using various other modalities like depth maps or point cloud, finger gestures recorded using sensors, skeleton representation of the signer, etc. In this work, we focus on ISLR using pose-skeleton modality. A pose representation, extracted using pose estimation models, provides the spatial coordinates at which the joints (such as elbows and knees), called keypoints, are located in a field or video. This pose information can be represented as a connected graph with nodes representing keypoints and edges may be constructed across nodes to approximately represent the human skeleton. Initial methods for SLR focused on hand gestures from either video frames (Reshna et al., 2020) or sensor data such as from smart gloves (Fels and Hinton, 1993). Given that such sensors are not commonplace and that body posture and face expressions are also of non-trivial importance for understanding signs (Hu et al., 2020a), convolutional network based models have been used for SLR (Rao et al., 2018). ISLR can be considered as a multiclass classification task and generally accuracy metric is used the to evaluate the performance of the models. The ISLR task is related to the more widely studied action recognition task (Zhu et al., 2020). Like in action recognition task, highly accurate pose recognition models like OpenPose (Cao et al., 2018) and MediaPipe Holistic (Grishchenko and Bazarevsky, 2020) are being used for ISLR models (Li et al., 2020; Ko et al., 2018), where frame-wise keypoints are the inputs. Although RGB-based models may outperform pose-based models (Li et al., 2020) narrowly, pose-based models have far fewer parameters and are more efficient for deployment if used with very-fast pose estimation pipelines like MediaPipe BlazePose. In this work, we focus on lightweight pose-based ISLR which encode the pose frames and classify the pose using specific decoders. We briefly discuss the two broad types of such models: sequence-based and graph-based. tially along time either on one or both directions. Initially, RNNs were used for pose-based action recognition to learn from temporal features (Du et al., 2015; Zhang et al., 2017; Si et al., 2018). Specifically, sequence of pose frames are input to GRU or LSTM layers, and the output from the final timestep is used for classification. Transformer architectures with encoder-only models like BERT (Vaswani et al., 2017) have also been studied for pose-based ISLR models (De Coster et al., 2020). The input is a sequence of pose frames along with positional embeddings. A special [CLS] token is prepended to the sequence, whose final embedding is used for classification. Graph convolution networks (Kipf and Welling, 2017), which are good at modeling graph data have been used for skeleton action recognition to achieve state-of-the-art results, by considering human skeleton sequences as spatio-temporal graphs (Cheng et al., 2020a; Liu et al., 2020). Spatial-Temporal GCN (ST-GCN) uses human body joint connections for spatial connections and temporal connections across frames to construct a 3d graph, which is processed by a combination of spatial graph convolutions and temporal convolutions to efficiently model the spatio-temporal data (Lin et al., 2020). Many architectural improvements have been proposed over ST-GCN for skeleton action recognition (Zhang et al., 2020; Shi et al., 2019b,a; Cheng et al., 2020b,a; Liu et al., 2020). MS-AAGCN (Shi et al., 2020) uses attention to adaptively learn the graph topology and also proposes STC-attention module to adaptively weight joints, frames and channels. Decoupled GCN (Cheng et al., 2020a) improves the capacity of ST-GCN without adding additional computations and also proposes attention guided drop mechanism called DropGraph as a regularization technique. Sign-Language GCN (SL-GCN) (Jiang et al., 2021) combines STC-attention with Decoupled-GCN and extends it to ISLR achieving state-of-the-art results. Although there are works which use an already trained classifier (on a large dataset) to finetune for smaller datasets and obtain state-of-the-art results in the latter (Albanie et al., 2020), there are currently no works which study the value of pretraining on openly available unlabelled data. On this front, we now survey three broad classes of self-supervised pretraining strategies that we reckon could be applied to SLR. Masking-based pretraining: In NLP, masked language modelling is a pretraining technique where randomly masked tokens in the input are predicted. This approach has been explored for action recognition (Cheng et al., 2021), where certain frames are masked and a regression task estimates coordinates of keypoints. In addition, a direction loss is also proposed to classify the quadrant where the motion vector lies. Contrastive-learning based: Contrastive learning is used to learn feature representations of the input to maximize the agreement between augmented views of the data (Gao et al., 2021; Linguo et al., 2021). For positive examples, different augmentations of the same data item are used, while for negative samples randomly-chosen data items usually from a few last training batches are used. A variant of contrastive loss called InfoNCE (van den Oord et al., 2018) is used to minimize the distance between positive samples. Predictive Coding: Predictive Coding aims to learn data representation by continuously correcting its predictions about data in future timesteps given data in certain input timesteps. Specifically, the training objective is to pick the future timestep's representation from other negative samples which are usually picked from recent previous timesteps of the same video. Similar to contrastive learning, a loss function based on NCE is used (Mikolov et al., 2013; van den Oord et al., 2018). This technique was explored for action recognition in a model called Dense Predictive Coding (DPC) (Han et al., 2019). Instead of predicting at the frame-level, DPC introduces coarse-prediction at the scale of non-overlapping windows. In this section, we describe our efforts to curate standardized pose-based datasets across multiple sign languages and benchmark multiple ISLR models on them. Multiple datasets have been created for the ISLR task across sign languages. However, the amount of data significantly varies across different sign languages, with American and Chinese having the largest datasets currently. With a view to cover a diverse set of languages, we study 7 different 2117 Dataset Language Vocab Signers Videos Hrs Data AUTSL (Sincan and Keles, 2020) Turkish 226 43 38,336 20.5 RGBD CSL (Huang et al., 2019) Chinese 500 50 125,000 108.84 RGBD DEVISIGN (Chai et al., 2014) Chinese 2000 30 24,000 21.87 RGBD GSL (Adaloglou et al., 2021) Greek 310 7 40,785 6.44 RGBD INCLUDE (Sridhar et al., 2020) Indian 263 7 4,287 3.57 RGB LSA64 (Ronchetti et al., 2016) Argentinian 64 10 3,200 1.90 RGB WLASL (Li et al., 2020) American 2000 119 21,083 14 RGB Table 1: The diverse set of existing ISLR datasets which we study in this work through pose-based models datasets across 6 sign languages as summarised in Table 1. For each of these datasets, we generate pose-based data using the Mediapipe pose-estimation pipeline (Grishchenko and Bazarevsky, 2020), which enables real-time inference in comparison with models such as OpenPose (Cao et al., 2018). Mediapipe, in our chosen Holistic mode, returns 3d coordinates for 75 keypoints (exclud-ing the face mesh). Out of these, we select only 27 sparse 2d keypoints which convey maximum information, covering upper-body, hands and face. Thus, each input video is encoded into a vector of size F  K  D , where F is the number of frames in the video, K is the number of keypoints (27 in our case), and D is the number of coordinates (2 in our case). In addition, we perform several normalizations and augmentations explained in Section 5. On the 7 different datasets we consider, different existing ISLR models have been trained which are mentioned in Table 2 which produce their current state-of-the-art results. For INCLUDE dataset, an XGBoost model is used (Sridhar et al., 2020) with direct input as 135 pose-keypoints obtained using OpenPose. For AUTSL, SL-GCN is used (Jiang et al., 2021) with 27 chosen keypoints as input from HRNet pose estimation model. For GSL, the corresponding model (Parelli et al., 2020) is an attention-based encoder-decoder with 3D hand pose and 2D body pose as input. For WLASL, Temporal-GCN is used (Li et al., 2020) by passing 55 chosen keypoints from OpenPose. For LSA64, 33 chosen keypoints from OpenPose are used as input to an LSTM decoder (Konstantinidis et al., 2018). For DEVISIGN, RGB features are used (Yin et al., 2016) and the task is approached using a clustering-based classic technique called Iterative Reference Driven Metric Learning. For CSL dataset, an I3D CNN is used as encoder with input as RGBD frames and BiLSTM as decoder (Adaloglou et al., 2021). For DEVISIGN_L and CSL datasets, we report RGB model results in the table as there are no existing works using pose-based models. The differences in the above models make it difficult to compare them on effectiveness, especially across diverse datasets. To enable standardized comparison of models, we train pose-based ISLR models on all datasets with similar training setups for consistent benchmarking. These models belong to two groups: sequence-based models and graph-based models. For sequence-based models we consider RNN and Transformer based architectures. For the RNN model , we use a 4-layered bidirectional LSTM of hidden layer dimension 128 which takes as input the framewise pose-representation of 27 keypoints with 2 coordinates each, i.e., a vector of 54 points per frame. We also use a temporal attention layer to weight the most effective frames for classification. For the Transformer model , we use a BERT-based architecture consisting of 5 Transformer-encoder layers with 6 attention heads and hidden dimension size 128, with a maximum sequence length of 256. For the graph-based models we consider ST-GCN (Yan et al., 2018) and SL-GCN (Jiang et al., 2021) models as discussed in section 2. For ST-GCN model , we use 10 spatio-2118 Dataset State-of-the-art model Model available in OpenHands Model (Params) Accuracy LSTM BERT ST-GCN SL-GCN AUTSL Pose-SL-GCN 2 (4.9M) 95.02 77.4 81.0 90.4 91.9 CSL RGBD-I3D (27M) 95.68 75.1 88.8 94.2 94.8 DEVISIGN_L RGB-iRDML 56.85 37.6 48.9 55.8 63.9 GSL Pose-Attention (2.1M) 83.42 86.6 89.5 93.5 95.4 INCLUDE Pose-XGBoost 63.10 86.3 90.4 91.2 93.5 LSA64 Pose-LSTM (1.9M) 93.91 90.2 92.5 94.7 97.8 WLASL2000 Pose-TGCN (5.2M) 23.65 20.6 23.2 21.4 30.6 Average accuracy  67.7 73.5 77.3 81.1 Table 2: Accuracy of different models across datasets. temporal GCN layers with the spatial dimension of the graph consisting of the 27 keypoints. For the SL-GCN model , we use again 10 SL-GCN blocks with the same graph structure and hyperparameters as the ST-GCN model. We train 4 models LSTM, BERT, ST-GCN, and SL-GCN for each of the 7 datasets. We use Py-Torch Lightning to implement the data processing and training pipelines. We use Adam Optimizer to train all the models. We search for optimal hyperparameters using grid search to find the best hyperparams for each model on a standard dataset, and report the best configuration per model. For the LSTM model, we set the batch size as 32 and initial learning rate (LR) as 5 e  3 , while for BERT, we set a batch size 64, and LR of 1 e  4 . For ST-GCN and SL-GCN, we use a batch size of 32 and LR of 0.001. We train all our models on a NVIDIA Tesla V100 GPU. Also for all datasets, we only train on the train-sets given and we use valid-sets to do early stopping, whereas some works (like AUTSL) train on combination of train-set and val-set to report the final test accuracy. We run each experiment around 3 times, and report the best accuracy, eventhough we do not see significant difference in accuracies across the runs. All trained models and the training configurations are open-sourced in OpenHands . Accuracy We report the obtained test-set accuracy of detecting individual signs, for each model against each dataset in Table 2. On all datasets, graph-based models report the state-of-the-art results using pose data. Except for AUTSL 2 , on 6 of 2 SOTA AUTSL model is trained on high quality pose data from HRNet model with more keypoints. the 7 datasets, models we train improve upon the accuracy reported in the existing papers sometimes significantly (e.g., over 10% on GSL). These uniform results across a diverse set of SLs confirm that graph-based models on pose modality data define the SOTA. In summary, the standardized benchmarking of multiple models in terms of accuracy on datasets and, measurements of latency on devices (ex-plained in appendix) informs model selection. Making the trade-off between accuracy and latency, we use the ST-GCN model for the pretrained model we discuss later. Our choice is also informed by the cost of the training step: The more accurate SL-GCN model takes 4  longer to train than ST-GCN. In this section, we describe our efforts in building the largest corpus for self-supervised pretraining and our experiments in different pretraining strategies. Indian SL Corpus for Self-supervised pretraining Wikipedia dumps, OSCAR, etc. have enabled pretraining of large language models. Although there are large amounts of raw sign language videos available on the internet, no existing work has studied how such large volumes of open unlabelled data can be collected and used for SLR tasks. To address this, we create a corpus of Indian SL data by curating videos, pre-process the videos, and release a standardized pose-based dataset compatible with the models discussed in the previous section. We manually search for freely available major sources of Indian SL videos. We restrict our search to a single sign language so as to study the effect of pretraining on same language and crosslingual ISLR tasks. We sort the sources by the number of hours of videos and choose the top 5 sources for download. All of these 5 sources, as listed in Table 3 are YouTube channels, totalling over 1,500 hours before preprocessing. We pass these videos through a processing pipeline as described in Figure 2. We initially dump the pose data for all videos, then process them to remove those which are noisy or contain either no person or more than 1 person. This resulted in 1,129 hours of Indian SL data, as detailed source-wise in Table 3. This is significantly larger than all the training sets in the datasets we studied which is on average 177 hours. We pass these videos through MediaPipe to obtain pose information as described earlier, i.e., 75 keypoints per frame. The resultant Indian SL corpus has more than 100 million pose frames. We convert this to the HDF5 format to enable efficient random access, as is required for training. We open-source this corpus of about 250 GB which is available in OpenHands . We explore the three major pretraining strategies as described in Section 2.3 and explain how and why certain self-supervised settings are effective for ISLR. We pretrain on randomly sampled consecutive input sequences of length 60-120 frames (approximating 2-4 secs with 30fps videos). After pretraining, we fine-tune the models on the respective ISLR dataset with an added classification head. We follow the same hyperparameter settings as described in Motion-Transformer (Cheng et al., 2021), to pretrain a BERT-based model with random masking of 40% of the input frames. When using only the regression loss, we find that pretraining learns to reduce the loss as shown in appendix. However, when fine-tuned on the INCLUDE dataset, we see no major contribution of the pretrained model to increasing the accuracy as shown in Table 4. We posit that while pretraining was able to approximate interpolation for the masked frames based on the surrounding context, it did not learn higher-order features relevant across individual signs. We also experiment with different masking ratios (20% and 30%) as well as different length of random contiguous masking spans (randomly selected between 2-10), and obtain similar results. Inspired from the work by Gao et al. (2021), we consider Shear, Scaling and Rotation augmentations to generate the 2 augmented copies of the input pose sequence and we pretrain the model and observe that it converges on reducing the InfoNCE loss (see appendix for plot). We then fine-tune on INCLUDE and again did not observe any gain over the baseline of training from scratch as seen in Table 4. To understand this, we analyzed the embeddings of data from the pretrained model and observed two facts: (a) Embeddings of different augmentations of a video clip are similar indicating successful pretraining, but (b) Embeddings of different videos from the INCLUDE dataset do not show any clustering based on the class (see visualization in appendix). Again, we posit that 2120 pretraining did not learn higher order semantics that could be helpful for ISLR. 4.2.3 Predictive-coding based Our architecture is inspired from Dense Predictive Coding (Han et al., 2019), but using pose modality. The architecture is represented in Figure 3. The pose frames from a video clip will be partitioned into multiple non-overlapping windows with equal number of frames in each window. The encoder f takes each window of pose keypoints as input and embeds into the hidden space z . We use ST-GCN as the encoder. The ST-GCN encoder embeds each input window x i , and the direct output is average pooled across the spatial and temporal dimensions to obtain the output embedding z i for each window. The embeddings are then fed to a Gated Recurrent Unit (GRU) as a temporal sequence and the future timesteps  z i are predicted sequentially using the past timestep representations from GRU, with an affine transform layer  . We use 4 windows of data as input to predict the embeddings of the next 3 windows, each window spanning 10 frames, which we empirically found to be the best setting. For pretraining, we used a batch size of 128 and for finetuning, we used a batch size of 64. For both pretraining and finetuning, we used Adam optimizer with an initial learning rate of 1e-3. The pretraining was done for 200k iterations on a NVIDIA V100 GPU, taking around 26 hours (on Microsoft platform's Azure NC6s_v3 machine). Time Upon fine-tuning on INCLUDE, DPC provides a significant improvement of 3.5% over the baseline. We include a plot comparing the validation accuracy between baseline and finetuned model in appendix. We posit that Sign Language DPC (SL-DPC) is successful, while previous methods were not, as it learns coarse-grained representations across multiple frames and thereby captures motion semantics of actions in SL. To the best of our knowledge, this is the first comparison of pretraining strategies for SLR. We demonstrated that DPC-based pretraining is effective. We now analyze the effectiveness of such pretraining in two constrained settings - (a) when fine-tuning datasets are small, and (b) when fine-tuning on sign languages different from the sign language used for pretraining. The former captures in-language generalization while the latter crosslingual generalization. The INCLUDE dataset contains an average of 17 samples per class. For this setting, we observed a gain of 3.5% with DPC-based pretraining over training from scratch. How does this performance boost change when we have fewer samples per 2121 class? We present results for 10, 5, and 3 samples per class in Table 5. We observe that as the number of labels decreases the performance boost due to pretraining is higher indicating effective in-language generalization. Does the pretraining on Indian sign language provide a performance boost when fine-tuning on other sign languages? We study this for 3 different sign languages American, Chinese, and Argentinian and report results in Table 5. We see that crosslingual transfer is effective leading to gains of about 6%, 4%, and 2% on the three datasets, similar to the 3% gain on in-language accuracy. The increase in accuracy varies with datasets For Argentianian and Indian datasets which already have 90+% accuracy, there are small improvements. However, WLASL which is scraped from web and has a lot more variations, sees a much higher improvement due to pretraining. Further, we also observe that these gains extend to low-resource settings of fewer labels per sign. For instance on Argentinian SL, with 3 labels, pretraining on Indian SL given an improvement of about 18% in accuracy. To the best of our knowledge this is the first successful demonstration of crosslingual transfer in ISLR. In summary, we discussed different pretraining strategies and found that only SL-DPC learns semantically relevant higher-order features. With DPC-based pretraining we demonstrated both in-language and crosslingual transfer. As mentioned in the main paper, we open-source all our contributions through the OpenHands library. This includes the pose-based datasets for 5 SLs, 4 ISLR models trained on 7 datasets, the pretraining corpus on Indian SL with over 1,100 hours of pose data, pretrained models on this corpus using self-supervised learning, and models fine-tuned for 4 different SLs on top of the pretrained model. We also provide scripts for efficient deployment using MediaPipe pose estimation and our trained ISLR models. In addition, the library provides utilities that are helpful specifically for pose-based data. This includes methods to normalize keypoints by width and height of frames, to normalize all of the pose data to be in the same scale and reference coordinate system by using a constant feature of body as reference, and to fill missing keypoints. The library also includes utilities to create data augmentations such as ShearTransform to displace the joints in a random direction, RotatationTransform to simulate the viewpoint changes of the camera, ScaleTrans-form to simulate different scales of the pose data to account for relative zoomed-in or zoomed-out view of signers, PoseRandomShift to move a significant portion of the video by a time offset so as to make the ISLR models robust to inaccurate segmentation of real-time video, UniformTemporalSubsample to uniformly sample frames from the video instead of considering only the initial frames, in cases where the number of frames in a video clip exceeds a maximum limit, and RandomTemporalSubsample to sample a random fixed contiguous window of required size covering a maximum number of frames. We encourage researchers to contribute datasets, models, and other utilities to make sign language research more accessible. All the aspects of the toolkit are well-documented online 3 for anyone to get started easily. In this work, we make several contributions to make sign language research more accessible. We release pose-based datasets and 4 different ISLR models across 6 sign languages. This evaluation enabled us to identify graph-based methods such as ST-GCN as being accurate and efficient. We release the first large corpus of SL data for self-supervised pretraining. We evaluated different pretraining strategies and found DPC as being effective. We also show that pretraining is effective both for in-language and crosslingual transfer. All our models, datasets, training and deployment scripts are open-sourced in OpenHands . Several directions for future work emerge such as evaluating alternative graph-based models, experimenting with varying sequence lengths of input data, efficiently sampling the data from the raw dataset for pretraining such that the samples are diverse enough, using better pose estimator models and more keypoints, and quantized inference for 2  -4  reduced latency. On the library front, we aim to release updated versions incorporating more SL datasets, better graph-based models, studying the performance on low FPS videos (like 2-4 FPS), effect of pretraining using other high-resource SL datasets, extending to CSLR, and improving deployment features. We would like to thank Aravint Annamalai from IIT Madras for preparing the list of potential YouTube channels that can be crawled and for his help in downloading them. We would like to thank the entire AI4Bharat Sign Language Team 4 for their support and feedback for this work, especially from Rohith Gandhi Ganesan for his insights on code structuring, and Advaith Sridhar for managing the overall project. We would also like to extend our immense gratitude to Microsoft's AI4Accessibility program (via Microsoft Philanthropies India) for granting us the compute required to carry out all the experiments in this work, through Microsoft Azure cloud platform. Our extended gratitude also goes to Zenodo, who helped us with hosting our large datasets (NC and Selvaraj, 2021). Finally, we thank all the content creators and ISLR dataset curators without whose data this work would have been impossible.
abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain objective result abstain method abstain method objective abstain method objective abstain abstain abstain objective abstain result abstain result abstain method method result abstain abstain objective abstain method abstain result abstain objective other abstain other other other other other other other other other other other other method other other other other other other other other method abstain other other other other other other other other other other other other other method other other other other other other other other other other other abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method method abstain method method method method method method method method method method abstain method abstain method abstain abstain method method method abstain abstain abstain method abstain method method abstain method abstain abstain abstain method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain objective abstain method objective result result abstain abstain objective other other other other other