Univeristy of Chicago dbrentari@uchicago.edu Greg Shakhnarovich TTI-Chicago greg@ttic.edu Abstract Natural language processing for sign language videoincluding tasks like recognition, translation, and searchis crucial for making ar-tificial intelligence technologies accessible to deaf individuals, and is gaining research interest in recent years. In this paper, we address the problem of searching for fingerspelled keywords or key phrases in raw sign language videos. This is an important task since signifi-cant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before. We propose an end-to-end model for this task, FSS-Net, that jointly detects fingerspelling and matches it to a text sequence. Our experiments, done on a large public dataset of ASL fingerspelling in the wild, show the importance of fingerspelling detection as a component of a search and retrieval model. Our model significantly outperforms baseline methods adapted from prior work on related tasks. Sign languages are a type of natural language which convey meaning through sequences of handshapes and gestures as well as non-manual elements, and are a chief means of communication for about 70 million deaf people worldwide. 1 Automatic sign language technologies would help to bridge the communication barrier between deaf and hearing individuals, and would make deaf video media more searchable and indexable. Automatic sign language processing has recently received growing interest in the computer vision (CV) and natural language processing (NLP) communities. Yin et al. (2021) make several recommendations for the study of sign languages in NLP research, including greater emphasis on real-world data. Most studies on sign language are based on data collected in a controlled environment, either 1 From https://wfdeaf.org/our-work/ in a studio setting (Martnez et al., 2002; Kim et al., 2017) or in a specific domain (Forster et al., 2016). The challenges involved in real-world signing videos, including various visual conditions and different levels of fluency in signing, are not fully reflected in such datasets. Automatic processing of sign language videos "in the wild" has not been addressed until recently, and is still restricted to tasks like isolated sign recognition (Albanie et al., 2020; Joze and Koller, 2019; Li et al., 2020) and fingerspelling recognition (Shi et al., 2018, 2019). In this work we take a step further and study search and retrieval of arbitrary fingerspelled content in real-world American Sign Language (ASL) video (see Figure 1). in which words are signed by a series of handshapes or movements corresponding to single letters (see the Appendix for the ASL fingerspelling alphabet). Fingerspelling is used mainly for lexical items that do not have their own signs, such as proper nouns or technical terms, and has an important place in sign language. For example, fingerspelling accounts for 12-35% of ASL (Padden and Gunsauls, 2003). Since important content like named entities is often fingerspelled, the fingerspelled portions of a sign language video often carry a disproportionate amount of the content. Most prior work on fingerspelling has focused on recognition (Shi et al., 2018, 2019), that is, transcription of a fingerspelling video clip into text. However, automatic recognition assumes that the boundaries of fingerspelled segments are known at test time, and may not be the end goal in real-world use cases. In addition, complete transcription may not be necessary to extract the needed information. Fingerspelling search, such as retrieving sign language videos based on a query word, is a more practical task, and is an important component of general video search involving sign language. In addition to introducing the task, we address the research question of whether the explicit temporal localization of fingerspelling can help its search and retrieval, and how best to localize it. As fingerspelling occurs sparsely in the signing stream, explicit detection of fingerspelling could potentially improve search performance by removing unrelated signs. To this end, we propose an end-to-end model, FSS-Net, which jointly detects fingerspelling from unconstrained signing video and matches it to text queries. Our approach consistently outperforms a series of baselines without explicit detection and a baseline with an off-the-shelf fingerspelling detector by a large margin. In existing work on sign language video processing, search and retrieval tasks have been studied much less than sign language recognition (mapping from sign language video to gloss labels) (Koller et al., 2017; Forster et al., 2016) and translation (map-ping from sign language video to text in another language) (Yin and Read, 2020; Camgz et al., 2018). Work thus far on sign language search has been framed mainly as the retrieval of lexical signs rather than fingerspelling. Pfister et al. (2013); Albanie et al. (2020) employ mouthing to detect keywords in sign-interpreted TV programs with coarsely aligned subtitles. Tamer and Saralar (2020a,b) utilize whole-body pose estimation to search for sign language keywords (gloss or translated word) in a German Sign Language translation dataset PHOENIX-2014T (Camgz et al., 2018). All prior work on keyword search for sign language has been done in a closed-vocabulary setting, which assumes that only words from a pre-determined set will be queried. Searching in an open-vocabulary setting, including proper nouns, typically requires searching for fingerspelling. Some related tasks in the speech processing literature are spoken term detection (STD) and query-by-example search, which are the tasks of automatically retrieving speech segments from a database that match a given text or audio query (Knill et al., 2013; Mamou et al., 2007; Chen et al., 2015). In terms of methodology, our model also shares some aspects with prior work on moment retrieval (Gao et al., 2017; Xu et al., 2019; Zhang et al., 2020), which also combines candidate generation and matching components. However, we incorporate additional task-specific elements that consistently improve performance. We consider two tasks: Fingerspelled Word Search (FWS) and Fingerspelling-based Video Search (FVS). FWS and FVS respectively consist of detecting fingerspelled words within a given raw ASL video stream and detecting video clips of interest containing a given fingerspelled word. 2 Given a query video clip v and a list of n words w 1: n , FWS is the task of finding which (if any) of w 1: n are present in v . Conversely, in FVS the input is a query word w and n video clips v 1: n , and the task consists of finding all videos containing the fingerspelled word w . We consider an open-vocabulary setting where the word w is not constrained to a pre-determined set. The two tasks correspond to two directions of search (video  text and text  video), as is standard practice in other retrieval work such as video-text search (Zhang et al., 2018; Ranjay et al., 2017; Ging et al., 2020). We propose a single model, FSS-Net (for "Finger-Spelling Search Network"), summarized in Fig-2 Fig-2 We use "word" to refer to a fingerspelling sequence, which could be a single word or a phrase. ure 2, to address the two aforementioned search tasks. FSS-Net receives a pair of inputsa raw ASL video clip, and a written text sequenceand produces a score indicating the degree of match between the video clip and the text. The text is encoded into an embedding vector via a learned encoder. The visual branch of FSS-Net generates a number of fingerspelling segment proposals and each proposed visual segment is encoded into a feature space shared with the text embeddings. Paired embeddings from both modalities are drawn towards each other in the shared embedding space during training. Image encoding The input image frames are encoded into a sequence of feature vectors via an image encoder, which consists of the VGG-19 (Si-monyan and Zisserman, 2015) convolutional layers followed by a Bi-LSTM. 3 We use raw RGB images as input, instead of signer pose as used in some prior work (Tamer and Saralar, 2020b,a) on sign language search, as estimating pose for hands is particularly hard for signing videos in the wild (see Section 6 for details). Temporal proposal generation Suppose the visual feature sequence is f 1: T , where T is the number of frames in the video clip. The purpose of temporal proposal generation is to produce a number of candidate fingerspelling segments H ( I 1: T ) = { ( s i , t i ) } 1  i |H ( I 1: T ) | from f 1: T , where s i , t i are the start and end frame indices of 3 Transformers (Vaswani et al., 2017) can also be used, but in our initial experiments, they were outperformed by BiLSTMs on our tasks and data. the i th proposed segment. Below we use H as a shorthand for H ( I 1: T ) . Here we adopt the strategy in (Xu et al., 2017), which is commonly used to generate proposals for action detection. Briefly, the model assigns a probability p det of each proposal being fingerspelling. See (Xu et al., 2017) for more details. We denote the detection loss as L det . Note that the training requires known ground-truth fingerspelling boundaries. In the fingerspelling datasets we use here (Shi et al., 2018, 2019), the fingerspelling boundaries are already annotated, so no further annotation is needed. Filtering A visual embedding is produced for each segment. We denote a labeled fingerspelling segment (shortened as fingerspelling segment below) as a tuple ( s, t, w ) , where s , t and w represent the start frame index, the end frame index, and the written text it represents. A naive approach would be to use only the ground-truth fingerspelling segments P g = { ( s i , t i , w i ) } 1  i |P g | for training. However, this approach does not take into account the potential shifts (errors) that may exist at test time between the ground-truth and generated segment proposals. The embeddings produced by the fingerspelling encoder should be robust to such shifts. To this end, we incorporate proposals in forming positive pairs at training time. Formally, let the set of time intervals from the temporal proposal generator be H = { ( s i , t i ) } 1  i |H| . We sample K intervals from P t to form the set of generated 1701 fingerspelling segments: P k = { ( s k , t k , w g ) | IoU (( s k , t k ) , ( s g , t g )) >  IoU , IS (( s t , t k ) , ( s g , t g )) >  IS , ( s k , t k )  H , ( s g , t g , w g )  P g } (1) where IS ( x, y ) = Intersection ( x,y ) Length ( y ) and IoU ( x, y ) = Intersection ( x,y ) Union ( x,y ) . We use  IoU and  IS to control the degree to which the proposals can deviate from the ground-truth. In addition to the intersection over union (IoU), we use the normalized intersection IS to eliminate proposals with many missing frames. We take the union of the two sets, P + = P g  P k , as the filtered proposal set to be encoded. Fingerspelling visual encoding (FS-encoding) The visual encoding of each segment ( s, t, w )  P + is e ( s,t ) v = Bi-LSTM ( f s : t ) . 4 Text encoding A written word (or phrase) w is mapped to an embedding vector e wx via a text encoder. To handle words not seen at training time (and better handle rarely seen words), we first decompose w into a sequence of characters c 1: | w | (e.g. ASL'=A'-S'-L') and feed the character sequence c 1: | w | into a text encoder (here, a Bi-LSTM 5 ). Visual-text matching With the above pairs of visual and textual embeddings, we use a training objective function consisting of two triplet loss terms: L tri ( I 1: T , P + ) = (cid:88) ( s,t,w ) P + max { m + d ( e ( s,t ) v , e wx )  1 |N w | (cid:88) w (cid:48) N w d ( e ( s,t ) v , e w (cid:48) x ) , 0 } + max { m + d ( e ( s,t ) v , e wx )  1 |N v | (cid:88) ( s (cid:48) ,t (cid:48) ) N v d ( e ( s (cid:48) ,t (cid:48) ) v , e wx ) , 0 } (2) where d denotes cosine distance d ( a , b ) = 1  a  b (cid:107) a (cid:107)(cid:107) b (cid:107) , m is a margin, and N v and N w are sets of negative samples of proposals and words. To form negative pairs we use semi-hard negative sampling (Schroff et al., 2015): N v = { ( s (cid:48) , t (cid:48) ) | d ( e ( s (cid:48) ,t (cid:48) ) v , e wx ) > d ( e ( s,t ) v , e wx ) } N w = { w (cid:48) | d ( e ( s,t ) v , e w (cid:48) x ) > d ( e ( s,t ) v , e wx ) } (3) 4 We compared the Bi-LSTM encoder with average/max pooling of f s : t , and found the former to perform better. the corresponding mini-batch. Overall loss The model is trained with a combination of the detection loss and triplet loss: L tot ( I 1: T , P g ) =  det L det ( I 1: T , P g ) + L tri ( I 1: T , P + ) (4) with the tuned weight  det controlling the relative importance of detection versus visual-textual matching. Inference At test time, the model assigns a score sc ( I 1: T , w ) to a given video clip I 1: T and word w . The word is encoded into the word embedding e wx . Suppose the set of fingerspelling proposals generated by the temporal proposal generator is H ( I 1: T ) . We define a scoring function for the proposal h  H ( I 1: T ) and word w sc word ( h m , w ) = p  det (1  d ( e h m v , e wx )) (5) where p det is the probability given by the temporal proposal generator and  controls the relative weight between detection and matching. In other words, in order for a segment and word to receive a high score, the segment should be likely to be fingerspelling (according to p det ) and its embedding should match the text. Finally, the score for the video clip I 1: T and the word w is defined as the highest score among the set of proposals H ( I 1: T ) : sc ( I 1: T , w ) = max h H ( I 1: T ) sc word ( h, w ) (6) 5 Experimental Setup 5.1 Data We conduct experiments on ChicagoFSWild (Shi et al., 2018) and ChicagoFSWild+ (Shi et al., 2019), two large-scale publicly available fingerspelling datasets containing 7,304 and 55,272 fingerspelling sequences respectively. The ASL videos in the two datasets are collected from online resources and include a variety of viewpoints and styles, such as webcam videos and lectures. We follow the setup of (Shi et al., 2021) and split the raw ASL videos into 300-frame clips with a 75-frame overlap between neighboring chunks and remove clips without fingerspelling. The numbers of clips in the various splits can be found in the Appendix. On average, each clip contains 1.9/1.8 fingerspelling segments in the ChicagoFSWild and ChicagoFSWild+ datasets respectively. We compare the proposed model, FSS-Net, to the following baselines adapted from common approaches for search and retrieval in related fields. To facilitate comparison, the network architecture for the visual and text encoding in all baselines is the same as in FSS-Net. Recognizer In this approach, we train a recognizer that transcribes the video clip into text. Specifically, we train a recognizer to output a sequence of symbols consisting of either fingerspelled letters or a special non-fingerspelling symbol <x>. We train the recognizer with a connectionist temporal classification (CTC) loss (Graves et al., 2006), which is commonly used for speech recognition. At test time, we use beam search to generate a list of hypotheses  w 1: M for the target video clip I 1: T . Each hypothesis  w m is split into a list of words {  w nm } 1  n  N separated by <x>. The matching score between video I 1: T and w is defined as: sc ( I 1: T , w ) = 1  min 1  m  M min 1  n  NLER (  w nm , w ) (7) where the letter error rate LER is the Leven-shtein edit distance. This approach is adapted from (Saralar and Sproat, 2004) for spoken utterance retrieval. Fingerspelling boundary information is not used in training this baseline model. Whole-clip The whole-clip baseline encodes the whole video clip I 1: T into a visual embedding e Iv , which is matched to the textual embedding e wx of the query w . The model is trained with contrastive loss as in equation 2. At test time, the score for video clip I 1: T and word w is: sc ( I 1: T , w ) = 1  d ( e Iv , e wx ) (8) where d is the cosine distance as in FSS-Net. Fingerspelling boundary information is again not used in this baseline. External detector (Ext-Det) This baseline uses the off-the-shelf fingerspelling detectors of (Shi et al., 2021) to generate fingerspelling proposals, instead of our proposal generator, and is otherwise identical to FSS-Net. For each dataset (ChicagoF-SWild, ChicagoFSWild+), we use the detector trained on the training subset of that dataset. This baseline uses ground-truth fingerspelling boundaries for the detector training. Saralar, 2020b)'s approach for keyword search in sign language. The model employs an attention mechanism to match a text query with a video clip, where each frame is weighted based on the query embedding. The attention mechanism enables the model to implicitly localize frames relevant to the text. The model of (Tamer and Saralar, 2020b) is designed for lexical signs rather than fingerspelling. To adapt the model to our open-vocabulary fingerspelling setting, we use the same text encoder as in FSS-Net to map words into embeddings instead of using a word embedding matrix as in (Tamer and Saralar, 2020b). Fingerspelling boundary information is again not used in training this model, which arguably puts it at a disadvantage compared to FSS-Net. More details on the formulation of the model can be found in the Appendix. For FWS, we use all words in the test set as the test vocabulary w 1: n . For FVS, all video clips in the test are used as candidates and the text queries are again the entire test vocabulary. We report the results in terms of standard metrics from the video-text retrieval literature (Momeni et al., 2020; Tamer and Saralar, 2020a): mean Average Precision (mAP) and mean F1 score (mF1), where the averages are over words for FVS and over videos for FWS. Hyperparameters are chosen to maximize the mAP on the dev set, independently for the two tasks (though ultimately, the best hyperparameter values in our search are identical for both tasks). Additional details on data, preprocessing, model implementation, and hyperparameters can be found in the Appendix. Table 1 shows the performance of the above approaches on the two datasets. First, we notice that embedding-based approaches consistently outperform the recognizer baseline in the larger data setting (ChicagoFSWild+) but not the smaller data setting (ChicagoFSWild), which suggests that embedding-based models generally require more training data. The inferior performance of recognizer also shows that explicit fingerspelling recognition is not necessary for the search tasks. In addition, explicit fingerspelling detection (Ext-Det, FSS-Net) improves performance over implicit fin-1703 Table 1: FWS/FVS performance on the ChicagoFSWild and ChicagoFSWild+ test sets. gerspelling detection (Attn-KWS) and detection-free search (Whole-clip). Explicit fingerspelling detection requires boundary information during training. Of the models that don't use such supervision, Attn-KWS is the best performer given enough data, but is still far behind FSS-Net. Our model outperforms all of the alternatives. The relative performance of different models remains consistent across the various metrics and the two search tasks. For completeness, we also measure the performance of different models in terms of ranking-based metrics (e.g., Precision@N, Recall@N), as in prior work on video-text retrieval (Ging et al., 2020; Ranjay et al., 2017) (see full results in the Appendix). The relative performance of different models remains consistent on these metrics. The analysis below is done on ChicagoFSWild for simplicity. The conclusions also hold for ChicagoF-SWild+. Does better localization lead to better search? In the previous section we have seen that models that explicitly detect and localize fingerspelling outperform ones that do not. Next we look more closely at how well several modelsExt-Det, Attn-KWS and FSS-Netperform on the task of localizing fingerspelling, which is a byproduct of these models' output. We measure performance via AP@IoU, a commonly used evaluation metric for action detection (Idrees et al., 2016; Heilbron et al., 2015) that has also been used for fingerspelling detection (Shi et al., 2021). AP@IoU measures the average precision of a detector under the constraint that the overlap of its predicted segments with the ground truth is above some threshold Intersection-over-Union (IoU) value. For Attn-KWS, the model outputs an attention vector, which we convert to segments as in (Shi et al., 2021). In general, the models with more accurate localization also have higher search and retrieval performance, as seen by comparing Table 2 with Table 1. However, differences in AP@IoU do not directly translate to differences in search performance. For example, the AP@IoU of Ext-Det (0.344) is an order of magnitude higher than that of Attn-KWS (0.035) while their FVS mAP results are much closer (0.593 vs. 0.573). Raw images vs. estimated pose as input Prior work on sign language search (Tamer and Saralar, 2020a,b) has used estimated pose keypoints as input, rather than raw images as we do here. For comparison, we extract body and hand keypoints with OpenPose (Cao et al., 2019) and train a model with the pose skeleton as input. As is shown in Table 3, the pose-based model has much poorer search performance than the RGB image-based models. We believe this is largely because, while pose estimation works well for large motions and clean visual conditions, in our dataset much of the handshape information is lost in the estimated pose (see the Appendix for some qualitative examples). Within our model, the proposal generator produces a subset of all possible fingerspelling proposals, intended to represent the most likely fingerspelling segments. To measure whether this component is important to the performance of the model, we compare our full model with the proposal generator to one where the proposal generator is removed (see Table 4). When the proposal generator is not used, the model is trained only with ground-truth fingerspelling segments ( P g ) and considers all possible proposals within a set of sliding windows. Such a "sliding-window" approach is commonly used in previous sign language keyword search (Al-banie et al., 2020; Pfister et al., 2013) and spoken keyword spotting (Chen et al., 2015). As can be seen from Table 4 (Full model vs. row (1)), the proposal generator greatly improves search performance. This is not surprising, since the proposal generator greatly reduces the number of non-fingerspelling segments, thus lowering the chance of a mismatch between the text and video, and also refines the segment boundaries through regression, which should improve the quality of the visual segment encoding. The fingerspelling detection component of our model has two aspects that may affect performance: imposing an additional loss during training, and rescoring during inference. We disentangle these two factors and show their respective benefits for our model in Table 4 (row (2) and (3)). The auxiliary detection task, which includes classification between fingerspelling and non-fingerspelling proposals, helps encode more comprehensive visual information into the visual embedding. In addition, the proposal probability output by the detector 1705 contains extra information and merging it into the matching score further improves the search performance. Table 4 (row (4)) shows the effect of sampling additional proposals ( P k ) in fingerspelling detection. Additional positive samples make the visual embedding more robust to temporal shifts in the generated proposals, thus improving search performance. The performance of our model is worse for short fingerspelled sequences than for long sequences (see Figure 4). This may be because shorter words are harder to spot, as is shown from the trend in fingerspelling detection in the same figure. The datasets we use are collected from multiple sources, and the video quality varies between them. To quantify the effect of visual quality on search/retrieval performance, we categorize the ASL videos into three categories according to their source: YouTube, DeafVIDEO, and other miscellaneous sources (misc). YouTube videos are mostly ASL lectures with high resolution. DeafVIDEO videos are vlogs from deaf users of the social media site deafvideo.tv , where the style, camera an-gle, and image quality vary greatly. The visual quality of videos in the miscellaneous category tends to fall between the other two categories. Typical image examples from the three categories can be found in the Appendix (figure 7). The FWS performance of our model on videos in YouTube, DeafVIDEO, and misc are 0.684, 0.584, 0.629 (mAP) respectively. The results are overall consistent with the perceived relative visual qualities of these categories. words/phrases with the highest/lowest FVS performance. The best-performing queries tend to be long and drawn from the highest-quality video source. Another common source of error is confusion between letters with similar handshapes (e.g., "i" vs. "j"). A final failure type is fingerspelling detection failure. As our model includes a fingerspelling detector, detection errors can harm search performance. Our work takes one step toward better addressing the need for language technologies for sign languages, by defining fingerspelling search tasks and developing a model tailored for these tasks. These tasks are complementary to existing work on keyword search for lexical signs, in that it addresses the need to search for a variety of important content that tends to be fingerspelled, like named entities. Fingerspelling search is also more challenging in that it requires the ability to handle an open vocabulary and arbitrary-length queries. Our results demonstrate that a model tailored for the task in fact improves over baseline models based on related work on signed keyword search, fingerspelling detection, and speech recognition. However, there is room for improvement between our results and the maximum possible performance. One important aspect of our approach is the use of explicit fingerspelling detection within the model. An interesting avenue for future work is to address the case where the training data does not include segment boundaries for detector training. Finally, a complete sign language search system should consider both fingerspelling and lexical sign search.
abstain abstain abstain abstain abstain method objective result result abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result other other other other other other other method abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain other other abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective result abstain abstain abstain