The DeepMind Chinese–English Document Translation System at WMT2020

This paper describes the DeepMind submission to the Chinese\rightarrowEnglish constrained data track of the WMT2020 Shared Task on News Translation. The submission employs a noisy channel factorization as the backbone of a document translation system. This approach allows the flexible combination of a number of independent component models which are further augmented with back-translation, distillation, fine-tuning with in-domain data, Monte-Carlo Tree Search decoding, and improved uncertainty estimation. In order to address persistent issues with the premature truncation of long sequences we included specialized length models and sentence segmentation techniques. Our final system provides a 9.9 BLEU points improvement over a baseline Transformer on our test set (newstest 2019).

Introduction
The WMT2020 Shared Task on translating news data from Chinese into English provides a challenging test for machine translation systems and an ideal domain for researchers to evaluate new techniques. The DeepMind submission to the constrained data track is based on the modular noisy channel document translation architecture advocated by Yu et al. (2020). In this formulation, the posterior probability of a translation is the product of the unconditional probability of the output document (the language model) and the conditional probability of the translation from the output to source (the channel model). By assuming sentences within a document are independently translated, we can train the channel model using readily available parallel sentences, rather than being reliant on less numerous parallel documents, and the language model on monolingual documents. This modular approach allows the components of the system to be * *Equal contribution.
implemented and optimized independently while at inference time, when we reason over the posterior distribution of translations given the source document, conditional dependencies between translations are induced by the language model prior.
The core of our document-level translation architecture is the noisy channel reranker. It requires proposal, channel, and language models, each of which is optimized separately using different techniques and approaches. For the proposal and channel models we use Transformer models (Vaswani et al., 2017) ( §4.1) with data augmentation ( §4.2), such as back translation (Edunov et al., 2018), distillation (Kim and Rush, 2016;Liu et al., 2016), and forward-translated parallel documents. We further improve these sequence-to-sequence (seq2seq) models by fine-tuning them with in-domain data ( §4.3). To improve the robustness of the reranker we apply adversarial training and contrastive learning methods for uncertainty estimation ( §4.4). Finally, we include candidate translations generated by Monte-Carlo Tree Search (MCTS) ( §B) in order to improve the diversity of the candidate pool for the reranker. Our language models are based on the Transformer-XL architecture (Dai et al., 2019) and optimized with distillation and fine-tuning with in-domain data ( §5).
During development, we observed weaknesses in our system's translations for long sentences, largely due to premature truncations. We developed several techniques to mitigate this issue such as sentence segmentation (breaking sentences into logical complete segments) and training specialized models with synthetically constructed long sequences to generate additional proposals for our reranker ( §A).
Experiments show that the aforementioned techniques are very effective: our system outperforms the Transformer baseline by 9.9 BLEU points on our test set (newstest2019). Our final system achieves a BLEU score of 35.4 on the Chinese→English news test set of WMT2020.

Document Translation via Bayes' Rule
Following Yu et al. (2020), we model document translation via Bayes' rule. We define X = (x 1 , x 2 , . . . , x I ) as the source document with I sentences, and similarly, Y = (y 1 , y 2 , . . . , y J ) as the target document with J sentences, where x i and y j denote the ith sentence in the source document and the jth sentence in the target document respectively. We assume that I = J.
The translation of a document X is determined by finding the documentŶ , where p(Ŷ | X) is maximal.
We further assume that sentences are independently translated, and that the sentences within a document admit a left-to-right factorization according to the chain rule. Therefore, we havê where Y <i = (y 1 , . . . , y i−1 ) denotes a document prefix consisting of the first i − 1 target sentences.
The advantages of this formulation is that during training the translation models can be learned from parallel sentences and monolingual documents which are vastly available in practice unlike parallel documents. During test time, when a source document is observed, conditional dependencies between the translation of the source sentences are created in the posterior.

Reranking
Because of the global dependencies in the posterior distribution, decoding in the aforementioned document translation model is computationally expensive. Following Yu et al. (2020), we use an auxiliary proposal model q(y | x), that approximates the posterior distribution using a direct model, to focus our search on promising parts of the output space. We then carry out the reranking process using an iterative beam search, over candidates generated by the proposal model q, to optimize the objective: where p PM is the proposal probabilities model, p AM is the adversarially trained proposal model ( §4.4.1), p CM is the channel model ( §4.4.2), p LM is the language model ( §5.1), and |y| denotes the number of tokens in the sentence y. The weights of component models (λs) are hyperparameters to be tuned in experiments.
In practice, we generate for each source sentence x i in the document X, a series of candidates y i , using the proposal model q. As all of the terms in the objective, except for p LM , only involve independent target sentences, they can be computed ahead of time in a scoring phase. The scored candidates are then passed to the reranker, where the language model is evaluated on the successive prefixes explored by the search, and which outputs the final documentŶ .
Iterative beam search The algorithm starts with k complete documents using randomly selected candidates for each of the source sentences. We then iterate through every source sentence x i , replacing the randomly picked initial candidate with every available candidate y i . We pick the top k scoring complete documents and continue iterating over the document. Unlike traditional beam search used by Yu et al. (2020), we go through every sentence in the document multiple times, until the top 1 translation converges (usually 2 to 4 full iterations). This allows for context from latter sentences in the document to inform the choice of earlier candidates.
Iterative beam search found improvements in the model objective over traditional beam search in 63% of the documents in our test set. Improvements in objective did not translate in a stable improvement in BLEU or META scores (Eqn. 4)in fact those scores were slightly reduced for a number of documents. Nevertheless, an informal human evaluation of translated documents showed preference for iterative beam search.
Selection of the hyperparameters λ We perform a grid search over the hyperparameters λ to maximize a metric on the validation set. The metric we use is the following META score, combining corpus-level BLEU, TER, METEOR, and the 0.1quantile of per-document BLEU, such that: When several configurations of hyperparameters achieve values of META very close to the maximum (within 0.02), we pick the one maximizing BLEU and/or minimizing the L 2 norm of the λs, considered as a vector. This corresponds to an intuitive prior towards giving more weight to the language model.

Training Data
To train all of the models used in our system, we made use only of the constrained data provided to shared task participants. In this section, we discuss the preprocessing and normalization techniques we carried out in an attempt to reduce spurious uncertainty in the modeling problem.
Text preprocessing We carried out the following text normalization steps prior to use in any models: • Text normalization. Unicode canonicalization (NKFD from), replacement of common multiple encoding errors present in training data, standardization of quotation marks into "directional" variants, conversion of any traditional Chinese characters into simplified forms. Replacement of non-American spelling variants with American spellings using the aspell library. 1 • Segmentation into words. Chinese was segmented into word-like units using the Jieba segmentation tool. 2 Punctuation was split from English words using a purpose-built library. These processes were not completely invertible, but they could be undone with simple rules so as to generate presentation-ready English and Chinese.
• True-casing. Words containing only an initial capital letter that occurred at the start of a sentence were replaced with the capitalized variant that occurred most frequently in other positions of the English monolingual training data. Thus, in the previous sentence the initial token would have been words rather than Words.
Subword units To encode text into sub-word units, we used the sentencepiece tool (Kudo and Richardson, 2018). For seq2seq models (i.e., the channel model and proposal models), we trained the segmentation model on the first 10 million sentences of the parallel training corpus, 3 using joint source and target unigram (Kudo, 2018) subword segmentation algorithm with a target vocabulary of 32K tokens and minimum character coverage of 0.9995, which resulted in 32,768 word pieces. 4 For the language model, we used the English side alone with the same vocabulary size and a character coverage of 1.0.

Proposal and Channel Models
The proposal model, used to generate candidate translations, and the scoring models (proposal probability model, adversarially-trained proposal model, channel model), used to compute features for the reranker, are seq2seq models. We describe here how we train and use them.

Sequence-to-Sequence Model
All our models are based on the Transformer architecture (Vaswani et al., 2017). We increased the inner dimension of the feed-forward network from 4,096 to 8,096 and decreased the model size (d model ) from 1,024 to 512, which allowed us to use 12 layers with 16 attention heads each. Additionally, we tied the source and target embedding layers. Following (Vaswani et al., 2018), we applied layer normalization to the input of every sublayer as opposed to its original placement after the element-wise residual addition. We used different dropout values for different components: 0.1 for the multi-head attention, 0.05 in the feed-forward network, and finally 0.3 after the sub-layer. Learning rate schedule and dropout were found using the Batched Gaussian Process Bandits (Desautels et al., 2014) algorithm as implemented by Vizier (Golovin et al., 2017). All other hyperparameters were decided upon using grid search. During training, we used a maximum sequence length of 96. For decoding, we used beam search with beam size 6, and set the length penalty alpha to 0.8, and a maximum decoding length of 384. Multi model ensembling was done via softmax output averaging as described in (Freitag et al., 2017).

Data Augmentation
In this section, we introduce how we augment data based on the given bilingual data and monolingual data. When we train the proposal and channel models, we use all the augmented data along with the original bilingual data.
Back-translation We perform back-translation from monolingual English data using fine-tuned channel models (English→Chinese) with top-k sampling following (Edunov et al., 2018) with k = 50 during decoding. We used the same indomain monolingual data as described in §5.2. We score the back-translated data with fine-tuned proposal (Chinese→English) models, and filter them based on the quantiles of length ratios, sequence log-probability and cross-entropy between one-hot empirical translations and logits from the scorer model. The filtering helped to reduce the size of data from 43.4M to 29.9M paired sentences.
Forward translation to generate synthetic parallel documents We applied a version of our system to monolingual Chinese documents from Gigaword to get synthetic English documents. We only kept documents having between 4 and 25 sentences, we rejected outliers according to their probabilities under the language model, the channel model, and to the overall objective. These were then used to train subsequent versions of the forward (Chinese→English) models.

Data distillation
We use knowledge distillation (Kim and Rush, 2016) to do distillation on the original dataset. Specifically, we translate the source-side of the bilingual data using previously trained proposal models (including Right-to-Left (Liu et al., 2016) and Left-to-Right models) and generate distilled candidates. The generated sentences are filtered if BLEU scores are below 30 (Wang et al., 2018;Sun et al., 2019). We then train models on the filtered data along with the original bilingual data and back-translation data. We repeat this process three times using models trained on newly generated data from the previous iteration.
We empirically do not find Right-to-Left models significantly differ from Left-to-Right models in performance. Qualitatively we find that distilled data correct few errors in the original bilingual data.

Fine-tuning
Fine-tuning with in-domain data has been an effective approach for improving translation quality as shown by existing work (Sun et al., 2019;Ng et al., 2019). After training the proposal models with the mix of real and synthetic parallel data, we fine-tuned the models with CWMT and a subset of newstest2017 and newstest2018 which were not used for validation.

Improving Uncertainty Estimation
To improve the robustness of noisy channel reranking, we explore two approaches for improving uncertainty estimation of the seq2seq scoring models.

Adversarially Trained Proposal Models
To simulate different wordings and noises in source and candidate sentences, we follow Cheng et al.
(2019) to train the models on noisy adversarial inputs and targets. We use bidirectional languagemodels to provide the noisy candidates and select the candidates with highest loss (i.e., adversarial source-target inputs). During the training, we optimize the original loss with clean source-target pairs, the language model losses for source and target sides, and the adversarial loss using adversarial source-target inputs. In the final scoring, we use an ensemble of eight adversarially trained models with few differences from Cheng et al. (2019): (a) We explore training with and without the language model losses. Though the models trained without the language model loss generate quite noisy sentences, we empirically find this approach still helps the overall performance. (b) In addition to using the clean hard-labels for the noisy source-target pairs for the adversarial loss as in the original work, we explore a variation using a KL loss between the adversarial source-target logits and the clean source-target logits. We find this variant also improves the overall performance.

Contrastive Channel Models
When scoring candidates, we want the channel models to be sensitive to translation noise (dropped words, permutation, or blanked words) (Edunov et al., 2018). Hence, we develop contrastive training (Yang et al., 2019;Welbl et al., 2020) to train the models such that it will be more robust in estimating the channel probabilities. Specifically, we use n-gram Transformers (Chelba et al., 2020) with the contrastive loss: (5) wherex denotes a noisy version (random word deletion, blank, or permutation) of x, and p(x | y) is the perturbed loss term. We ensemble 8 models with a few variants for final channel model scoring. These variants consist of the followings: (1) We use η = {0.01, 0.001}.
(3) We use models with perturbing source sentences p(x | y) and models with perturbing target sentences p(x |ỹ). (4) Instead of using the contrastive loss, we include two models trained to minimize the perturbed loss terms directly. (5) Unlike Yang et al. (2019), where the authors firstly train with the maximum likelihood objective and then finetune with the contrastive loss, we find it empirically works better to train models with linearly increased weights (increasing from 0 to 1 during training) to the contrastive loss (Eq. (5)) along with the original negative log likelihood loss.

Filtering Candidate Translations
After obtaining candidate translations from strong proposal models, we filter out candidates with length ratio outside of [e −1 , e 1 ], or which do not end with end-of-sentence punctuation when the source does, or with more than 4 consecutive identical tokens, or which are excessively compressible, indicating repeated contents, according to the following. We learn a piece-wise linear ordinary least squares model of the zlib-compressed length of true English sentences from their uncompressed length in UTF-8, using the English side of the training data. We then reject candidates the actual compressed length of which is more than 12 standard deviations below their predicted compressed length.

Language Model
In this section, we describe the architecture of the language models we used and how we trained them.

Model
The auto-regressive document language model is a Razavi (2020), we also used 4-layer blocks of short and long (128-128-128-512) attention memories, capturing short-range correlations in the earlier layers and long-range correlations in the later ones. This led to a 20% speedup of training, and helped the model generalize better to the validation set. We also used knowledge distillation in our Transformer-XL model with a setup similar to Born Again Neural Networks (BANN) (Furlanello et al., 2018), where we regularize the original loss function with term based on the cross-entropy between the new models outputs (student) and the outputs of the original (teacher) model. Let L denote cross entropy loss function, y onehot encoded label, s and t outputs of the student and teacher model respectively, then the BANN loss is defined as follows: We trained our student network on the loss function in Eqn. 6 and found that λ = 1 had the best validation perplexity.

Data
The English data used to train our language models was prepared as described in §3.

In-domain document data for LM training
We found that training LMs on a subset of training data that was more closely aligned with the validation set vastly improved the perplexity on the validation and test sets (≈ 10%). To select a well-aligned subset of training data, we ranked the training data according to TF-IDF similarity with each validation document and collected the top 1,000 documents for each validation query together, to form our training data for the LM training. We also tried mixing this sub-sampled in-domain data with the raw data using different weights (essentially equivalent to up-weighting the in-domain data) and found that using purely in-domain data outperformed all other mixing schemes in terms  of held-out perplexities and thus, the in-domain data became our training dataset. As an auxiliary benefit, the model trained on the in-domain dataset (340K iterations) also converged much earlier than the one trained on the raw dataset (500K iterations).
Similar to the sequence model fine-tuning outlined in §4.3, we also fine-tuned our trained language model in order to align the model more closely with the language constructs and domain information in our test data. Table 1 shows the perplexity numbers on the validation set obtained by different train data and model variants described above on the validation dataset.

Experiments and Results
We use the original Chinese subset of newstest2017 and newstest2018 as our validation set and new-stest2019 as our test set.
The candidate translations for the reranker are generated by 8 ensemble models (6 from each). Table 2 presents the results of our models on the test set. We report case-sensitive SacreBLEU scores (Post, 2018). Both data augmentation and fine-tuning significantly improve the performance. Ensembling and noisy channel reranking gives about 0.8 and 0.6 BLEU boost, respectively. Finally, our specialized methods for handling long sequences (described in §A) yield a further 0.8 BLEU improvement.
In our final submitted system, we tune the weights of component models and the hyperparameters of sentence segmentation models using a combination of our validation set and test set. For the candidate translations of the reranker, apart from the existing 48 proposals generated by 8 ensemble models, we include additional 48 proposals generated by 8 ensemble models which are finetuned with CWMT, newstest2017, newstest2018, and newstest2019. We also include translations generated by MCTS decoding ( §B) in our nonprimary system. We find that adding a feature marking the length of source sentences longer than 60 words helps the reranker handle long sentences better. We therefore include this feature in addition to proposal probability, adversarial proposal probability, channel probability, language model probability, and length bonus (Eqn. 3).
Our system achieves a 35.4 BLEU score on new-stest2020.

Conclusion
This paper describes the DeepMind submission to the WMT2020 news Chinese-English translation task. Using the noisy channel model (Yu et al., 2020) as our core document translation system, we optimized its component models using data augmentation, fine-tuning with in-domain data, MCTS decoding ( §B), and knowledge distillation. We also addressed premature termination in long sentences by training specialized length expert models and segmenting long sentences into multiple shorter sentences ( §A). We have demonstrated the marginal contributions of these methods in our analysis and our final system comprising all these methods outperforms the Transformer baseline by 9.9 BLEU points on newstest2019. In the appendix, we additionally include the description of our specialized methods for handling long sequences and the MCTS decoding algorithm. The candidate translations generated by MCTS decoding are added to the candidate pool of the noisy channel reranker in our non-primary system.

A Length Considerations
The WMT Chinese evaluation data presents documents as a sequence of segments. These segments are sentence-like units that were delimited in the original article by either unambiguous structural transitions, such as the end of a headline, or unambiguous end-of-sentence punctuation (.!?). However, in some contexts, a complete Chinese sentence may be ended with a comma (Xue and Yang, 2011). This leads to disproportionately more segments in the evaluation data being multi sentence than in the training data, which consists primarily of paired sentences, phrases, and words. Since generalization from short sequences to longer ones is a weakness of neural sequence to sequence models (Lake and Baroni, 2018), to ensure that our candidate pool contains adequate translations of long sentences, we had special handling for long sequences.

A.1 Length Analysis
There is a strong linear relationship between the number of Chinese words in the source segment and the number of English words in the translation (Figure 1 Left). The translations from our initial system were able to match this relationship when the source segment contained less than 60 words. After this point, the translations became too short. Inspection of these translations showed that the primary cause of failure was emitting the EOS token too early. Since the translations were good up to the point of truncation, we focused on methods to prevent early termination. Our final pool consisted of candidates from our original proposal models only for sentences that had less than 80 words and the rest were generated from the techniques outlined here.

A.2 Length Experts
We trained a number of length expert models with a sequence length of 384 tokens. As few (≤ 1%) sentences in the training data were longer than our default sequence length, we used a mixture of real parallel data, synthetic data as described in §??, and concatenation of consecutive synthetic sentences to train these length experts. In addition, we also used the original proposal models which were further fine-tuned with long sequences(≥ 60 tokens) as additional length experts. These specialized models to handle longer sentences were used to generate proposals for sentences between 60 and 100 words.

A.3 Sentence Segmentation
We found that a lot of the long (≥ 60 words) sentences in our dataset had complete sentences concatenated with commas, semicolons, full-stops, exclamations and question marks. While the latter ones are all conclusive end of sentences, commas are ambiguous as an end of sentence. Hence, we built a comma classifier that distinguished commas that signify end-of-sentence from the normal commas. While training data for this classifier was generated as outlined in (Xue and Yang, 2011), our classification model had feed-forward layers on top of the Transformer encoders (further fine-tuned on this task) that we trained for our translation task. During inference time, we recursively split every sentence on standard end-of-sentence punctuation and then semicolons followed by terminal commas (as determined by the comma classifier), into reasonably sized segments (10-60 words); Very short segments (< 10 words) were merged with their neighboring segments. After this first wave of splits, if long segments (≥ 60 words) still persisted, we further split them recursively on all colons, commas and reverse commas into segments between 40-60 words. Each segment was then translated independently using our translation models. This segmentation procedure was used to generate proposals for all sentences that had more than 60 spaceseparated words.
Sentence remerging model For each split of a sentence, we obtain a list of candidate translations, and need to combine this. For n splits with k translations each, we have k n translations in total. Provided n and k are reasonable, we can enumerate these, but it is still too many candidates to present directly to our global reranker, so we need to have a "local" reranker that will select the best k of these.
To select these, we define a reranking model in terms of our usual features (language model log probability for the remerged sentence to ensure coherence, channel log probabilities for the remerged candidate, sum of the direct translation log probabilities for each segment, and the total length). We Figure 1: Relationship between the number of white space separated words on normalized text translation pairs. Left is the ground truth for our validation and test datasets. Middle is our initial candidate pool for this data while Right is our pool including candidates from length expert models and our segmentation technique. The black line is the maximum likelihood fit under |y i | ∼ N (β · |x i |, σ) with the dotted grey dashed lines representing the 3σ bounds.
select the remerged candidates k maximizing an approximate lower bound of the corpus-BLEU.
Approximation to BLEU The weights of the features are learned so as to minimize an approximation to the expected negative log BLEU score. For the reference sentence y and hypothesisŷ, the negative expected log BLEU score is defined as: where c i is a function that counts the clipped igram matches against a reference, and r i counts the i-grams in a reference (Papineni et al., 2002).
Although our model assigns probabilities independently to sentences, BLEU is defined on an entire corpus, and because of the nonlinear functions in BLEU, we cannot compute this expectation tractably. We therefore approximate it by moving the expectations inside the nonlinear functions: In this approximation, the corpus-level expectations for i-gram counts and the length can be computed tractably using the linearity of expectation. To obtain a learning algorithm, we differentiate this quantity with respect to the weightings of the scoring models and perform gradient descent.
Even with our small number of features, this objective has many local optima and in practice we run the optimizer starting from different positions and find a solution that obtains a high BLEU score and highly weights the language model (during development, we noticed that a higher weight to LM probabilities corresponded to noticeably more fluent translations, even if there was little difference in BLEU).

B MCTS Candidates
The candidate pool generated by the sequence to sequence model optimize the search space that is favorable according to those models and may fail find certain translations that score poorly according to sequence to sequence models, but receive high scores from the noisy channel model. In order to include such translations into the candidate pool, we employ Monte Carlo Tree Search (MCTS), to optimize the noisy channel objective directly. MCTS does not require a partial translation evaluation function, as opposed to Beam Search, making it an appropriate choice for decoding non-factorizable objective functions.
We define the translation environment as a progressive left-to-right language generation process, where each state s y defines a sequence of tokens y, and each action a w appends a word type w at the end of the sequence generating a new state s y . When an action appends the end of sentence token, a terminal state is generated. The reward R for terminal states corresponds to a log-linear combi-nation of model scores φ ∈ M as follows: where λ denotes the weights attributed to each of the models. Our MCTS decoder is composed of two optimization processes running simultaneously.
The former aims at finding the optimal state R(s y ) for each of the sentences in the validation and test sets. The latter maximizes the correlation between the BLEU score on the validation set by optimizing the weights λ.

B.1 Monte Carlo Tree Search with Log Linear Models
MCTS decodes a sentence by growing a search tree. Each node in the tree corresponds to a state but adds additional statistics in order to optimally expand the search tree. Expansion is achieved by applying actions to existing nodes in the tree, generating child nodes. The search process starts with a tree with root node, which corresponds to an empty translation, and gradually expands the tree by append new words to existing nodes in the tree. Each MCTS iteration performs the following four steps: selection, expansion, simulation and backpropagation.
The selection step aims at choosing the most likely node in the tree to generate the optimal translation. We employ a standard criteria UCT (Upper Confidence Bounds for Trees (Kocsis and Szepesvri, 2006)), which selects nodes recursively starting from the root according to the following criteria: where s denotes the current node and s is the parent node. N (s) denotes the number of times s was traversed by the selection process and Q(s) denotes the average reward obtained from s in the N (s) traversals. b is a constant that quantifies the trade-off between exploitation and exploration. We set it to b = 1 in our experiments. In general the average value Q(s) is computed by accumulating the reward V (s) obtained one any traversals containing s, then computing Q(s) = V (s) N (s) . However, as our reward function (Eqn. 7) is concurrently updated by optimizing the weights λ, we store the accumulative individual scores V i (s) of each of the models φ i . Prior to a MCTS iteration, we update λ and compute Q(s) as follows: This allows changes to the weights to be directly reflected in the entire search tree without the recomputation of any tree statistics. As for models M , we trained 9 proposal models and 7 channel models with the architecture defined in §4.1, and 8 language models with the architecture defined in §5.1.
Once a node s is selected, a new child s is added to the tree in the expansion step.
The simulation step attempt to compute the expected reward for s . This is generally accomplished by performing multiple random rollouts starting from state s , where actions are sampled from an uniform distribution until a terminal state is found. These states are then scored according to Eqn. 7. The estimate of the expected reward of s is computed as the mean of the scores of different rollouts. However, the sparsity underlying natural language generation and the computational complexity of the scoring function makes this practice computationally challenging. Rather than performing multiple rollouts that sample from an uniform distribution at each timestamp, we perform a single rollout that runs greedy decoding starting from the translation prefix defined from state s . As it is computationally expensive to perform decoding for each MCTS iteration, use a light-weight proposal model trained on the same data, but with a simpler architecture. For this purpose, we reduce the architecture described in §4.1 into a single layer seq2seq layer with 128 hidden units. While this network underfits the data, we found that this trade-off is desirable as it reduces the large dimensionality of the vocabulary to the few examples that are sensible at each prefix leading to a significant speed-up. The quality of the found translation remains unaltered as the reward is still computed with the full models.
Finally, each of the model scores V i (s ) is propagated to all nodes from the root to s in the backpropagation step.

B.2 Pairwise Reranking Optimization
In order to optimize the weights λ in Eqn. 7, we we employ the weight optimization method for log-linear models described in (Hopkins and May, 2011), which allows us to optimize our log-linear model with respect to the non-differentiable objective function that is BLEU. This method, denominated as PRO, approximates the objective function by training a binary classifier, such that two translations y and y respect the following equality: where g is the objective function, namely BLEU. Thereby, this requires the generation of pairs y and y , where the LHS property holds. These samples are then used as data to train the model defined in the RHS.
We generate the data by sampling from the MCTS tree for each sentence pair in the development set. As Q(s) is the expected score of a log-linear model with probabilities as components φ, we expect that Q(s) is bounded in the [0, 1] interval. Thus, at node s , the probability of sampling child s is given by Q(s) c∈C(s ) Q(c) , where C(s) denotes all children of node s . Once a leaf node is found, if it's terminal we sample its translation, otherwise we sample the translation obtained from its rollout.
Both MCTS search and PRO optimization are executed in parallel, as the former needs optimal weights in order to optimally grow the search tree, and the latter needs the search tree to generate candidates. Thus, at each iteration, the PRO optimizer samples from the most updated version of the tree for each data point in the development set, and updates the set of weights λ, which are then used in the subsequent MCTS iterations.


rrSDS: Towards a Robot-ready Spoken Dialogue System

Spoken interaction with a physical robot requires a dialogue system that is modular, multimodal, distributive, incremental and temporally aligned. In this demo paper, we make significant contributions towards fulfilling these requirements by expanding upon the ReTiCo incremental framework. We outline the incremental and multimodal modules and how their computation can be distributed. We demonstrate the power and flexibility of our robot-ready spoken dialogue system to be integrated with almost any robot.

Introduction
Spoken Dialogue Systems (SDSs) are well-suited to handle complex artifacts of dialogue such as hesitations and clarification requests in many domains. However, to further extend SDSs to work effectively on physical robots, we offer the following additional requirements towards a robot-ready SDS: modular: robot components are modular and individual modules must be able to integrate with SDS modules, multimodal: robots are situated dialogue partners whose many sensors must be integrated with the SDS speech input, distributive: robot and SDS modules should easily communicate with each other in a distributed environment, incremental: modules must be able to process input quickly and immediately, aligned: sensors must be temporally aligned, i.e., synchronized in time.
Existing systems offer solutions to some of the requirements. The OpenDial toolkit gives researchers the ability to model dialogue states using probabilistic rules (Lison and Kennington, 2016), but any incrementality has not been systematically evaluated. InproTK (Baumann and Schlangen, 2012), is incremental and InproTK s (Kennington et al., 2014) added distributiveness and multimodality, and Kennington et al. (2017) offered an ap-proach to temporal alignment, albeit with offline evaluation.
The PSI framework is inherently modular, multimodal, temporally aligned, has been evaluated on robot platforms, and has several options for distributing computation (Bohus et al., 2017). However, the PSI framework does not yet build on any incremental framework. Also similar to our work is the platform MultiBot presented in Marge et al. (2019), but that model does not work incrementally nor does it consider temporal alignment.
In this paper, we design and evaluate a modular, incremental, multimodal, and distributive robotready SDS, called rr SDS which is primarily written in the Python programming language. 1 To address the requirements of modularity and incrementality, we adopt the Incremental Unit Framework (Schlangen and Skantze, 2011) where incremental units (IUs) are passed between modules (IUs can be added to reflect new information, or revoked if a previously added IU needs to be updated) by building on the ReTiCo (Michael and Möller, 2019) platform. To handle distributiveness, rr SDS has modules (i.e., ZeroMQ, ROS) that afford interopability with processes outside of the system. To address the requirement of multimodality, we build on top of the modularity requirement and incorporate additional sensors (e.g., cameras and internal robot states).

The rr SDS Spoken Dialogue System
ReTiCo has existing modules for built-in microphones and Google Speech API for speech recognition. We extend it to be multimodal by adding additional sensor modules such as cameras and internal robot states, depicted in Figure 1. We add distributive modules that handle interopability with outside modules. The rest of this section explains the modules for rr SDS.
Dialogue Management OpenDial is a toolkit for developing SDSs with probabilistic rules (Lison and Kennington, 2016) that can be used as a rule-based dialogue manager (DM), but can be extend any domain to include stochastic processes when data is available. We incorporate a recent Python implementation called pyOpenDial (Jang et al., 2019) into our SDS as a DM. Our pyOpenDial module takes any IU payload, expecting a list of attributes (i.e., variables) and values that it adds to pyOpen-Dial's dialogue state as attribute/value pairs.
Natural Language Understanding Words-as-Classifiers (WAC) is a model of grounded semantics that learns a 'fitness' score between physical entities and words (Kennington and Schlangen, 2015), where each word in a known vocabulary is represented as a classifier. WAC is inherently incremental and can learn word groundings with minimal training data. This module takes words from ASR and features from detected objects. It outputs the best fit word for the detected object as well as confidence scores for all the words in its vocabulary and their fitness to all detected objects.

Object Detection & Feature Extraction
This module uses Huang et al. (2017), which builds on several other advances in fast object detection. The output of this module is a list of bounding boxes for each object, along with corresponding labels and confidence scores. The feature extractor takes those object bounding boxes, isolates the bounded object from the rest of the image, and passes that single object image through a pre-trained imagenet model, for example, EfficientNet (Tan and Le, 2019) or InceptionV3, 2 but designers can specify any Keras network and target layer. This module outputs a list of vectors that represent each object that was found in the input image.

Seeed Respeaker
The respeaker is an array microphone with 6 individual microphones on a disclike board. 3 Respeaker has built-in functionality for direction-of-arrival, noise suppression, keyword wake up, and network connectivity. Distributive Interop ZeroMQ is a universal message passing library that builds and maintains sockets that carry atomic messages across various transports. 4 ZeroMQ supports most programming languages and operating systems. The amount of code required to use ZeroMQ to pass messages between separate processes is very minimal. For our SDS, we have two types of ZeroMQ modules: Readers and Writers.
A key interopability module in rr SDS is the Robotics Operating System (ROS), which is widely adopted in the robotics community. 5 ROS has a built-in communication layer across any robotic system's architecture that provides data pipelines on different scopes (Quigley et al., 2009). Our rr SDS interfaces with ROS using Publish and Subscribe modules (similar to ZeroMQ's Writer and Reader modules). We evaluated our implementation using Turtlesim, a common test bed simulation for ROS. 6 Additional Modules rr SDS has additional modules that we do not use in our evaluation, but we do mention them for completeness: Azure Cognitive Services Speech Recognition (ASR), Azure Emotion Recognition API (takes in an image and returns a distribution over 8 emotional states), Azure Object Detector (similar to the MaskRCNN module above, this takes an image as input and returns a list of bounding boxes and corresponding labels), RASA (NLU) (Bocklisch et al., 2017) which has been evaluated to be competitive with commercial NLU platforms (Braun et al., 2017;Liu et al., 2019) Figure 2: Misty in its task setting: Misty could move its head left and right, and had to look down at the objects on the table. and has recently been incrementalized Rafla and Kennington (2019).

Evaluation
We used the Mistyrobotics Misty II and Anki Cozmo robot platforms to evaluate rr SDS, depicted in Figures 2 and 3. We briefly explain the two platforms and the modules we built to integrate them into our rr SDS, then we describe the evaluation.

Robot Modules Integration of Cozmo with
rr SDS is done using its Python SDK and Misty using its REST API, each broken into three ReTiCo modules: (1) camera, (2) internal state, and (3) action control. The output of each camera module is an IU with a still image as its payload. Both robots have internal state variables (e.g., left-wheel-speed, head-height, light-height). As the state of the robot changes, this module produces an IU containing a full attribute-value matrix of the internal state representation (e.g., wheel speed, lift height) at the state update. The action modules use the decisions made by the DM to produce the following actions: explore, align, approach, confirm, and speak.
A human user utters a short description and the robot attempts to explore its surroundings until it finds an object that matches the description. After a user description, the robot enters an explore state to seek out an object, then an align state to move the object into center view. Then the robot confirms if the description matches the object it is looking at. The robot speaks, uttering either That looks X or Uhh that's not X that's Y where X is the description and Y is the robot's best guess at a description (i.e., a better color word).
An overview of our rr SDS is depicted in Figure 1. We use the Respeaker microphone, Google ASR, Figure 3: Cozmo in its task setting: in order for Cozmo to observe objects with it's camera, its head had to be pointed slightly down, and its lift had to be raised. and WAC modules for spoken input, recognition, and understanding, respectively. Each robot's camera passed image frames to the MaskRCNN Object Detection module, then we used the VGG19 fc1 layer (4096 features; pre-trained on imagenet data) to represent objects for the WAC module. For dialogue and action management, we used the pyOpen-Dial module. For the WAC module, we used logistic regression classifiers pretrained on words that only focused on colors. We obtained the training data for WAC by capturing objects using Cozmo's camera; 5-10 training instances per color (trained using l2 normalization).
We recruited 15 participants from Boise State University (4 female, 11 male) to interact with each robot and fill out the Godspeed Questionnaire (Bartneck et al., 2009) after each interaction.
Our rr SDS can run completely on a single machine; 7 output from all system processing modules were logged using PSI on a separate machine. We used the ZeroMQ modules to send information from rr SDS to PSI. We found in our evaluation that participants were able to accomplish the same number of tasks with both robots, but generally found Cozmo interesting, likeable and pleasant whereas Misty was judged as more mechanistic, rigid, stagnant and machine like.

Conclusions & Future Work
Our rr SDS is flexible, being evaluated on multiple robot platforms to create engaging human-robot interactions, and fulfills the modular, incremental, multimoal, and distributive requirements for a robot-ready SDS. Our evaluation of rr SDS allowed users to successfully interact with two different robots to accomplish a simple task with comparable performance. rr SDS is agnostic to the robot platform used, enabling future research to experiment with robot platforms using our flexible system. For future work, we plan to add natural language generation modules and integrate rr SDS more directly with PSI to make use of its architecture, thereby allowing developers and researchers to make use of PSI temporal alignment functionality, but spend most of their development time with Python.


Score-level Multi Cue Fusion for Sign Language Recognition

Sign Languages are expressed through hand and upper body gestures as well as facial expressions. Therefore, Sign Language Recognition (SLR) needs to focus on all such cues. Previous work uses hand-crafted mechanisms or network aggregation to extract the different cue features, to increase SLR performance. This is slow and involves complicated architectures. We propose a more straightforward approach that focuses on training separate cue models specializing on the dominant hand, hands, face, and upper body regions. We compare the performance of 3D Convolutional Neural Network (CNN) models specializing in these regions, combine them through score-level fusion, and use the weighted alternative. Our experimental results have shown the effectiveness of mixed convolutional models. Their fusion yields up to 19% accuracy improvement over the baseline using the full upper body. Furthermore, we include a discussion for fusion settings, which can help future work on Sign Language Translation (SLT).

Introduction
Sign Language is the means of communication of the Deaf, and each Deaf culture has its own sign language. Sign languages differ from the spoken language of the culture. Communication between the Deaf and the hearing relies mostly on the Deaf individual learning the spoken language and using lipreading and written text to communicate: A huge and unfair burden on the Deaf. The reverse, teaching the general population at least some sign language may be more feasible, and there are available educational courses for such aim. However, gaining expertise in sign language is difficult, and the communication problem is still unsolved. Automatic interpretation of sign languages is a necessary step for not only enabling the human-computer interaction but also facilitating the communication between the Deaf and the hearing individuals.
Automatic Sign Language Recognition (ASLR) refers to a broad field with different tasks, such as recognizing isolated sign glosses and continuous sign sentences. The objective of the ASLR system is to infer the meaning of the sign glosses or sentences and translate it to the spoken language. Recently, there has been an increased progress in these efforts: Sign Language Translation (SLT) has become an active research problem for creating interactive sign language interfaces for the deaf [2,1,3,18]. A number of recent papers on the topic made use of neural network generated features. However, while the quality and representative power of these features in SLT are essential, and it is difficult to evaluate the representative potential of the elements in a pipeline setting where the overall system error is cumulative. For this reason, in this study, we aim to evaluate 3D Residual CNN Based Sign Language embeddings in terms of explanatory power in an Automatic Sign Language Recognition (ASLR) setting where temporal mix-up between signs and co-articulation is minimal. For the general case of Isolated SLR, the system aims to process a sign gloss and assign it to a single sign gloss label. In a limited context of supervised learning set-up, labels are glosses, which are transcription symbols assigned by sign language experts. There may be a single signer or multiple signers in communication; however, the ASLR system should be signer independent.
To convey the meaning of a performed sign gloss, Sign Languages use multiple channels, which are manifested as visual cues. We can classify these visual cues into two categories; (1) cues that are denoted as manual cues including hand shape and movement, and (2) cues that are non-manual features including facial expressions and upper body pose focusing on details without definitive large displacements.
Solving the problem of Isolated SLR requires specialized methods, which can be grouped into two categories. The first category is using handcrafted features, focusing on video trajectories and flow maps [29,19,27]. The second set of methods includes machine learning algorithms and neural networks to improve classification performance [14,25,19]. 3D CNN models have proven successful in various video tasks [23,24]. Li et al. [14] adopted the same architecture in SLR and reported improved performance. However,Özdemir et al. [19] provided the comparison of 3D CNN models and handcrafted methods but have found that 3D CNNs are inferior to the state-of-the-art handcrafted IDT approach.
The aim of this work is to investigate why 3D CNN models may fail to show similar success in sign language recognition and to observe what modifications improve their performance. We hypothesize that the performance drop occurs because of the common practice of scaling images into smaller size and sampling frames [23,24], due to computational requirements and difficulty of training bigger neural networks. One solution is handling the negative effect of the sampling by increasing the model complexity as in [6,30,12], yet this increases computational requirements. Instead, we firstly apply attentive data selection at the pre-processing phase by determining cues in SLR data. Secondly, we divide the problem into multiple cues and train different expert classifiers on each kind of dense feature. Thirdly, we refine the expert cue network knowledge into one result, by applying score-level fusion.
The paper organization is as follows. Sections 2 reviews related work, Section 3 explains the presented method, Section 4 presents the experimental re-sults, Section 5 contains the analysis of experiments and Section 6 presents the conclusions.

Related Work
Sign Language Recognition (SLR) aims to infer meaning from a performed sign. In the sign classification task, an isolated sign gloss is assigned to a class label. A sign gloss, the written language counterpart of the performed sign, can be used as a mid-level or final stage label for sign language recognition.
SLR is closely connected with video recognition or human action recognition methods, and similar architectures have been used for both. Two popular approaches to sign language representation uses handcrafted features and deep neural network based methods.
Prior to the performance leap achieved by neural networks, hand-crafted features were the best performing approach for representing human actions in a sequential video setting. For a two-frame dynamic flow map estimation, optical flow is used to generate feature-level information. These features perform better representation than RGB image sequences where the motion information is more indicative than appearance [5]. There exist numerous handcrafted feature extraction methods and their application to image sequences such as STIP [15] and spatio-temporal local binary patterns [28]. State of the art performances with constructed features in action recognition and isolated sign language recognition were obtained using Improved Dense Trajectories [27,19], which is an outlier independent trajectory-based motion specialized feature extractor.
Neural Network based methods focus on convolutional architectures for the classification task. Simonyan et al. [21] use a branched CNN architecture that splits the information into spatial and temporal streams, and fuses them to perform video classification. Tran et al. [23] use 3D convolutional kernels to build a 3D CNN variant to process video data in an end-to-end fashion.
One prerequisite for using deep neural networks is the presence of large datasets with ground truth annotations. Recently, big-scale isolated sign language recognition datasets have become publicly available. Isolated SL datasets contain videos of a user performing a single gloss, usually a single word or a phrase. MS-ASL [25] is an American Isolated SL dataset including 200 native performers performing more than a thousand word categories. WL-ASL [14] is a bigger dataset with two thousand word categories performed by one hundred people. For other languages, Chinese [29] and Turkish [19] are among available datasets. Popular human activity recognition datasets [22,13,10,11] are also used as extra data and for finetuning in Isolated SLR. Continuous SL datasets are acquired in a less controlled setting, where a user can perform longer sign sequences [8,7]. SLR methods often use video pre-processing to reduce network bias and variance, and to increase network performance. Random cropping is one of the popular spatial augmentation techniques when training CNNs. Since CNN variants have small input spatial resolution, e.g., 224 × 224 for the popular ResNet50 network [9], such methods increase the transitive invariance of the models by processing different parts of the image in higher resolution compared to directly downsampling the whole image frame.
Temporal pre-processing techniques operate on the temporal dimension of the video data. The aim is to locate the dense temporal regions which have an increased likelihood of the action flow. In recent work, different approaches are applied for the temporal activity localization, e.g., exploiting both short term and long term samples [26], combining high and low-frequency learners [6], and detecting active window boundaries for the long sequences [17]. Our work differs by applying cue selection before the training phase and combining the classifiers in the feature construction stage.
Combining both pre-processing techniques allows an opportunity to exploit covariance between these spatio-temporal features. Spatio-temporal pre-processing can possibly improve the signal to noise ratio of the processed data when the region of interest is selected from dense regions. This process is shown to be beneficial on other video recognition tasks, e.g., when extracted through handcrafted methods such as optical flow [21], or directly through 3D CNNs [24]. In SLR, due to the nature of the task, SL videos consist of the sparse hand and upper body movements as well as facial expressions. It is possible to use the domain-specific knowledge to exploit spatio-temporal sampling using a guided pre-processing technique. Spatio-temporal multi cue networks [30] exploit spatial regions of interest by firstly using a branch to estimate the region of interest, then training different networks for each unit. However, applying sampling at the training phase becomes more computationally expensive and requires deeper architectures. Our score-level multi cue fusion approach addresses this problem as described in the next section.

Method
In this section, we describe our method. We firstly describe the mixed convolutional model, follow up with our multi cue sampling process, and finally discuss the score-level fusion method.

3D Resnets with Mixed Convolutions
Mixed convolutional networks are 3D Residual CNNs [23], which use 3D convolutional kernels to process video frames in an end-to-end fashion. Tran et al. [24] investigate the success of 3D CNNs and shares two effective variants with strong empirical results. The first is mixed convolutional networks, and the second is residual bottleneck based 2+1 convolutional networks.
The mixed CNN variant builds on the plain 2D residual networks, with the difference that the first layers are replaced with 3D convolutional kernels. While the first layers are capable of processing input video directly with 3D convolutional kernels, later layers efficiently model the semantic knowledge using 2D convolutional kernels. Then, a fully connected layer is employed after the final convolutional layer for the video classification task.
Mixed Convolutional networks are denoted with MCx, where x is the number of 3D convolutional layer blocks. Following the baseline, we empirically experiment with different mixed convolutional variants and employ the MC3 variant of the mixed convolutional network.

Spatial and Temporal Sampling
The message in a sign gloss is conveyed through manual and non-manual cues. Information is conveyed through the shape and configuration of the hand, body, and face regions. The informative regions and intervals can be sampled with the help of a state-of-the-art pose estimation approach such as OpenPose [4]. Making use of pose estimation allows researchers to filter the entire frame by cropping specific regions according to keypoints, which are hand, face, and upper body keypoints in the case of SLR.
We would like to sample informative body regions to increase efficiency, and to filter out noise. Our approach is two-fold; (1) We design a SLR system by extracting the body, hand, and face regions by cropping the RGB frame spatially (in Figure 1) using the pose data which was provided inÖzdemir et al. [19], (2) We focus on the temporal dense regions in which we define the active window as the temporal window where the active hand is moving. Then, we filter out the sparse frames and only feed the network with the frames in the active window (in Figure 2).
Using isolated sign gloss clips guarantees that the temporal sequence is centered on the hand movement. The following steps are used to extract the active window at the center.
1. Use the moving hand detection framework in 4.2 to detect the active hand(s). 2. Define a selected hand as the active hand. If both hands are active, select the dominant hand. 3. For the selected hand, track hand movements using Euclidean distance. Keep the frame ids of the start of the first-hand movement and end of the last hand-movement. 4. Define two thresholds T S , T E . Filter the boundary regions from the start and end frame ids using corresponding thresholds defined earlier, and use extracted frames for the training.
In some videos, the movement is not in the middle of the video. We detect such exception cases by checking the position of the hand relative to the hip. We also filter out segments too short to be a sign.

Multi Cue Score Fusion
Extracting multiple cues from different settings allows each model to build expertise on each cue. Therefore, there is a need to combine the cues of each model   [21] combines different branches while training, but processes the spatial and temporal branch separately at test time using a score fusion approach. They propose firstly direct score fusion via averaging through the network outputs and secondly, training a meta classifier above the extracted features. We follow the former score fusion approach since it has less model complexity and can achieve better run-time performance.
We experiment with two different multi cue fusion settings. First, we apply the averaging operation to the softmax outputs of each cue network results. Secondly, we apply a weighted fusion, where each cue network is weighted by its validation set performance.

Experimental Setup
Dataset To achieve a competitive experimental setting, and to implement our proposal effectively, we have used a recently published Turkish Isolated SLR dataset BosphorusSign22k [19]. The dataset contains 6 different native signers, performing 744 different sign glosses. Each category is labeled with a sign gloss, that describes the performed sign. The dataset contains over 22, 000 video clips. Authors also share 3D body pose keypoints in Kinectv2 format, and 2D body and hand keypoints obtained from OpenPose [4].
Evaluation Metric Following the work ofÖzdemir et al. [19], we aim to compete on the sign language classification task. It is described as estimating the corresponding sign gloss for a given input video at test time, and scoring is evaluated in the accuracy of all of the test estimations. Out of all 6 performers, video clips of User 4 is defined as the test set, which is about 1/6 of the total dataset and it includes samples from all of the 744 classes.
Implementation Details Our experiment setting follows the baseline paper's [19] neural network based experimental setting. We apply the proposed preprocessing pipeline, resize the image into 640 × 360, crop the center square region then resize via bilinear interpolation to achieve 112 × 112 input resolution. Then, we adopt the PyTorch implementation [20] of the mixed convolutional MC3 CNN model which was pretrained on the Kinetics dataset [11]. In our experiments, we only fine-tuned the last 3 residual blocks, and apply uniform frame sampling to input video frames. All experiments has been performed with 32 batch size on a Nvidia 1080TI GPU (with 11GB memory).
Our replicated network resulted in 75.23% accuracy, which is more than 3% lower comparing the reported 78.85% accuracy inÖzdemir et al. [19]. We suspect that the difference is caused by randomized states such as optimizer initialization and different hyperparameter choices such as the learning rate.

Experimental Results
Spatial sampling. Spatial sampling operation is applied through two phases. First, the cue region is detected, cropped, and optionally concatenated in a multiple cue setting. Secondly, sampling is applied using bilinear interpolation. Body Setting. Following the standard SLR pipeline, we crop human body region before training.
Hand Setting. SLR work suggests that the dominant hand, the most used hand, conveys the most information in communication. To detect the dominant hand in the BosphorusSign22k dataset, we employ a hand motion tracking algorithm. The detection process is achieved by the following: 1. Detect the Thumb keypoints on each frame, 2. Define the first thumb keypoint on each hand as two anchors, 3. If the following thumb keypoint on the next frames has greater distance than threshold compared to the anchor, conclude the hand as moving.
To compare keypoints for detecting the dominant hand with threshold values (which is predefined as 150 pixels), we use Euclidean distance. Table 1 provides the detection results on moving hands on BosphorusSign22k dataset. After the detection process, We have seen that signers in the dataset are using their left hands dominantly when performing a sign.
During signing, only one hand may be active, or both hands may be active. We have adopted three different policies; (1) The single cue setting is applied by selecting the dominant hand in which hand crops with 350 × 350 resolution are obtained around the keypoint #2. (2) Both cue setting is applied by selecting both hands where hand crops with 175 × 350 resolution are obtained around the Thumb keypoint, and concatenated horizontally. (3) The mixed setting uses the single cue setting when a single hand is active, and uses both cue setting when both hands are active. All three settings are followed by downsampling with bilinear interpolation. Experimental results are provided at the right-hand side of the Table 1.
Face Setting. Signers often have cues with facial expressions or lip movements (mouthings) that can give hints about the sign gloss. For this purpose, we have also experimented on a face setting where we crop the entire face from frames. To crop the face, we have used the Nose keypoints which are provided

Score-Level Fusion
We follow the insight that the different cue models can capture a different subset of features, which can lead to better results when combined effectively. Standard fusion is applied by averaging softmax outputs as in [21]. In the weighted setting, we have applied weights to each model proportional to their validation accuracy via standard multiplication. Table 2 provides the result of the fusion.
Temporal Sampling Standard SLR training pipeline involves using the standard uniform frame sampling. We propose the active window based temporal sampling, applied by firstly extracting the dense cue regions before applying the uniform selection. Active window is detected as the part that the active hand is moving and discard the rest of the temporal information.
We used double thresholding for finding the active window. We have found that the start threshold T S = 90, and the end threshold T E = 50 generates competitive empirical results. Using the temporal sampling framework, we have successfully segmented the active window for each video. Then, we applied uniform sampling along with our standard training pipeline. Experimental results can be seen in Table 2.
Spatio-Temporal Sampling We applied active window based temporal sampling on top of the spatial multi cue regions. Our experiments have shown that the final spatio-temporal sampling framework has improved on both single cue settings. With the addition of score-level fusion, test accuracy reached to 94.94%, which is the best result in all proposed settings as seen in the Table 2.
Our best setting provides 16.09% improvement on our baseline neural network setting [19]. We also managed to improve their previous best hand-crafted result with 6.41% accuracy rate. Whereas the previous best method uses more Table 3. Comparison with the baseline approaches IDT and MC3 18 model.

Discussion and Analysis
Accuracy lacks informativeness when considering whether the fusion will be beneficial or not. Top-N Accuracy measures how often the Top-N ranks contain the correct class. In our experiments, we also analyze Top-5 Accuracy along with Top-1 Accuracy. Top-N Accuracy results will increase with an increasing N, and are expected to be settled to 1 when N approaches to the maximum class number. Our Top-N accuracy analysis can be seen in Figure 4.
In the plot on the left-hand side, we report Top-N accuracy of the individual cues. Our analysis shows that hand cue yields the best performance, which is followed by the body cue. In both, there is a sharp increase between ranks 1 and 2. This shows that in a large number of cases, although the correct class fails to be predicted, it is the runner-up. This explains why the fusion is beneficial. Although the Top-N accuracy of the face cue is much lower, it is still beneficial for fusion.
Top-N accuracy of the muti-cue fusion is given in the right-hand side of Figure 4. We start by the hand model, then include the body model, and finally add the face model to the mix. This analysis allows us to see the cumulative progress over different fusion models. We observe that the Top-2 accuracy of hand alone is higher than Top-1 accuracy of both fusion settings. We believe that this observation is why the weighted fusion outperforms score fusion, and shows that more advanced models can attain higher performance.

Spatial Ablation Study
To analyse which cue benefits the fusion results the most, we have performed score fusion to all combination pairs of cue settings. According to this ablation study, we were able to observe the effect of each cue to the overall fusion. For example, to find the effect of the face model, we subtract the Body+Hand setting from the Body+Hand+Face setting. Table 4 shows the results of the ablation study. In our analysis, we can see that hand cue has the most effect on the fusion by 9.22% which is followed by the body model with 5.18%.  We have provided an analysis of the two most effective cues by comparing the gloss based performance. As a comparison metric, we adopted the F1-score, which should be more representative of false positives and false negatives, thus is more suitable for the gloss based evaluation.
Gloss Based Cue Comparison We share the top ten sign glosses that the hand cue model has a major advantage compared to the body cue model in Table 5.
In Figure 5, we provide detailed analysis for the IDENTIFY(v) sign gloss. IDENTIFY(v) sign gloss is performed by using only the left hand, touching the head with the index finger, and the rest of the fingers are on the semiopen position. In this particular example, the body cue model only achieves success in the 5th guess, while the hand cue model has the correct prediction.
Additionally, Figure 5 also shows that misclassifications of the body cue model which are HEAD, EAT, PSYCHOLOGY, and PHONE sign glosses. We inspect each confusion as follows:  By evaluating the confused cases, we have concluded that the hand model has an advantage of capturing hand shape information which was possibly due to increased spatial resolution of the hand region. Effect of the Score-Level Fusion. We share our fusion result of the body and hand cue models in Table 5. Data has shown that the fusion model successfully captures the hand cue features. We have also seen that the fusion model even outperforms both single cue models in 7 out of 10 glosses.

Analysis of Method on Types of Gestures Recognized
To further analyze the types of signs which the proposed method performs well and fails, we have labeled the 744 sign glosses in the dataset according to specific sign attributes. The sign classes are grouped into categories such as one-handed signs, two-handed signs, mono-morphemic signs, compound signs, and signs involving repetitive and circular movements of the hands. Table 6 summarizes the analysis: The experiments are performed using temporal sampling with the best performing mixed convolution approach. Attributewise accuracy scores are calculated using the test set samples belonging to the classes containing the selected attributes. Overall, the accuracy scores in Table  6 demonstrate that for nearly all the subsets in the dataset, hand, body, and face-based features show consistency in their relative performance.
Looking at the results for different attributes one by one, we can see that signs involving two moving hands are better recognized than the one-handed sign glosses in the dataset. The performance difference can be explained by the fact that in one-handed signs, the weight of handshape may be more critical than the two-handed signs. The relative positioning and appearance of both hands, which is more apparent, may be easier to represent for the neural network.
Secondly, compound signs have a greater recognition accuracy than monomorphemic signs (95.96% vs 92.21%). Considering the number of signing hands, the amount of additional information in the form of consecutive morphemes present in an isolated sign makes recognition easier, thus improving the performance system. From this result, we can infer that the method's representation power is higher when a sign is greater in length and contains different hand shape and position combinations.
Looking at repetitive gestures, we see a 1.6% improvement in accuracy when the signs do not contain repetitive hand gestures. The issue with repetitions, which we can attribute to this difference, is that the temporal and spatial forms of repetitions are more prone to differ between performances and users, in comparison to the static hand shape parts of the signs that follow specific rules.
Finally, we take a look at circular signs, which include circular hand and arm movements, which involve at least one entire rotation. These signs are dynamic signs where the hands do not stop while presenting a handshape. As these signs do not conform to the movement-hold phonological model of sign languages [16], representing them by choosing temporal frames is more complicated, reducing the effectiveness of keyframe based approaches [12]. Overall, the method performs well with circular signs, making fusion attempts with methods focusing more on the handshape of signs promising future leads.

Conclusion
In this paper, we have proposed a score-level multi cue fusion approach for the Isolated SLR task. Unlike the previous work [19,12], we focused on both spatial and temporal cues. We employed 3D Residual CNNs [24], and trained different models as an expert on the single cue. We distilled the expert knowledge using the weighted and unweighted score-Level fusion. In our experiments, we have seen that our approach has outperformed the baseline results on the Bosphorus-Sign22k Turkish Isolated SL dataset [19].
We have provided the single cue and multi cue Top-N accuracies to demonstrate incremental performance gain with each cue. Our gloss-level study shows that each cue model has specific expertise and provides an indispensable knowledge source to the fusion model. Our analysis of sign gloss attributes hints that the method performs better on temporally more complex signs with two-handed gestures, while performing comparatively worse on mono-morphemic gestures with a single hand. For that reason, the primary approach to improving performance lies in improving hand shape recognition. Possible strategies involve increasing model depth, finding better optimization techniques, or increasing the model input size. We hope that this work will extend the SLR cues into other Sign Language problems, help progress in unresolved SL tasks such as translation, and help uncover language-independent cues. Prob