Multilayer architectures are currently the gold standard for large-scale neural machine translation. Existing works have explored some methods for understanding the hidden representations, however, they have not sought to improve the translation quality rationally according to their understanding. Towards understanding for performance improvement, we first artificially construct a sequence of nested relative tasks and measure the feature generalization ability of the learned hidden representation over these tasks. Based on our understanding, we then propose to regularize the layer-wise representations with all tree-induced tasks. To overcome the computational bottleneck resulting from the large number of regularization terms, we design efficient approximation methods by selecting a few coarse-to-fine tasks for regularization. Extensive experiments on two widely-used datasets demonstrate the proposed methods only lead to small extra overheads in training but no additional overheads in testing, and achieve consistent improvements (up to +1.3 BLEU) compared to the state-of-the-art translation model. Neural machine translation (NMT) has witnessed great successes in recent years (Bahdanau et al., 2014; Wu et al., 2016). Current state-of-the-art (SOTA) NMT models are mainly constructed by a stacked neural architecture consisting of multiple hidden layers from bottom-up, where a classifier is built upon the topmost layer to solve the target task of translation (Gehring et al., 2017; Vaswani et al., 2017). Most works tend to focus on the translation performance of the classifier defined on the topmost layer, however, they do not deeply understand the learned representations of hidden layers. Shi et al. (2016) and Belinkov et al. (2017) attempt  Conghui Zhu is the corresponding author. to understand the hidden representations through the lens of a few linguistic tasks, while Ding et al. (2017) and Strobelt et al. (2018) propose appealing visualization approaches to understand NMT models including the representation of hidden layers. However, employing the analyses to motivate new methods for better translation, the ultimate goal of understanding NMT, is not achieved in these works. In our paper, we aim at understanding the hidden representation of NMT from an alternative viewpoint, and particularly we propose simple yet effective methods to improve the translation performance based on our understanding. We start from a fundamental question: what are the characteristics of the hidden representation for better translation modeling? Inspired by the lessons from transfer learning (Yosinski et al., 2014), we propose to empirically verify the argument: good hidden representation for a target task should be able to generalize well across any similar tasks. Unlike Shi et al. (2016) and Belinkov et al. (2017) who employ one or two linguistic tasks involving human annotated data to evaluate the feature generalization ability of the hidden representation, which might make understanding bias to a specific task, we instead construct a nested sequence of many relative tasks with entailment structure induced by a hierarchical clustering tree over the output label space (target vocabulary). Each task is defined as predicting the cluster of the next token according to a given source sentence and its translation prefix. Similar to Yu et al. (2018), Zamir et al. (2018) and Belinkov et al. (2017), we measure the feature generalization ability of the hidden representation regarding each task. Our observations are (  2):  The hidden representations learned by NMT indeed has decent feature generalization ability for the tree-induced relative tasks compared to the randomly initialized NMT model and a strong baseline with lexical features. The hidden representations from the higher layers generalize better across tasks than those from the lower layers. And more similar tasks have closer performances. Based on the above findings, we decide to regularize and improve the hidden representations of NMT for better predictive performances regarding those relative tasks, in hope of achieving improved performance in terms of the target translation task. One natural solution is to feed all relative tasks to every hidden layer of the NMT decoder under the framework of multi-task learning. This may make the full coverage of the potential regularization effect. Unfortunately, this vanilla method is inefficient in training because there are more than one hundred task-layer combinations. 1 Based on the second finding, to approximate the vanilla method, we instead feed a single relative task to each hidden layer as a regularization auxiliary in a coarse-to-fine manner (  3.1). Furthermore, we design another regularization criterion to encourage predictive decision consistency between a pair of adjacent hidden layers, which leads to better approximated regularization effect (  3.2). Our method is simple to implement and efficient for training and testing. Figure 1 illustrates the representation regularization framework. To summarize, our contributions are as follows:  We propose an approach to understand hidden representation of multilayer NMT by 1 There are about 22 tasks that we have constructed and 6 layers in SOTA NMT models (Vaswani et al., 2017). measuring their feature generalization ability across relative tasks constructed by a hierarchical clustering tree. We propose two simple yet effective methods to regularize the hidden representation. These two methods serve as trade-offs between regularization coverage and efficiency with respect to the tree-induced tasks. We conduct experiments on two widely used datasets and obtain consistent improvements (up to +1.3 BLEU) over the current SOTA Transformer (Vaswani et al., 2017) model. In this section, we first introduce some background knowledge and notations of the multilayer NMT model. Then, we present a simple approach to better understand hidden representation through transfer learning. By analyzing the feature generalization ability, we draw some constructive conclusions which are used for designing regularization methods in Section 3. 2.1 Background and Notations Suppose x = (cid:104) x 1 ,    , x | x | (cid:105) is a source sentence, i.e. a sequence of source tokens, and a target sentence y = (cid:104) y 1 ,    , y | y | (cid:105) is a translation of x, where each y t in y belongs to Y , the target vocabulary. A translation model minimizes the following chain-rule factorized negative log-likelihood loss: (cid:96) mle =  log P ( y | x ;  ) =  (cid:88) t log P ( y t | x , y <t ;  ) , (1) where  denotes the overall parameter of the translation model. According to Eq.(1), an alternative view of the translation problem can be cast to token-level stepwise classification (Daume et al., 2009): predict the target token y t given a context consisting of x and y <t = (cid:104) y 1 ,    , y t  1 (cid:105) corresponding to each factor P ( y t | x , y <t ;  ) . The SOTA multilayer NMT models parameterize P ( y t | x , y <t ;  ) via powerful multilayer encoder and stacked layers of feature transformations (cid:104) h 1 ,    , h L (cid:105) at the decoder side: P ( y t | x , y <t ;  ) = P ( y t | x , y <t , h L ;  ) , (2) where h l ( x , y <t ) =  l (cid:0) x , y <t ; h l  1 ;  (cid:1) is the l th hidden layer recursively defined by  l on h l  1 . We also use h l ( x , y <t ) to represent the output hidden representation of layer l for a specific context. Note that,  l bears several types of instantiation and is an active area of research (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017). Inspired by feature transfer learning (Yosinski et al., 2014), we attempt to understand hidden representations of NMT by evaluating their generalization abilities across any tasks that are related to translation. There are some researchers who study hidden representations of NMT by using linguistic tasks such as morphology, named entity, part-or-speech or syntax (Shi et al., 2016; Belinkov et al., 2017, 2018). They typically rely on human annotated resources to train a model for each linguistic task, so their methods can not be used for languages which lack human annotations. Moreover, their considered tasks are too few to have a good coverage over task space for measuring transferability (Yu et al., 2018; Zamir et al., 2018), and their understanding results may bias to a specific task. As a result, to evaluate the feature generalization ability of hidden representation, we artificially construct plenty of relative tasks which do not employ any human annotation. This makes our evaluation approach more general. Definition of the relative tasks Suppose Y k denotes any partition (or clustering) regarding the output label space (target vocabulary) Y . That is, Y k is a set of subsets Y ki  Y where i = 1 ... |Y k | , such that  i, j, Y ki  Y kj =  and  i Y ki = Y . We define the following relative task: given a context (cid:104) x , y <t (cid:105) , predict the subset or the cluster to which the t th token y t belongs in Y k , denoted as Y k ( y t ) . To simplify notation, we regard Y k both as a relative task and as a partition. It is clear that the above type of tasks are similar to the task of translation according to the description in Section  2.1. Furthermore, different k represents different relative task and thus we actually obtain a great many relative tasks in total. However, it is impossible to evaluate the hidden representation on all those tasks; moreover, due to relationship between tokens (Hu et al., 2016) in Y , not all partitions are reasonable. As a consequence, motivated by the analysis of VC Dimension (Vapnik, 1995), we construct a sequence of nested partitions with an entailment structure: 2 Y 1 (cid:22)    (cid:22) YK . The benefit is that a spectrum of task hardness can be constructed due to the increased partition or task cardinalities. As a matter of fact, we instantiate the above nested partitions through brown clustering (Brown et al., 1992; Stratos et al., 2014) over Y to get a hierarchical clustering tree and then consider each tree depth along the tree as a partition representing a relative task Y k (as shown in Figure 1). In the following experiments, we run brown clustering algorithm over a Ch  En dataset (  4) and construct a tree of English with depth 21. Without loss of generality, we regard the task Y 22 at a virtual 22 depth of the tree as equivalent to the translation task Y . Actually, Y and Y 22 have the same cardinality but are different in definition. 3 Evaluating generalization We use multi-class logistic regression to fit the layer-wise hidden representation learned by a well-trained 6-layer Transformer (Vaswani et al., 2017) over each training instance (cid:104) x , y <t (cid:105) . Specifically, given a context (cid:104) x , y <t (cid:105) , for each task Y k and a hidden representation h l ( x , y <t ) of this context, which is fixed as constant, we predict the cluster Y k ( y t ) according to the following probability: P (cid:0) Y k ( y t ) | h l ( x , y <t );  l Y k (cid:1) , (3) where  l Y k is the parameter of the logistic regression model for task Y k at l th layer. The difference between Eq.(3) and Eq.(2) is that the former is the linear model parameterized by  l Y k while the latter is the NMT model parameterized by  . Since there are L = 6 layers in Transformer's decoder and K = 22 relative tasks, we have more than one hundred such linear models defined with Eq. (3) in total. Therefore, it is costly to train them independently. Since the loss for each linear model is convex, joint training leads to exactly the same optimum as independent training and thus we employ mini-batch stochastic gradient descent to minimize the joint loss as follows:  (cid:88) k (cid:88) l (cid:88) t log P (cid:0) Y k ( y t ) | h l ( x , y <t );  l Y k (cid:1) . 2 Here what we mean entailment relation ( (cid:22) ) between two partitions Y k and Y k +1 is: Y k +1 i ,  ! Y kj , s.t. Y k +1 i  Y kj . 3 Please refer to Appendix A for detailed preprocessing of the tree to get nested partitions. After training, we fix each  l Y k and then measure the feature generalization ability of each h l by validating on the task Y k regarding a heldout dataset, following Yu et al. (2018). For validation, we report accuracy on a heldout dataset through the strategy of maximum a posteriori (MAP). 4 Analysis To figure out how good the learned hidden representations are, we consider two baselines to extract features regarding each context (cid:104) x , y <t (cid:105) to train logistic regression models for comparison. For the first baseline, the features of the context are the hidden representations from the last layer of a randomly initialized Transformer; for the second, the features are derived by lexical feature templates, which include the source-side bag-of-words (BOW) features and target-side BOW features indexed by relative positions of y t 's previous with up to m (Markov length) tokens. 5 As shown in Figure 2, the lexical baseline delivers comparable accuracies for fine-grained tasks with respect to well learned Transformer's first layer, thanks to its discriminant ability with abundant lexical features. For example, its accuracy reaches about 26% for the task with cardinality |Y 21 | . The random baseline performs worse for tasks with cardinality |Y 8 | , which indicates that random representations in NMT have limited generalization abilities to fine-grained tasks as expected. The well-trained low-layer hidden representations yield much higher accuracies than the random baseline and are even better than the lexical baseline. This shows that the hidden representations from a well-trained NMT have good generalization abilities across relative tasks. In addition, as the layer goes up, the performance of hidden representations increase significantly over differ-4 The accuracy is measured by whether arg max z Y k P ( z | h l ( x , y <t );  l Y k )) = Y k ( y t ) . ent relative tasks, which clearly demonstrates that more complex neural architecture leads to stronger expressibility. This provides a quantitative evidence to support the statement in Bengio et al. (2009), Goodfellow et al. (2016). In this section, we propose two simple methods, which respect the above findings, to enhance the hidden representations in NMT such that they generalize well across those relative tasks. A natural method to improve feature generalization of hidden representation is to jointly train the target task with all relative tasks for all hidden layers, which we call full-coverage method. As mentioned in Section  2.2, this method will lead to training more than one hundred tasks ( K  L ) in total, where K denotes the depth of the hierarchical clustering tree (aka. the number of tasks) and L the number of hidden layers. Unfortunately, since each task involves a softmax operation which may be the computation bottleneck for the task Y k with large cardinality, this method is inefficient for training. As a solution to approximate the potential regularization effect of the full-coverage method, we confine each hidden layer to engage in a single relative task. Motivated by the observation that representations from higher layers have better expressibility than lower layers, as claimed in  2.2, we instead employ a coarse-to-fine strategy to select one task for each layer: finer-grained tasks for higher layers while coarser-grained task for lower layers. Specifically, suppose 1  s ( l )  K is the selected index regarding task Y s ( l ) for the l th layer, then it subjects to s ( l ) < s ( l + 1) for each l . In addition, to encourage the diversity among the selected L tasks, we require s ( l + 1)  s ( l ) to be large enough for all l . Formally, the loss of the hierarchical regularization (HR) method is: (cid:96) hr =  (cid:88) l (cid:88) t log P ( Y s ( l ) ( y t ) | x , y <t , h l ; ,  l Y s ( l ) ) , (5) where P ( Y s ( l ) ( y t ) | x j , y j <t , h l ; ,  l Y s ( l ) ) is similar to Eq. 3 except that it treats the parameters  in NMT as parameters besides  l Y s ( l ) . Compared to Eq. 4, it includes fewer terms for summation. The HR method is very simple and computationally efficient, however, using one task to regularize a layer may not be a good approximation of the full-coverage method, since HR method might lead to inconsistent decisions for two different layers, which is formalized through the following entailment structure as introduced in Section 2.2: where s ( l ) is the selected task for the l th layer by HR, 1  l 1 < l 2  L and P ( z | x , y <t , h l ; ,  l Y s ( l ) ) is similar to Eq. (3) for the task Y s ( l ) and l th layer except that it does not treat the NMT parameters  as constant. However, it always occurs on the training data that Y s ( l 1 ) ( y t )  Y s ( l 2 ) ( y t ) . To alleviate this inconsistency issue for better approximating the full-coverage method, we leverage the above structural property by adding another regularization term. Firstly, we project the distribution P ( | x , y <t , h l ; ,  l Y s ( l ) ) into the domain of Y s ( l  1) . Then we calculate KL divergence between the projected distribution and P ( | x , y <t , h l  1 ; ,  l  1 Y s ( l  1) ) . Figure 3 illustrates the idea. Since it is inefficient to consider all pairs of l 1 and l 2 , so we instead consider the consistency between all adjacent layers. Formally, we obtain Method # Param. (cid:96) shr = (cid:96) hr + 1 L  1 (cid:88) l KL (cid:18) P ( | x , y <t , h l ; ,  l Y s ( l ) ) || PROJ (cid:2) P ( | x , y <t , h l +1 ; ,  l +1 Y s ( l +1) ) (cid:3)(cid:19) , (7) where PROJ is the projection defined in Figure 3, and other notations are defined as before. We call the above regularization as structural hierarchical regularization (SHR) since it takes advantage of the structure of the tree. In our experiments, we add HR (Eq.(5)) and SHR (Eq.(7)) losses respectively into the negative log-likelihood regarding Eq. (1) for training all parameters  and  l Y s ( l ) . One of our advantage is that we only use  for testing and thus our testing is as efficient as that for the baseline NMT model. We conduct experiments on two widely-used corpora. We choose from the LDC corpora about 1.8M sentence pairs for Zh  En translation with word-level vocabulary of 30k for both languages. We use the WMT14 En  De task which consists 4.5M sentence pairs and the vocabulary is built by joint BPE with 32k merging operations. Besides the baseline, we also conduct experiments on 3 regularization variants:  Baseline : the Transformer base model proposed in Vaswani et al. (2017). FHR : fine-grained HR based Transformer, which adopts the original label space as task for all selected layers for regularization. This variant is used to demonstrate that low layers which are weak in expressibility can mess up hard tasks which are unsuitable to learn. HR and SHR : as proposed in Section 3. Method MT02 MT03 MT04 MT05 MT06 MT08 Avg. Choice of relative tasks Based on the heuristics in Section 3.1, we first choose the task with the largest cardinality from the hierarchical clustering tree without the virtual depth, because this task is most related to translation (close cardinal-ities). Then we balance task diversity through a 5 times cardinality difference between tasks from the previous chosen task. As a result, we can obtain 4 tasks with s ( l ) = 5 , 8 , 11 , 20 for the Zh  En task and s ( l ) = 5 , 7 , 10 , 21 for the En  De task, where l = 2 , 3 , 4 , 5 of the 6-layer decoder. 6 4.2 Efficiency Comparison Table 1 summarizes the total number of parameters for the baseline and 3 regularization variants. As in Eq. (5), HR introduces extra parameters compared with the baseline. Besides, calculating the second term in Eq. (7) requires modest overheads. Therefore, training our SHR is slower than training the baseline. Although the proposed HR and SHR introduce extra parameters during training, they do not involve them during testing and thus testing is as efficient as the baseline. Table 2 shows the evaluation results of the baseline and 3 regularization variants on the Zh  En dataset. Since there are no recent work reporting Transformer's performance on this dataset, we choose a recurrent SOTA model to show that our baseline is already better than it, which is a common knowledge that Transformer can outperform recurrent NMT models. Our HR method surpasses the baseline 0.6 BLEU point, while the SHR method can improve upon HR by about a further 0.8 point, namely about 1.4 points over the baseline. Interestingly, the FHR method only performs on par with baseline, which indicates that forcing low layers to learn fine-grained tasks will not lead to beneficial intermediate representations since they struggle to learn a well-structured rep-6 Please refer to Appendix C for detailed information. resentation space. This matches the finding in Section 2: low layers may not be expressible enough to perform well on tasks with large cardinalities. In the following, we conduct several quantitative experiments to demonstrate the advantages of our proposed two regularization methods over the baseline. Note that, since we need to guarantee that the decoded sequence has the same length with the reference for one-by-one token comparison, the following experiments are all conducted with teacher forcing and greedy decoding. 4.4.1 Better Feature Generalization Ability In the same manner as Section 2, we learn softmax weights for all relative tasks by fixing model weights learned by HR and SHR methods. Figure 4(a), (b) show the  feature generalization ability (absolute accuracy difference) of HR and SHR over baseline. Since layer 1 is not selected as the regularized layer, no significant gap is observed. However, since layer 1 is close to the loss directly imposed on layer 2, improvements about 5% and 8% are obtained. Since in the baseline, layer 5, 6 are already close or with the ultimate fine-grained loss, HR method shows very small gain. But our SHR method can still improve about 4% absolute points. Except for layer 1, it is also evident to see larger gaps (more than 20%) at lower layers than higher layers due to the fact that lower layers, which are distant from the topmost loss in the baseline, require more supervision signals to shape their latent representation space. We measure decision consistency for a specific layer and decision consistency between a pair of layers using two metrics. The first metric is measured by conditional accuracy, which is the possibilities of the classifier parameterized by  l Y k correctly predicting Y k ( y t ) if the classifier parameterized by  l Y k (cid:48) correctly predicts Y k (cid:48) ( y t ) for any k (cid:48) < k . The second metric is measured by the counts of consistent decision pairs between any pair of regularized layers as defined in Eq. (6). Figure 4(c), (d) shows the absolute conditional accuracy difference of our HR and SHR over baseline. In accordance with the observations in previous subsection, except for layer 1, other layers show significant gains (HR more than 7%, SHR more than 10%) over baseline. Decision consistency for each layer proves the well-shaped layer-wise representation and potentially paves the way for better inter-layer decision consistency. Figure 5 illustrates the consistency counts between any regularized layer pairs, including those without KL-based regularization. Deeper color represents more consistency counts. It is evident that the baseline has a very poor consistency between any layers. Our HR method is almost 2 times better, and the SHR obtains further improvement. A better decision consistency can couple the decision between relative tasks, so that by reaching a high accuracy on easier tasks can benefit the harder ones. Another interesting observation is that non-adjacent layers without KL loss also obtain significant improvements on decision consistency, because the KL term is actually transitive between layers where the predictive distributions are in accordance with the tree structure. In this subsection, we clarify that the coarse-to-fine regularized representations can also benefit low-frequency words. We divide the vocabulary into ten equally-sized bins, and summarize token accuracy for each bin over the development set. As shown in Figure 6, the x-axis represents the frequency spectra, that is, we sort the bins by word frequency from rank 1 (the most frequent words) to 10 (the rare words). We can see that both HR and SHR methods demonstrate a gradually increased gap over the baseline as the word frequency decreases, which means our methods become better for less frequent word bins. However the gap shrinks at the 10 th bin. This may be the fact that for those words that appears with less than 50 counts, both methods are helpless. For baseline, it is hard to train well-shaped hidden representations for low-frequent words; in addition, due to the distance between the loss and the low layers, it is also hard to train weights due to the unstable gradient signal. By adding our regularization terms, every level of the multilayer decoder will receive supervision signals directly and lower layers will receive coarser grained thus higher frequency signals to shape their representations. Table 3 shows the evaluation results of the baseline and the 3 regularization variants on the En  De dataset. Notice that we use the base model while Chen et al. (2018) and Ott et al. (2018) use big models. The FHR method still does not show significant improvement over the baseline (less than 0.2 BLEU point), which verifies the hypothesis that we make by analyzing the Zh  En results. Our HR method is already stronger than Chen et al. (2018) which uses a multilayer RNN as decoder. Compared to the current state-of-the-art in Ott et al. (2018) who utilize huge batch size and over 100 GPUs on the Transformer big model, our SHR method can be on par with them. This comparison indicates that better regularized hidden representations can be potentially powerful than increasing model capacity when using the same optimization method. in the learned hidden representations. Shi et al. (2016) are the first to investigate source syntax encoded in source hidden representations. Similarly, Belinkov et al. (2017) and Belinkov et al. (2018) give detailed analyses of both encoder and decoder's learned knowledge about part-of-speech and semantic tags at different layers. Unlike those works that employ one or two linguistic tasks, we instead construct plenty of artificial tasks without any human annotations to analyze the hidden representations. This makes our approach more general and may potentially lead to less biased conclusions. Based on our understanding of the hidden representations, we further develop simple methods to improve NMT through representation regularization. Many works regularize NMT with lexical knowledge such as BOW (Weng et al., 2017) and morphology (Niehues and Cho, 2017; Zaremoodi et al., 2018), or syntactic knowledge (Kiperwasser and Ballesteros, 2018; Eriguchi et al., 2017). One significant difference is that we take into account the structure among plenty of artificial tasks and design a well motivated regularization term to encourage the structural consistency of tasks, which further improves NMT performance. In addition, our coarse-to-fine way to select tasks for regularization is also inspired by recent works using a coarse-to-fine mechanism for learning better word embeddings in NMT (Zhang et al., 2018) and predicting intermediate solutions for semantic parsing (Dong and Lapata, 2018). In this work, we present a simple approach for better understanding NMT learned layer-wise representations with transfer learning over plenty of artificially constructed relative tasks. This approach is general as it requires no human annotated data, only demanding target monolingual corpus. Based on our understanding, we propose two efficient yet effective methods for representation regularization which further pushes forward the SOTA NMT performances. In the future, we want to dig deeply into the subspace regularities of the learned representations for more fine-grained understanding. The authors would like to first thank all the anonymous reviewers for their critical suggestion and valuable experimental advice. The authors would also like to thank Yong Jiang for discussion; Shangchen Zhou for better figure design; Haozhe Xie, Chaoqun Duan, Xin Li, Ziyi Dou and Mengzhou Xia for proof reading. Tiejun Zhao is supported by National Key RD Program of China Project 2017YFB1002102.
abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain method method objective objective result abstain abstain abstain method method method abstain abstain objective objective objective result abstain method method abstain abstain abstain other method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method method method method method abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other other method abstain objective other abstain method method abstain objective abstain other other other