Attention mechanisms have achieved substantial improvements in neural machine translation by dynamically selecting relevant inputs for different predictions. However, recent studies have questioned the attention mechanisms' capability for discovering decisive inputs. In this paper, we propose to calibrate the attention weights by introducing a mask perturbation model that automatically evaluates each input's contribution to the model outputs. We increase the attention weights assigned to the indispensable tokens, whose removal leads to a dramatic performance decrease. The extensive experiments on the Transformer-based translation have demonstrated the effectiveness of our model. We further find that the calibrated attention weights are more uniform at lower layers to collect multiple information while more concentrated on the specific inputs at higher layers. Detailed analyses also show a great need for calibration in the attention weights with high entropy where the model is uncon-fident about its decision 1 . Attention mechanisms have been ubiquitous in neural machine translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017). It dynamically encodes source-side information by inducing a conditional distribution over inputs, where the ones that are most relevant to the current translation are expected to receive more attention. However, many studies doubt whether highly-attended inputs have a large impact on the model outputs. Smith, 2019), which can be attributed to that unimportant words (e.g., punctuations) are frequently assigned with high attention weights (Mohanku-mar et al., 2020). On the other hand, Jain and Wallace (2019) state that attention weights are inconsistent with other feature importance metrics in text classification tasks. It further proves that attention mechanisms are incapable of precisely identifying decisive inputs for each prediction, which would result in wrong-translation or over-translation in NMT (Tu et al., 2016). We take Figure 1 as an example. After producing the target-side word deaths, attention mechanisms wrongly attribute most attention to the  (cid:104) EOS (cid:105) , making parts of the source sentence untranslated. inputs. To test what inputs affect the model prediction most, we tend to observe how the model decision changes as perturbing parts of inputs. We define the perturbation operation as applying a learnable mask to scale each attention weight. Then, we perform a deletion game, which aims to find the smallest perturbation extents that cause the significant quality degradation. In this manner, we can find the most informative inputs for the prediction. Based on the results detected by the mask perturbation model, we further calibrate attention weights by reallocating more attention to informative inputs. We design three fusion methods to incorporate the calibrated attention weights into original attention weights: (1) fixed weighted sum, (2) annealing learning, and (3) gating mechanism. The mask perturbation model and NMT model are jointly trained, while the attention weights in NMT are corrected based on the actual contributions measured by the mask perturbation model. Recall the example in Figure 1. After producing the target word in, our mask perturbation model finds that the source word    [countryside] with a high attention weight is exactly the decisive input for the prediction. Therefore, we strengthen the corresponding attention weight of    [coun-tryside]. However, after the prediction deaths, the highly-attended  (cid:104) EOS (cid:105)  is not the decisive input at the current step. We redistribute the attention weights to the source words (   [traffic] and    [interruption]) which receive little attention but are important for the subsequent translation discovered by our mask perturbation model. After calibration, the missing source information traffic interruption is well-translated. We conduct extensive experiments to verify our method's effectiveness on Transformer-based translation (NIST Zh  En, WMT14 En  De, WMT16 En  Ro, WMT17 En  Fi, and En  Lv). Experimental results show that our calibration methods can significantly boost performance. We further visualize calibrated attention weights and investigate when attention weights need to be corrected. The contributions of this paper are three-fold:  We propose a mask perturbation model to automatically assess each input's contribution for translation, which is simple yet effective. We design three methods to calibrate original attention weights by highlighting the informative inputs, which are experimentally proved to outperform strong baselines. Detailed analyses show that calibrated attention weights are more uniform at lower layers while more focused at the higher layers. High-entropy attention weights are found to have great needs for calibration at all layers. In this section, we first briefly introduce the framework of Transformer (Vaswani et al., 2017) with a focus on the Multi-head attention (MHA). Then we present an analysis of the learned attention weights, the correlation with feature importance measures, which motivates our ideas discussed afterward. The Transformer is an encoder-decoder framework with stacking layers of attention blocks. The encoder first transforms an input x = { x 1 , x 2 , ...x n } to a sequence of continues representations h = { h 1 , h 2 , ...h n } , from which the decoder generates an output sequence y = { y 1 , y 2 , ...y m } . Multi-head attention between encoder and decoder enables each prediction to attend overall inputs from different representation subspaces jointly. For the single head, we first project h = { h 1 , h 2 , ...h n } to keys K and values V using different linear projections. At the t -th position, we project the hidden state of the previous decoder layer to the query vector q t . Then we multiply q t by keys K to obtain an attention a t , which is used to calculate a weighted sum of values V . Attn ( q t , K , V ) = a t  V a t = softmax (cid:18) q t KT  d k (cid:19) (1) where d k is the dimension of the keys. For MHA, we use different projections to obtain the queries, keys, and values representations for each head. It is noted that Transformer (base model) performs N = 6 cross-lingual attention layers and employs h = 8 parallel attention heads for each time. Thus we implement our methods on N  h attention operations separately. For simplicity, we next denote the query, keys, and values as q t , K , V regardless of what layers and heads they come from. Attention mechanisms provide a distribution over the context representations of inputs, which are often presented as communicating the relative importance of inputs. However, recent work has cautioned against whether the inputs accorded high attention weights decide the model outputs (Jain and Wallace, 2019). Our analysis examines the correlation with attention weights and feature importance metrics in NMT to test if the attention mechanisms focus on the decisive inputs. We apply gradient-based methods (Simonyan et al., 2014; Li et al., 2016) to measure the importance of each contextual representation h i for model output y t :  it = | h i p ( y t | x 1:n ) | (2) We train a baseline Transformer model on NIST Zh  En dataset and extract the averaged attention weights over heads. Figure 2 reports the statistics of Kendall correlation for each attention layer, where the observed correlation is all modest (0 indicates no correlation, while 1 implies perfect concordance). The inconsistency with feature importance metrics reveals that the high-attention inputs are not always responsible for the model prediction. It further motivates us whether we can calibrate the attention weights to focus more on the decisive inputs to achieve better translation. We aim to make the attention mechanism more focused on the informative inputs. The first step is to discover what inputs are essential for the model prediction. As shown in Figure 3, we design a Mask Perturbation Model to worsen the performance with limited perturbation on the original attention weights. By doing this, we can automatically detect what inputs decide the model outputs. Then, we design an Attention Calibration Network (ACN) to correct the original attention weights, highlighting the decisive inputs based on what inputs are perturbed by the mask perturbation model. To search the source-side inputs that the model relies on to produce the output, we can observe how the model prediction changes as perturbing different parts of the input sentence. We apply a mask to scale each input's attention weight, which simulates the process of perturbation. Formally, let m t be a mask at t -th step. where  0 is a uniform distribution (an average vector of 1 n ) and (cid:12) denotes element-wise multiplication. The mask m t is obtained based on the hidden state in the decoder q t and keys K : m t =  (cid:32) q t WQ ( KWK ) T  d k (cid:33) (4) Here,  (  ) is the sigmoid function. A smaller value of m t means a larger perturbation extent on original attention weights. Considering the structure of multi-head attention in Transformer, WQ and WK differ among layers and heads. To test the effect of perturbing distinct regions of inputs, we borrow the idea deletion game to find the smallest perturbation extent, which leads to a significant performance decrease. The objective function of mask perturbation model is: L (  m ) = L NMT ( a pt ,  ) +  L c (  m ) (5) where  denotes the parameters of the original Transformer. LNMT ( a pt ,  ) is the cross-entropy loss of the translation model when using perturbed attention weights a pt . m = { WQ , WK } represents the parameters of mask perturbation model. The first term indicates that the perturbation operation aims to harm the translation quality. The second one serves as a penalty term to encourage most of the mask to be turned off (perturb inputs as few as possible). The perturbation extent is determined by the hyperparameter  . Notably, earlier studies employ masks and deletion game as the analytical tools to explore the importance of each attention head (Fong and Vedaldi, 2017) or the contributions of the pixels in the figure to the model outputs (Voita et al., 2019). However, we extend to probing the inputs' contributions to the model prediction in NMT and further use the masks to calibrate the attention mechanisms based on the analytical results. As aforementioned, our mask perturbation model removes the most informative input to deteriorate the translation by setting the corresponding masks to zero. In other words, a smaller mask means a larger perturbation, namely a more significant impact on the prediction. We propose to calibrate the original attention weights in NMT by highlighting the essential inputs for each model prediction. We increase the attention weights of key inputs which suffer large perturbation extents. The attention weights of other less-informative inputs are correspondingly decreased. We design three methods to incorporate a ct into the original one a t to obtain combined attention weights a combt :  Fixed Weighed Sum . In this method, the calibrated attention weights are added to the original attention weights of fixed ratio  as: a combt = softmax( a t +   a ct ) (8)  Annealing Learning . Considering the mask perturbation model is not well-trained at the early stage, we expect the effect of a ct to be smaller at first and gradually grow with the training step s . To this end, we use annealing learning to control the ratio of a ct as: a combt =  ( s )  a t + (1   ( s ))  a ct  ( s ) = e  s/ 10 5 (9)  Gating Mechanism . the information from the perturbation model in the decoding process. a combt = g t  a t + (1  g t )  a ct g t =  ( q t W g + b g ) (10) where W g and b g are trainable parameters vary among different layers and heads. Our mask perturbation model and NMT model are jointly optimized. As shown in Figure 3, the mask perturbation model is trained to worsen the performance by limited perturbation on the attention weights (Equation 5). Given what inputs are perturbed, we can figure out the decisive inputs for each model prediction and calibrate the original attention weights in the NMT model by ACN. With the calibrated attention weights, the NMT model is finally optimized by: LNMT (  ) =  m (cid:88) t =1 log p ( y t | y <t , x ; a combt ,  ) (11) During testing, the mask perturbation model also helps identify the informative inputs based on the hidden state in the decoder at each step (as seen in Equation 4). The NMT model decodes with the calibrated attention weights. Moreover, our method can provide the saliency map between inputs and outputs based on the generated mask, an accessible measurement of the inputs' contributions to the model predictions. We evaluate our method in LDC Chinese-English (Zh  En), WMT14 English-German (En  De), WMT16 English-Romanian (En  Ro), WMT17 English-Finnish (En  Fi) and English-Latvian (En  Lv). We tokenize the corpora using a script from Moses (Koehn et al., 2007). Byte pair encoding (BPE) (Sennrich et al., 2016) is applied to all language pairs to construct a join vocabulary except for Zh  En where the source and target languages are separately encoded. For Zh  En, we remove the sentences of more than 50 words. We use NIST 2002 as validation set, NIST 2003-2006 as the testbed. For En  De, newstest2013 and newstest2014 are set as validation and test sets. We use the standard 4-gram BLEU (Papineni et al., 2002) on the true-case output to score the performance. For En  Ro, we use newsdev2016 and newstest2016 as development and test sets. For En  Lv and En  Fi, news-dev2017 and newstest2017 are validation set and test set. See Table 1 for statistics of the data. We implement the described models with fairseq 5 toolkit for training and evaluating. We experiment with Transformer Base (Vaswani et al., 2017): hidden size d model = 512 , 6 encoder and decoder layers, 8 attention heads and 2048 feed-forward inner-layer dimension. The dropout rate of the residual connection is 0.1 except for Zh  En (0.3). During training, we use label smoothing of value (cid:15) ls = 0 . 1 and employ the Adam (  1 = 0 . 9 ,  2 = 0 . 998 ) for parameter optimization with a scheduled learning rate of 4,000 warm-up steps. All the experiments last for 150k steps except for small-scale En  Ro translation tasks (100k). For evaluation, we average the last ten checkpoints and use beam search 1 The corpora includes LDC2000T50, LDC2002T01, LDC2002E18, LDC2003E07, LDC2003E14, LDC2003T17 and LDC2004T07. Following previous work, we use case-insensitive tokenized BLEU to evaluate the performance. 2 http://www.statmt.org/wmt14/translation-task.html 3 http://www.statmt.org/wmt17/translation-task.html 4 http://www.statmt.org/wmt16/translation-task.html 5 https://github.com/pytorch/fairseq Model TEST GNMT (Wu et al., 2016) 24.61 Conv (Gehring et al., 2017) 25.16 AttIsAll (Vaswani et al., 2017) 27.3 (Feng et al., 2020) 27.55 (Weng et al., 2020) 27.7 Our Implemented Baseline 27.37 Ours Fixed 27.38 Anneal 28.1  Gate 27.75 Table 2: The comparison of our model, Transformer baselines and related work on the WMT14 En  De using case-sensitive BLEU. Besides, the hyperparameter  in Equation 8 decides how much the calibrated attention weights are incorporated in the Fixed Weighted Sum method. We set  = 0 . 1 in all experiments for comparison. To comprehensively compare with the existing baselines and similar work, we report the results of some competitive models including GNMT (Wu et al., 2016), Conv (Gehring et al., 2017) and AttIsAll (Vaswani et al., 2017) on WMT14 En  De translation task. Besides, we also compare our method against related researches about introducing word alignment information to guide translation (Weng et al., 2020; Feng et al., 2020). As presented in Table 2, our method exhibits better performance than the above models. Unlike supervised attention with external word alignment, our model yields a significant gain by looking into what inputs affect the model's internal training. Table 3 shows the translation quality measured in BLEU score for NIST Zh  En. Our proposed model significantly outperforms the baseline by 0.96 (MT02), 0.84 (MT03), 0.58 (MT04), 1.02 (MT05) and 0.76 (MT06), respectively. We also conduct our experiments on WMT17 En  Fi and En  Lv. As shown in Table 4, our methods improve the performance over baseline by 0.54 BLEU (En  Fi), 0.6 BLEU (Fi  En), 0.57 BLEU (En  Lv) and 0.95 BLEU (Lv  En). For the small-scale WMT16 En  Ro, our methods achieve a substantial improvement of 1.44 more BLEU (En  Ro) and 0.95 BLEU (Ro  En). Com-Model DEV MT03 MT04 MT05 MT06 AVE Baseline 48.56 49.58 48.58 49.95 47.22 48.24 Ours Fixed 48.42 49.41 48.56 50.32 47.89 48.44 Anneal 48.22 49.73 48.85 50.97  47.49 48.74 Gate 49.52  50.42  49.16  50 . pared to the large-scale dataset, the insufficient training data make it harder to learn the relationship between inputs and outputs, leaving a greater need for calibrating attention weights. Overall, our proposed model significantly outperforms the strong baselines, especially for the small-scale dataset. More importantly, the parameter size is tiny (6M), which cannot add much cost to the training and inference process. Effect of Fusion Methods For three fusion methods, the fixed weighted sum has a limited gain. Annealing learning is comparatively more stable, which reduces the impact of ACN when the mask perturbation model is not well-trained at the initial stage. But it is challenging to design an annealing strategy that can be applied to all language pairs. Gate mechanism mostly achieves the best performance for dynamically controlling the proportions of original and calibrated attention weights. Effect of Hyperparameter The hyperparameter  in the loss function of the mask perturbation model (as in Equation 5) decides how much masks would turn on to perturb the original attention weights. Figure 4 exhibits the average value of generated masks across heads as the function of the setting of  . A larger  forces the model to turn off most masks, which makes the value of the mask closer to 1, resulting in a smaller perturbation extent on the attention weights. Correlation with Feature Importance Metrics Figure 5 reports the correlation between our generated mask ( m ) and the gradient-based importance measures 6 (  it ). We find that the masks are relatively closer to the gradient-based importance measures than the original attention weights, which 6 Though these measures are insufficient for telling what inputs are important (Kindermans et al., 2019), they do provide measures of individual feature importance with known semantics (Ross et al., 2017). prove the effectiveness of our mask perturbation model to discover decisive inputs. In this section, we explain how our proposed method helps produce better translation by investigating: (1) what attention weights need to calibrate and (2) calibrated attention weights are more focused or more uniform. Specifically, we delve into the differences between layers, which give insights into the attention mechanism's inner working. We conduct analyses on Zh  En NIST03 and En  De newstest2014 to understand our model from different perspectives. where a = a 1 + a 2 2 . A high JSD means the calibrated attention weights are distant from the original one. Besides, we use the entropy changes of attention weights to test whether the calibrated attention weights become more uniform or focused. (cid:52) Ent ( a 1 , a 2 ) = ent ( a 1 )  ent ( a 2 ) (13) where ent ( a ) =  (cid:80) mi =1 a i log a i , a metric to describe the uncertainty of the distribution. attention layers are not well-trained in the original NMT model and have an urgent need to calibrate. Figure 6 depicts the JSD between original and calibrated attention weights. We find high JSD for high layers and low JSD for low layers in Zh  En task. However, a different pattern is observed in En  De task, where JSD in the high layer is lower than in the low layers. We speculate that the difference is due to the language discrepancy and we will explore this phenomenon in our future work. High or low entropy? More focused contributions of inputs suggest that the model is more con-fident about the choice of important tokens (Voita et al., 2020). We attempt to validate whether the attention weights are more likely to be calibrated when the NMT model is uncertain about its decision. Figure 7 shows the positive relationship between calibration extent and the entropy of attention weights. Take the 6-th attention layer in Zh  En translation as an example (as seen in Figure 7(b)). The averaged JSD is 0.0084 for the attention weights in rang [0,0.8], while the value is 0.0324 for the attention weights where the entropy is larger than 3.2. These findings can also be observed at different attention layers and language pairs. We infer that a higher entropy indicates the NMT model relies on multiple inputs to generate the layer Zh  En En  De 1 + 0.0203 + 0.1846 2 0.011 + 0.0762 3 0.0023 + 0.0207 4 0.0224 0.0336 5 0.0303 0.0595 6 0.0083 0.01 All 0.0336 0.0224 Table 5: Entropy differences ( (cid:52) Ent ) between the original and calibrated attention weights. translation, which increases the probability of information redundancy or error signals. Our proposed model is more likely to calibrate these attention weights to makes the NMT model pay more attention to the informative inputs. There are multiple reasons why the calibrated attention weights can boost performance. Section 4.3 states that our generated masks are much closer to the gradient-based feature importance measures compared with attention weights. On the other hand, we present the entropy differences of the original and calibrated attention weights in Table 5 where the entropy of attention weights are overall smaller after calibration. However, the changes vary across layers. For En  De translation, the calibrated attention weights are more uniform at 1-3 layers and more focused at 4-6 layers, while the attention weights become more focused for all layers except the 1-st layer on Zh  En task. These findings prove that each attention layer plays a different role in the decoding process. The low layers generally grasp information from various inputs, while the high layers look for some particular words tied to the model predictions. The attention mechanism is first introduced to augment vanilla recurrent network (Bahdanau et al., 2015; Luong et al., 2015), which are then the backbone of state-of-the-art Transformer (Vaswani et al., 2017) for NMT. It yields better performance and provides a window into how a model is operating (Belinkov and Glass, 2019; Du et al., 2020). This section reviews the recent researches on analyzing and improving attention mechanisms. The Attention Debate Many recent studies have spawned interest in whether attention weights faithfully represent each input token's responsibility for model prediction. Serrano and Smith flip the model's decision by permuting some small attention weights, with high-weighted components not being the reason for the decision. Some work (Jain and Wallace, 2019; Vashishth et al., 2019) find a weak correlation between attention scores and other well-ground feature importance metrics, specially gradient-based and leave-one-out methods, in various text classification tasks. We also present the correlation analysis in the less-discussed Transformer-based NMT and reach a similar conclusion. As opposed to the critiques of regarding attention weights as explanation, Wiegreffe and Pinter claim that the trained attention mechanisms do learn something meaningful about the relationship between inputs and outputs, such as syntactic information (Raganato and Tiedemann, 2018; Vig and Belinkov, 2019; Pham et al., 2019). Can Attention be improved? There is plenty of work on supervising attention weights with lexical probabilities (Arthur et al., 2016), word alignment (Chen et al., 2016; Liu et al., 2016; Mi et al., 2016; Cohn et al., 2016; Garg et al., 2019; Feng et al., 2020), human rationales (Strout et al., 2019) and sparsity regularization (Zhang et al., 2019). Unlike them, we never introduce any external knowledge but highlight the inputs whose removal would significantly decrease Transformer's performance. Another work line aims to make attention better indicative of the inputs' importance (Kitada and Iyatomi, 2020; Tutek and Snajder, 2020; Mohanku-mar et al., 2020) which is designed for analysis with no significant performance gain, while our methods incorporate the analytical results to enhance the NMT performance. In this paper, we present a mask perturbation model to automatically discover the decisive inputs for the model prediction. We propose three methods to calibrate the attention mechanism by focusing on the discovered vital inputs. Extensive experimental results show that our approaches obtain significant improvements over the state-of-the-art system. Analytical results indicate that our proposed methods make the low layer's attention weights more dispersed to grasp multiple information. In contrast, high-layer attention weights become more focused on specific essential inputs. We further find a greater need for calibration in the original attention weights with high entropy. Our work provides insights on future work about learning more useful information via attention mechanisms in other attention-based frameworks. The research work has been funded by the Natural Science Foundation of China under Grant No. U1836221 and the National Key Research and Development Program of China under Grant No. 2018YFC0823404. The research work in this paper has also been supported by Beijing Academy of Artificial Intelligence (BAAI2019QN0504). This work is also supported by Youth Innovation Promotion Association CAS No. 2017172.
abstain abstain objective abstain objective result abstain abstain abstain abstain abstain abstain abstain method abstain abstain result method objective result result method abstain abstain result abstain abstain abstain abstain result result objective objective result objective objective abstain objective other other other objective abstain abstain other abstain other method abstain other other other method method other other objective method abstain method method method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain method method abstain abstain method abstain method abstain abstain method abstain method abstain method method abstain method method abstain abstain method method abstain method abstain abstain method method other abstain abstain abstain result abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain other other other other other other method other other other abstain abstain method objective result objective abstain result method other other other other