Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks , which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters. Self-attention networks (SANs) (Parikh et al., 2016; Lin et al., 2017) have shown promising empirical results in various natural language processing (NLP) tasks, such as machine translation (Vaswani et al., 2017), natural language inference (Shen et al., 2018a), and acoustic modeling (Sperber et al., 2018). One appealing strength of SANs lies in their ability to capture dependencies regardless of distance by explicitly attending to all the elements. In addition, the performance of SANs can be improved by multi-head attention (Vaswani et al., 2017), which projects the input sequence into multiple subspaces and applies attention to the representation in each subspace. count all the elements, which disperses the attention distribution and thus overlooks the relation of neighboring elements and phrasal patterns (Yang et al., 2018; Wu et al., 2018; Guo et al., 2019). Second, multi-head attention extracts distinct linguistic properties from each subspace in a parallel fashion (Raganato and Tiedemann, 2018), which fails to exploit useful interactions across different heads. Recent work shows that better features can be learned if different sets of representations are present at feature learning time (Ngiam et al., 2011; Lin et al., 2014). To this end, we propose novel convolutional self-attention networks (CSAN s), which model locality for self-attention model and interactions between features learned by different attention heads in an unified framework. Specifically, in order to pay more attention to a local part of the input sequence, we restrict the attention scope to a window of neighboring elements. The localness is therefore enhanced via a parameter-free 1-dimensional convolution. Moreover, we extend the convolution to a 2-dimensional area with the axis of attention head. Thus, the proposed model allows each head to interact local features with its adjacent subspaces at attention time. We expect that the interaction across different subspaces can further improve the performance of SANs. We evaluate the effectiveness of the proposed model on three widely-used translation tasks: WMT14 English-to-German, WMT17 Chinese-to-English, and WAT17 Japanese-to-English. Experimental results demonstrate that our approach consistently improves performance over the strong TRANSFORMER model (Vaswani et al., 2017) across language pairs. Comparing with previous work on modeling locality for SANs (e.g. Shaw et al., 2018; Yang et al., 2018; Sperber et al., 2018), our model boosts performance on both translation quality and training efficiency. Bush held a talk with Sharon 2 Multi-Head Self-Attention Networks SANs produce representations by applying attention to each pair of tokens from the input sequence, regardless of their distance. Vaswani et al. (2017) found it is beneficial to capture different contextual features with multiple individual attention functions. Given an input sequence X = { x 1 , . . . , x I }  RI  d , the model first transforms it into queries Q , keys K , and values V : Q , K , V = XWQ , XWK , XWV  RI  d (1) where { WQ , WK , WV }  R d  d are trainable parameters and d indicates the hidden size. The three types of representations are split into H different subspaces, e.g., [ Q 1 , . . . , QH ] = Q with Q h  RI  dH . In each subspace h , the element o hi in the output sequence O h = { o h 1 , . . . , o hI } is calculated by o hi = ATT ( q hi , K h ) V h  R dH (2) where ATT (  ) is an attention model (Bahdanau et al., 2015; Vaswani et al., 2017) that retrieves the keys K h with the query q hi . The final output representation O is the concatenation of outputs generated by multiple attention models: O = [ O 1 , . . . , OH ]  RI  d (3) 3 Approach As shown in Figure 1(a), the vanilla SANs use the query q hi to compute a categorical distribution over all elements from K h (Equation 2). It may inherit the attention to neighboring information (Yu et al., 2018; Yang et al., 2018; Guo et al., 2019). In this work, we propose to model locality for SANs by restricting the model to attend to a local region via convolution operations (1D-CS AN s, Figure 1(b)). Accordingly, it provides distance-aware information (e.g. phrasal patterns), which is complementary to the distance-agnostic dependencies modeled by the standard SANs (Section 3.1). Moreover, the calculation of output o h are restricted to the a single individual subspace, overlooking the richness of contexts and the dependencies among groups of features, which have proven beneficial to the feature learning (Ngiam et al., 2011; Wu and He, 2018). We thus propose to convolute the items in adjacent heads (2D-CS AN s, Figure 1(c)). The proposed model is expected to improve performance through interacting linguistic properties across heads (Section 3.2). For each query q hi , we restrict its attention region (e.g., K h = { k h 1 , . . . , k hi , . . . , k hI } ) to a local scope with a fixed size M + 1 ( M  I ) centered at the position i : Accordingly, the calculation of corresponding output in Equation (2) is modified as: As seen, SANs are only allowed to attend to the neighboring tokens (e.g., (cid:98) K h , (cid:98) V h ), instead of all the tokens in the sequence (e.g., K h , V h ). The SAN-based models are generally implemented as multiple layers, in which higher layers tend to learn semantic information while lower layers capture surface and lexical information (Pe-ters et al., 2018; Raganato and Tiedemann, 2018). Therefore, we merely apply locality modeling to the lower layers, which same to the configuration in Yu et al. (2018) and Yang et al. (2018). In this way, the representations are learned in a hierarchical fashion (Yang et al., 2017). That is, the distance-aware and local information extracted by the lower SAN layers, is expected to complement distance-agnostic and global information captured by the higher SAN layers. Mutli-head mechanism allows different heads to capture distinct linguistic properties (Raganato and Tiedemann, 2018; Li et al., 2018), especially in diverse local contexts (Yang et al., 2018). We hypothesis that exploiting local properties across heads can further improve the performance of SANs. To this end, we expand the 1-dimensional window to a 2-dimensional area with the new dimension being the index of attention head. Suppose that the area size is ( N + 1)  ( M + 1) ( N  H ), the keys and values in the area are: (cid:101) K h = (cid:91) [ (cid:98) K h  N 2 , . . . , (cid:98) K h , . . . , (cid:98) K h + N 2 ] (7) (cid:101) V h = (cid:91) [ (cid:98) V h  N 2 , . . . , (cid:98) V h , . . . , (cid:98) V h + N 2 ] (8) where (cid:98) K h , (cid:98) V h are elements in the h -th subspace, which are calculated by Equations 4 and 5 respectively. The union operation (cid:83) means combining the keys and values in different subspaces. The corresponding output is calculated as: o hi = ATT ( q hi , (cid:101) K h ) (cid:101) V h (9) The 2D convolution allows SANs to build relevance between elements across adjacent heads, thus flexibly extract local features from different subspaces rather than merely from an unique head. The vanilla SAN models linearly aggregate features from different heads, and this procedure limits the extent of abstraction (Fukui et al., 2016; Li et al., 2019). Multiple sets of representations presented at feature learning time can further improve the expressivity of the learned features (Ngiam et al., 2011; Wu and He, 2018). Self-Attention Networks Recent studies have shown that SAN s can be further improved by capturing complementary information. For example, Hao et al. (2019) complemented SAN s with recurrence modeling, while Yang et al. (2019) modeled contextual information for SAN s. 2014) to fuse local information, the output of which is fed to the subsequent SAN layer. Several researches proposed to revise the attention distribution with a parametric localness bias, and succeed on machine translation (Yang et al., 2018) and natural language inference (Guo et al., 2019). While both models introduce additional parameters, our approach is a more lightweight solution without introducing any new parameters. Closely related to this work, Shen et al. (2018a) applied a positional mask to encode temporal order, which only allows SANs to attend to the previous or following tokens in the sequence. In contrast, we employ a positional mask (i.e. the tokens outside the local window is masked as 0 ) to encode the distance-aware local information. In the context of distance-aware SANs, Shaw et al. (2018) introduced relative position encoding to consider the relative distances between sequence elements. While they modeled locality from position embedding, we improve locality modeling from revising attention scope. To make a fair comparison, we re-implemented the above approaches under a same framework. Empirical results on machine translation tasks show the superiority of our approach in both translation quality and training efficiency. Multi-Head Attention Multi-head attention mechanism (Vaswani et al., 2017) employs different attention heads to capture distinct features (Raganato and Tiedemann, 2018). Along this direction, Shen et al. (2018a) explicitly used multiple attention heads to model different dependencies of the same word pair, and Strubell et al. (2018) employed different attention heads to capture different linguistic features. Li et al. (2018) introduced disagreement regularizations to encourage the diversity among attention heads. Inspired by recent successes on fusing information across layers (Dou et al., 2018, 2019), Li et al. (2019) proposed to aggregate information captured by different attention heads. Based on these findings, we model interactions among attention heads to exploit the richness of local properties distributed in different heads. We conducted experiments with the Transformer model (Vaswani et al., 2017) on English  German (En  De), Chinese  English (Zh  En) and Japanese  English (Ja  En) translation tasks. For the En  De and Zh  En tasks, the models were trained on widely-used WMT14 and WMT17 corpora, consisting of around 4 . 5 and 20 . 62 million sentence pairs, respectively. Concerning Ja  En, we used the first two sections of WAT17 corpus as the training data, which consists of 2M sentence pairs. To reduce the vocabulary size, all the data were tokenized and segmented into subword symbols using byte-pair encoding (Sennrich et al., 2016) with 32K merge operations. Following Shaw et al. (2018), we incorporated the proposed model into the encoder, which is a stack of 6 SAN layers. Prior studies revealed that modeling locality in lower layers can achieve better performance (Shen et al., 2018b; Yu et al., 2018; Yang et al., 2018), we applied our approach to the lowest three layers of the encoder. About configurations of NMT models, we used the Base and Big settings same as Vaswani et al. (2017), and all models were trained on 8 NVIDIA P40 GPUs with a batch of 4096 tokens. We first investigated the effects of window size (1D-CS AN s) and area size (2D-CS AN s) on En  De validation set, as plotted in Figure 2. For 1D-CS AN s, the local size with 11 is superior to other settings. This is consistent with Luong et al. (2015) who found that 10 is the best window size in their local attention experiments. Then, we fixed the number of neighboring tokens being 11 and varied the number of heads. As seen, by considering the features across heads (i.e. > 1 ), 2D-CS AN s further improve the translation quality. However, when the number of heads in attention goes up, the translation quality inversely drops. One possible reason is that the model still has the flexibility of learning a different distribution for each head with few interactions, while a large amount of interactions assumes more heads make similar contributions (Wu and He, 2018). One intuition of our approach is to capture useful phrasal patterns via modeling locality. To evaluate the accuracy of phrase translations, we calculate the improvement of the proposed approaches over multiple granularities of n-grams, as shown in Figure 3. Both the two model variations consistently outperform the baseline on larger granularities, indicating that modeling locality can raise the ability of self-attention model on capturing the phrasal information. Furthermore, the dependencies among heads can be complementary to the localness modeling, which reveals the necessity of the interaction of features in different subspaces. We re-implemented and compared several exiting works (Section 4) upon the same framework. Table 1 lists the results on the En  De translation task. As seen, all the models improve translation quality, reconfirming the necessity of modeling locality and distance information. Besides, our models outperform all the existing works, indicating the superiority of the proposed approaches. In particular, CSAN s achieve better performance than Model Parameter Speed BLEU (cid:52) TRANSFORMER-BASE (Vaswani et al., 2017) 88.0M 1.28 27.31 -+ BIDIRECT (Shen et al., 2018a) +0.0M -0.00 27.58 +0.27 + RELPOS (Shaw et al., 2018) +0.1M -0.11 27.63 +0.32 + NEIGHBOR (Sperber et al., 2018) +0.4M -0.06 27.60 +0.29 + LOCALHARD (Luong et al., 2015) +0.4M -0.06 27.73 +0.42 + LOCALSOFT (Yang et al., 2018) +0.8M -0.09 27.81 +0.50 + BLOCK (Shen et al., 2018b) +6.0M -0.33 27.59 +0.28 + CNN s (Yu et al., 2018) +42.6M -0.54 27.70 +0.39 + 1D-CS AN s +0.0M -0.00 27.86 +0.55 + 2D-CS AN s +0.0M -0.06 28.18 +0.87 Table 1: Comparing with the existing approaches on WMT14 En  De translation task. CNN s, revealing that extracting local features with dynamic weights is superior to assigning fixed parameters. Moreover, while most of the existing approaches (except for Shen et al. (2018a)) introduce new parameters, our methods are parameter-free and thus only marginally affect training efficiency. To validate the universality of our approach on MT tasks, we evaluated the proposed approach on different language pairs and model settings. Table 2 lists the results on En  De, Zh  En and Ja  En translation tasks. As seen, our model consistently improves translation quality across language pairs, which demonstrates the effectiveness and universality of the proposed approach. It is encouraging to see that CSAN s with base setting yields comparable performance with TRANSFORMER-BIG . In this paper, we propose a parameter-free convolutional self-attention model to enhance the feature extraction of neighboring elements across multiple heads. Empirical results of machine translation task on a variety of language pairs demonstrate the effectiveness and universality of the proposed methods. The extensive analyses suggest that: 1) modeling locality is beneficial to SANs; 2) interacting features across multiple heads at attention time can further improve the performance; and 3) to some extent, the dynamic weights are superior to their fixed counterpart (i.e. CSAN s vs. CNN s) on local feature extraction. Moreover, it is interesting to validate the proposed model in other sequence modeling tasks. The work was partly supported by the National Natural Science Foundation of China (Grant No. 61672555), the Joint Project of Macao Science and Technology Development Fund and National Natural Science Foundation of China (Grant No. 045/2017/AFJ) and the Multiyear Research Grant from the University of Macau (Grant No. MYRG2017-00087-FST). We thank the anonymous reviewers for their insightful comments.
abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain result objective abstain method abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain abstain abstain objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain result objective other abstain abstain abstain abstain other other other other objective method objective other abstain method abstain other other other other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other objective other other abstain abstain abstain abstain abstain objective abstain abstain abstain abstain other other