Transformers have advanced the field of natural language processing (NLP) in many ways. At the heart of the Transformer architecture is the multi-head attention (MHA) mechanism which models pairwise interactions between the elements of the sequence. Despite its massive success, the current framework ignores interactions among different heads, leading to the problem that many of the heads are redundant in practice, which underutilizes the capacity of the model. To improve parameter efficiency, we re-formulate the MHA as a latent variable model from a probabilistic perspective. We present c ascaded head-c o lli d ing a ttention ( CODA ) which explicitly models the interactions between attention heads through a hierarchical variational distribution. We conduct extensive experiments and demonstrate that CODA outperforms the transformer baseline, by 0 . 6 perplexity on Wikitext-103 in language modeling, and by 0 . 6 BLEU on WMT14 EN-DE in machine translation, due to its improvements on the parameter efficiency. 1 1 Introduction Transformers (Vaswani et al., 2017) have advanced the field of natural language processing (NLP) on a variety of important tasks, including language modeling (Dai et al., 2019; Baevski and Auli, 2019), language understanding (Devlin et al., 2019; Yang et al., 2019b), and machine translation (Vaswani et al., 2017; Dehghani et al., 2019; Liu et al., 2020). It has also found its place in computer vision (Dosovitskiy et al., 2020), and in intelligent agents (Vinyals et al., 2019) where sequence modeling plays a key role as well. The cornerstone of the transformer architecture is the multi-head attention (MHA) mechanism which models pairwise interactions between the elements of the sequence. An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. A multi-head attention (MHA) mechanism extends the idea through performing multiple separately parameterized attention functions acting in parallel to contextualize the input representations. Their outputs are then gathered by an affine transformation, allowing the model to jointly attend to information from different representation subspaces at different positions. Despite its massive success, the current framework ignores the interactions among different heads, leading to the problem that many of the heads are redundant in practice (i.e., attending to the same regions of the sequence), which underutilizes the capacity of the model (Voita et al., 2019; Michel et al., 2019a). At the same time, recent research (Tang et al., 2018; Clark et al., 2019; Voita et al., 2019; Wu et al., 2020, inter alia ) demonstrates that heads in MHA have the potential to capture distinct information from input sequences, ranging from syntactic and semantic features to alignment information between source and target sentence pairs. These observations suggest that multiple heads should be encouraged to extract complementary information. Therefore, it is highly appealing to take into account the interactions among different attention heads from the perspective of parameter efficiency and the expressiveness of the model. In this work, we introduce head-colliding attention (3). We formulate MHA as a probabilistic model, where each attention head is represented by a latent variable and all of them collide into the observed sequence data (Figure 1a). In this probabilistic graphical model structure, attention heads work as individual factors to explain the data. Although each factor is independent of each other a priori , they interact with each other automatically, conditioning on observations, thanks to the explaining-away effects (Pearl, 1989; Wellman and Henrion, 1993). The head-colliding attention mechanism introduces new computational challenges in training the model. We will discuss how we tackle these using variational methods (Blei et al., 2017). We propose c ascaded head-c o lli d ing a ttention ( CODA , Figure 1b). As our main model, CODA adopts a hierarchical variational distribution (Ranganath et al., 2016) to allow both rich head interactions and effective computations (4). We validate our method in language modeling and machine translation experiments (5). CODA outperforms the vanilla MHA transformer on both tasks, on Wikitext-103 by 0 . 6 perplexity and on WMT14 EN-DE by 0 . 6 BLEU. Further analysis shows that CODA learns to encourage diversity in different heads (Figure 2) and to promote parameter efficiency when increasing the number of heads (5.3). Multi-head attention (MHA) mechanism plays an important role in modern transformer architecture (Vaswani et al., 2017). It extends the classical attention mechanism by running multiple attention function heads in parallel. An MHA module is composed of h identical blocks (usually referred to as attention heads). Each head will generate a hidden state H i based on the input Query, Key and Value matrices, denoted as Q , K , and V respectively. The hidden states from different heads are then aggregated as the output of the MHA module: (cid:80) ni =1 H i W o i , where W oi are model parameters. In the i -th head, the input matrices Q , K and V are first linearly projected into different subspace representations (cid:101) Q i , (cid:102) K i , and (cid:101) V i , based on different learnable parameters. After that, we compute the inner product over all projected queries and keys as the attention logits z i , which are then passed through a row-wise softmax 2 to obtain head attention weights a i : a i = softmax( z i ) = softmax( (cid:101) Q i (cid:102) K Ti ) . As we can see, the core of MHA is to calculate a i in each head. We thus refer to a i as the i -th attention head. In sequence prediction tasks, the model takes as input a source sequence of length m and outputs a target sequence of length n in an autoregressive manner. It predicts each token Y within the target sequence through a categorical distribution p vanilla ( Y | X ) , where X includes the source sequence as well as a previously generated pre-fix. With respect to an MHA block a 1 , . . . , a h , the model predicts target tokens Y by first feeding these heads into a complex non-linear transformation 3 denoted by  (  ) , and then passing it through a softmax function over the entire vocabulary. Therefore, the output probability can be written as p vanilla ( Y | X ) = f ( a 1 , . . . , a h ) , where f ( a 1 , . . . , a h ) := softmax(  ( a 1 , . . . , a h )) . In this section, we introduce head-colliding attention . Specifically, we formulate MHA as a probabilistic model, where each attention head is represented by a latent variable. The name reflects a  collider  in the context of probabilistic graphical models (Figure 1a). We will first explain how head-colliding attention permits the modeling of interactions among different heads and then discuss how vanilla MHA can be viewed as a marginalized version of head-colliding attention, which ignores any head interactions. Considering a single MHA block, we cast each attention head a i as a latent variable. The probability of target Y conditioning on input X can be obtained by marginalizing over all heads A (we denote A := { a 1 , . . . , a h } ): p ( Y | X ) = (cid:90) A p ( Y | A , X ) p ( A | X ) d A = E p ( A | X ) [ f ( A )] . p ( A | X ) is the joint prior distribution. The corresponding directed graphical model is demonstrated 3 Since a transformer typically stacks several attentive layers, for an MHA block in some layer, subsequent layers will induce a non-linear transformation  (  ) for its attention heads. For instance,  (  ) may include several other MHA blocks and feed-forward networks. in Figure 1a, where the links from different heads collide on the observation variable Y . A crucial property of this graphical model is the explaining-away effect (Pearl, 1989; Wellman and Henrion, 1993) of attention heads A when observing the output Y . In other words, if a head a i attends to part of the input which accords well with observation, it immediately discourages other heads from attending to the same part of the input but encourages them to look into complementary information. 4 This mechanism effectively reduces head redundancy and in turn improves parameter efficiency. Vanilla vs. head-colliding attention We now take a closer look at the vanilla MHA (2). Recall that in vanilla MHA, all attention heads are deterministic. From the perspective of latent variable models, this is computationally equivalent to taking expectations of latent head variables. The output probability distribution p vanilla ( Y | X ) can then be expressed as: f ( E p ( a 1 | X ) [ a 1 ] , . . . , E p ( a h | X ) [ a h ]) . This means we are only interested in the individual expectations when using the attention heads in vanilla MHA for predictions. On the contrary, 4 In other words, if we confirm that some head accords well with the observation, then the probability of other heads should be reduced since there is less need to invoke them, according to Occam's razor. in head-colliding attention the distribution of Y is defined as: p ( Y | X ) = E p ( a 1 ,..., a h | X ) [ f ( a 1 , . . . , a h )] . Note the inherent difference of when to take the expectation in vanilla and head-colliding attention. Since f (  ) is a complex non-linear function (2), these two formulations are not equivalent in general and may have a large gap between the two distributions. Concretely, vanilla MHA ignores any possible interactions among different heads. As indicated in equation 2, it first marginalizes out every single head before observing targets  one head will not learn what other heads are attending to despite the fact Y is observed. This is why vanilla MHA is prone to redundancy as many previous studies (Voita et al., 2019; Michel et al., 2019a, inter alia ) discovered. Head-colliding attention, on the other hand, permits rich head interactions due to the expressive non-linear function f (  ) inside the expectation over different latent variables a 1 , . . . , a h . However, the complexity of head interactions also leads to intractability in training the model, which we will discuss in the next section. We train the model by performing maximum likelihood estimation. Here, the log marginal likelihood can be expressed as: log p ( Y | X ) = log E p ( A | X ) [ p ( Y | A , X )] . Unfortunately, this is intractable in general because it requires marginalizing over all possible configu-rations of attention heads. The standard technique is to use variational inference, which optimizes the log marginal by maximizing its evidence lower bound (called ELBO) (Blei et al., 2017): L := E q ( A | X ) (cid:20) log p ( Y | A , X ) p ( A | X ) q ( A | X ) (cid:21) (3) = log p ( Y | X )  KL( q ( A | X ) || p ( A | X , Y ))  log p ( Y | X ) , where q ( A | X ) is the variational distribution 5 over latent variables A . p ( A | X , Y ) is the intractable posterior distribution of all heads given observations Y and the input X , which encodes the rich 5 Although the variational distribution q should depend on target Y in principle, such conditioning renders testing difficult since the target information is not available during testing. For this reason, we only consider the source X hereafter. head interactions we desire, as discussed in 3. Therefore, an ideal variational distribution q ( A | X ) should be close to the true posterior p ( A | X , Y ) . In this case, the samples would accurately reflect the head interactions and the variational distribution would yield a tighter bound to L to facilitate the training. A straight-forward choice of q ( A | X ) is to use the mean-field approximation (Kingma and Welling, 2013): q ( A | X ) = q ( a 1 , a 2 , . . . , a h | X ) = h (cid:89) i =1 q ( a i | X ) . However, it has similar drawbacks as the vanilla MHA. 6 The mean-field approximation assumes the independence of different heads and hence the interactions are greatly limited. Alternatively, one could parameterize q ( A | X ) using an auto-regressive model. 7 Although this is much more expressive, its sequential nature severely slows down training, making it infeasible in practice. Cascaded Head-colliding attention Our solution to this problem is to employ hierarchical structures for head-colliding attention, where interactions among heads could be effectively incorporated into the model (Snderby et al., 2016; Ran-ganath et al., 2016). Conveniently, the hierarchical nature of the transformer architecture offers an effective way of constructing such proposal distributions. Given a transformer with L layers, we denote the set of all attention heads at layer l  1 and l as A l  1 and A l , respectively. Following the bottom-up computation of the transformer, the distribution of A l must rely on the instantiated values of A l  1 . In this sense, A l  1 can be seen as the common variables that govern A l (Figure 1b). Formally, we have: q ( A 1 , ... , AL | X )= q ( A 1 | X ) L (cid:89) j =2 q ( A j | X , A j  1 ) . Despite the fact that each attention head a li  A l at l -th layer is conditionally independent given A l  1 , they become dependent when we marginalize A l  1 6 Note that the vanilla MHA does not define distributions over heads in its original context. out. In particular, the marginal distribution of each A l becomes: q ( A l | X )= (cid:90) A l  1 q ( A l  1 | X ) q ( A l | X , A l  1 ) d A l  1 . This corresponds to an infinite mixture of the mean-field distributions q ( A l | X , A l  1 ) and is able to capture rich head interactions (Ranganath et al., 2016). Our main model adopts this cascaded proposal distribution in figure 1b, and therefore we name it c ascaded head-c o lli d ing a ttention ( CODA ). The only problem left now is how to specify the conditional distribution q ( A l | X , A l  1 ) for all l = 1 , 2 , . . . , L . We first impose the basic constraints on head values as in vanilla MHA, that is, all head values must range within a simplex  n  1 :  n  1 = { A l | n (cid:88) k =1 a li, : k = 1 ,  i = 1 , . . . , h } . Here a li, : k is the k -th column of the i -th attention head at layer l and 1 denotes the vector of all 1's. For efficient training and inference, we adopt Gaussian-logistic distributions (Blei and Lafferty, 2006; Cohen et al., 2008), which not only satisfy the constraints above but also benefit from the effective reparameterization trick (Kingma and Welling, 2013; Rezende et al., 2014; Titsias and Lzaro-Gredilla, 2014). In particular, recall that in vanilla MHA, a i = softmax( z i ) = softmax( (cid:101) Q i (cid:102) K Ti ) (equation 1). We also denote the attention logits at l -th layer as Z l := { z l 1 , . . . , z lh } . For head i at layer l , we first sample from a multivariate Gaussian distribution q ( z li,j : | z l  1 i,j : ) 8 and pass the samples into a row-wise softmax function to yield head values: z li,j :  N (  li,j : , ) , a li,j : = softmax( z li,j : ) , where z l i,j : and a l i,j : represent the j -th row of the i -th attention logit and attention head at layer l respectively. To explicitly model hierarchical structures among attention heads, we propose to add a direct connection between attention heads at adjacent layers (Figure 1b). Such connections offer direct access to the information of attention in the previous layer. Specifically, for each head i at layer l we 8 We only explicitly define the attention logit z as random variables, while the distribution of heads a is induced via a deterministic transformation ( i.e. , softmax function) of z ; therefore it suffices to build dependencies between attentive logits instead. set the mean  i l as the sum of two parts:  i l = (cid:101) Q i (cid:102) K Ti (cid:124) (cid:123)(cid:122) (cid:125) vanilla MHA +  i ( Z l  1 ) (cid:124) (cid:123)(cid:122) (cid:125) direct connection , (4) where  i (  ) is a two-layer multilayer perceptron (MLP) to fuse information from different heads Z l  1 (see the cascading connections in Figure 1b for an illustration). We set the covariance matrix  to the identity matrix for all attentive logits. We give the prior the same form as the variational posterior and parameters are shared between q ( A 1 , ... , AL | X ) and p ( A 1 , ... , AL | X ) for our objective (equation 3). With the help of parameter sharing, the KL term in equation 3 is also cancelled out due to the identical distributions. 9 This choice works well in practice, where it not only allows CODA to use almost the same amount of parameters as vanilla Transformer, but also eliminates the need to invoke advanced training techniques for amortized variational inference. 10 More details can be found in Appendix A. 5 Experiments We conduct experiments on language modeling and machine translation tasks. Datasets First, we conducted experiments for token-level language modeling on a large-scale benchmark dataset Wikitext-103 (Merity et al., 2016), which consists of articles from Wikipedia with the token number around 103M/218K/246K for the training/validation/testing splits respectively. The vocabulary size is 267,744. For machine translation, we consider two standard datasets:  WMT14 EN-DE (Bojar et al., 2014), which contains about 4.5M/3K/3K sentences pairs for train-ing/validation/testing splits respectively. We follow Ott et al. (2018) and Peng et al. (2020) to preprocess the dataset, and obtain a shared vocabulary between source and target language of around 32K byte pair encoding (BPE, Sennrich et al. (2016)) types. 9 Therefore, it can also be derived by directly applying the Jensen's inequality on the log marginal likelihood. 10 For instance, training a standard variational auto-encoder (VAE) for NLP tasks often suffers from the posterior collapse problem due to the heavy KL regularization (Bowman et al., 2016), where some tricks have to be used to achieve good performance, such as KL annealing, etc. IWSLT14 DE-EN (Cettolo et al., 2014). Following standard practice (Edunov et al., 2018; Peng et al., 2020), we pre-process the 160K/7K/7K sentence pairs and build train-ing/validation/testing sets accordingly. This generates a vocabulary of around 9K(7K) BPE types for source(target). Implementation details We implement our model with PyTorch (Paszke et al., 2019) and FairSeq toolkit (Ott et al., 2019). In particular, our model is based on the vanilla transformer architecture (Vaswani et al., 2017). For CODA , we replace all vanilla MHA blocks with the cascaded head-colliding attention, for both self attention and cross attention (if any). In language modeling, we use adaptive input embeddings (Baevski and Auli, 2019) and set context size to 512 and 480 for training and testing respectively, due to constraints of computational resources. In machine translation, we set beam size to 5 and adopt the hyper-parameters from (Peng et al., 2020) for IWSLT14 DE-EN . For WMT14 EN-DE we set beam size to 4, length penalty to 0.6, and average last 10 checkpoints for testing, following Vaswani et al. (2017). Further implementation details can be found in Appendix A. 5.2 Main results The results of language modeling on Wikitext-103 dataset are reported in Table 1. As we can see from the table, CODA barely introduces any additional parameters. However, by taking into account head interactions, CODA significantly outperforms TRANSFORMER by over 0.6 perplexity. For reference, we also report the best setting (denoted by TRANSFORMER  ) in Baevski and Auli (2019), which uses a much larger context size (3072/2560 vs. 512/480 for training/testing), CODA still outperforms by a substantial margin of 0.3 perplexity. This indicates that encouraging head interactions can improve parameter efficiency. To show whether CODA has promoted head interactions and reduced head redundancy, we qualitatively visualize the attention heads in both CODA and TRANSFORMER via heatmaps. Concretely, we compute the Jensen-Shannon Divergence (JSD) between each pair of attention heads at the same layer. In particular, we assume head values define a categorical distribution in both TRANSFORMER and CODA model to facilitate comparison. That is, an Model # Params. attentive head a i induces n categorical distributions for each query position. For the j -th distribution, it indicates how the j -th target position attends to all m source positions and is denoted by p ( x | a i,j : ) . For two heads i and i (cid:48) , we first compute their average distribution as m := p ( x | a i,j : ) + p ( x | a i (cid:48) ,j : ) 2 Then the JSD value between the i -th and i (cid:48) -th attention head is computed by summing all of n induced distributions: n (cid:88) j =1 1 2 (cid:0) KL( p ( x | a i,j : ) || m )+KL( p ( x | a i (cid:48) ,j : ) || m )) (cid:1) We average computed JSDs for all validation samples. Note that a larger JSD value (darker color) indicates that two heads are behaving more differently (i.e. less redundancy between them), and vice versa. As shown in Figure 2, JSD heatmaps in CODA are clearly darker than those in TRANSFORMER . This suggests that CODA permits richer head interactions, which fosters different heads to communicate with each other and encourages them to become complementary. Consequently, our model effectively reduces head redundancy in MHA and improves parameter-efficiency. The results on IWSLT14 DE-EN and WMT14 EN-DE datasets are shown in Table 2. We see that CODA exhibits clear improvements over TRANSFORMER : a 1.1 point gain in BLEU on IWSLT14 DE-EN dataset and a 0.6 BLEU improvement on WMT14 EN-DE dataset. Despite such significant gains over the baseline, CODA only introduce very few additional parameters ( e.g. , 0.03% extra parameters on IWSLT14 DE-EN ). This, again, shows Model IWSLT14 DE-EN WMT14 EN-DE # Params. that CODA is more parameter efficient than vanilla Transformer due to the cascaded head-colliding attention we proposed. Similar to experiments on language modeling, we also visualize the head behaviors to measure attentive head interactions (See Figure 5 and Figure 6 in Appendix B), where we observe similar phenomena on translation tasks. Specifically, different heads in CODA are often complementary to each other and focus on quite different regions of sequences, rather than becoming redundant or even identical as observed in TRANSFORMER models. Despite one would hope increasing the head number in MHA leads to a free-ride in achieving better performance, in practice it is often not the case as vanilla MHA suffers from the problem of parameter redundancy. Following Vaswani et al. (2017), we vary the number of attention heads (4,8,16,32), but keep the amount of computation constant. Our results on IWSLT14 DE-EN are shown in Table 3. We observe that the translation quality of baseline transformer (which uses vanilla MHA as its main building blocks) decreases almost linearly when increasing number of attention heads (Figure 3), which agrees with previous studies (Vaswani et al., 2017; Voita et al., 2019; Michel et al., 2019b). Intuitively, since the total number of parameters in the model remains unchanged, more heads indicate that the number of parameters allocated to each head is reduced, which limits the representational power of every single attention head. Due to the independence assumption between the heads, many of them tend to focus on similar regions of the sequence, leading to a great waste of modeling capacity. In the case of CODA , we observe better BLEU scores in response to the increasing head number. Rich interactions in CODA could encourage different heads to cover broader regions of input sequence, which in turn offers more useful information for training. The perplexity (PPL) reflects 0 2 4 6 Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 Layer 12 Layer 13 Layer 14 Layer 15 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 2 4 6 0 200 400 Figure 2: Jensen-Shannon Divergences (JSD) for each pair of attention heads at all 16 layers on Wikitext-103 validation dataset. In this section, we present an ablation study to investigate effects of different components in CODA . Concretely, we compare four models on the IWSLT14 DE-EN machine translation task: (i) the full model CODA , (ii) a variant of CODA ablating the cascaded structure (4), (iii) a variant of CODA without using head-colliding attention (3) and (iv) the baseline TRANSFORMER model. In more details, for model (ii), we remove the second term in equation 4, which turns off the direct cascading structure, despite still being a proper hierarchical latent variable model 11 . In model (iii) , attention heads are deterministic (instead of being latent variables) as in vanilla Transformers, but cascading connections are incorporated. We observe its close connection with the recently proposed REALFORMER (He et al., 2020), a TRANSFORMER model that adds a residual connection between attention logits at adjacent layers. Since in model (iii) all attention heads are deterministic, it is unnecessary to fuse different heads (see 4). In this case, we simply implement model (iii) as a REALFORMER (and thus referred to as REALFORMER hereafter) to demonstrate the effect of cascading-like structures more clearly. 12 We report BLEU score for translation quality, and the Jensen-Shannon Divergences (JSD) averaged over all heads pairs of all MHA blocks for quantitative evaluation of head interactions. As demonstrated in Table 4 and Figure 4, even without cascading connections for explicit hierarchical structures, head-colliding attention has the ability (albeit limited) to induce reasonable correlations among different heads, reflected in the average JSD. This is due to the explaining-away effects and the native hierarchical structure in the transformers, as discussed in 3. In CODA , because individual heads have access to the other heads from a probabilistic perspective, they are more prone to offering complementary information for each other to jointly explain the observed data. This effect is further enhanced when cascading connections are added to the model. In contrast, if we simply incorporate such cascading connections into a vanilla TRANSFORMER model, we found it does not significantly 11 Note that the first term (cid:101) Q i (cid:102) K Ti in equation 4 also depends on the instantiated value of z l  1 i,j : , which induces an implicit hierarchical dependency for attention between adjacent layers. 12 The main difference between residual connections in REALFORMER and cascading connections in CODA is that, the former directly performs a head-wise addition of previous-layer attention logits; in contrast, our cascading connection makes use of an MLP  (  ) to mix different attention heads, which enhances head interactions for CODA . encourage head interactions and only improves the baseline marginally. In this case, the performance improvement might be mainly due to residual connections, which are often considered to be effective in facilitating training (He et al., 2016). Interestingly, we note a positive correlation between average JSD and BLEU, suggesting that encouraging complementary attention heads may help improve translation quality. Attention mechanisms were first applied to recurrent networks in (Bahdanau et al., 2014). It was then extended to multi-head attention (MHA) and became the key component in transformer architectures (Vaswani et al., 2017). To study the utility of multiple attention heads, Voita et al. (2019) focused on identifying individual contributions of each attention head. Michel et al. (2019a) conducted extensive experiments to demonstrate that pruning out most heads after training does not lead to a drop in performance during inference. You et al. (2020) further revealed that replacing learnable attention heads with samples from fixed Gaussian distributions can achieve almost the same performance as original models. Additionally, Behnke and Heafield (2020) proposed to iteratively prune attention heads during training based on the lottery ticket hypothesis. These works indicate that there is a lot of head redundancy in the MHA transformer architectures. Instead of pruning unnecessary parameters and down-sizing transformer models, there are also works that propose to improve parameter efficiency in transformers. For instance, Li et al. (2018) introduced a regularization term to explicitly promote diversity among different heads. Yang et al. (2019a) proposed to use convolutional kernels to capture correlations among not only local windows of sequences, but also different heads. An et al. (2020) considered each head as a sample from the same distribution, and presented a sampling algorithm that avoids samples from collapsing into local modes. It hence explicitly encouraged the repulsiveness in MHA. Besides, MAE (Peng et al., 2020) converted a vanilla MHA to a mixture-of-experts model, where each expert component activates only a subset of attention heads. With learned probabilities, different experts could be specialized on different inputs. Different from these works, CODA does not explicitly promote head diversity nor specialize different heads. Instead, we focus on studying head interactions from a probabilistic perspective, which reveals the close connection between vanilla MHA and CODA . Another research line relating to our work is to incorporate latent variables into attention modules. Xu et al. (2015) investigated the connection between vanilla deterministic single-head attention and its stochastic counterpart. Deng et al. (2018) explored this further and proposed to use variational inference techniques for training the model. They considered both cases of discrete and continuous latent variables. Bayesian attention modules (Fan et al., 2020) introduced continuous latent distributions for attention that are amenable to reparameterization tricks. Our work is different from them in that we mainly investigate the MHA mechanism and aim to improve parameter-efficiency by recovering potential interactions among different heads, which are ignored in vanilla MHA. Concurrently, He et al. (2020) proposed to add residual connections between attention scores at adjacent layers, similar to our cascading connections. Nevertheless, our motivation for using the cascaded structure is quite different: we aim to construct direct hierarchical dependencies for latent variable models, while He et al. (2020) is mainly motivated to improve transformer architectures and obtain performance gains. We present CODA by re-formulating the multi-head attention (MHA) as a latent variable model from a probabilistic perspective. CODA explicit models of the interactions among attention heads through a hierarchical variational distribution. We conduct extensive experiments and demonstrate that CODA outperforms the transformer baseline in language 0 1 2 3 Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 modeling and machine translation. The analysis shows that CODA learns to encourage the diversity in different heads and to promote parameter efficiency when increasing the number of heads. In this framework, we will be able to impose explicit constraints or regularization on different attention heads in a principal way (e.g. informative priors that promote diversity). Besides, we can also consider more expressive (data-driven) variational distributions. We leave these as the future work. Our code is publicly available at https://github.com/LZhengisme/CODA . We thank the anonymous reviewers whose suggestions helped clarify this work. This research was supported in part by the University of Hong Kong Research Committee under account 104006039.111994.14200.301.01.
abstain abstain abstain result method objective abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method objective method abstain abstain abstain abstain abstain other other other other other other abstain abstain abstain other other other other abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain other abstain abstain method other abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method abstain method method method method method method abstain method abstain method abstain method method method abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other abstain abstain other other other other objective abstain abstain method abstain objective abstain abstain abstain method abstain other other other