The success of language models based on the Transformer architecture appears to be inconsistent with observed anisotropic properties of representations learned by such models. We resolve this by showing, contrary to previous studies, that the representations do not occupy a narrow cone, but rather drift in common directions. At any training step, all of the embeddings except for the ground-truth target embedding are updated with gradient in the same direction. Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models we have empirically tested. Our experiments show that isotropy can be restored using a simple transformation. 1 1 Introduction Word embeddings, both static (Mikolov et al., 2013a; Pennington et al., 2014) and contextualized (Peters et al., 2018), have been instrumental to the progress made in Natural Language Processing over the past decade (Turian et al., 2010; Wu et al., 2016; Liu et al., 2018; Peters et al., 2018; Devlin et al., 2019). In recent years, language models based on Transformer architecture (Vaswani et al., 2017) have led to state-of-the-art performance on problems such as machine translation (Vaswani et al., 2017), question answering (Devlin et al., 2019; Liu et al., 2019b), and Word Sense Disambiguation (Bevilacqua and Navigli, 2020), among others. However, it has been observed that representations from Transformers exhibit undesirable properties, such as anisotropy, that is tend to occupy only a small subspace of the embedding space. The observation has been documented by a number of studies (Gao et al., 2019; Ethayarajh, 2019; Wang et al., 2020). A similar property has been iden-tified in the past in static word embeddings (Mu 1 The code and datasets used in this paper are available at https://github.com/danielbis/tooMuchInCommon. and Viswanath, 2018). To address the issues, postprocessing methods (Mu and Viswanath, 2018), and regularization terms have been proposed (Gao et al., 2019; Wang et al., 2019c, 2020). However, the mechanism that leads to undesirable properties remains unclear. Without understanding the mechanism, it is going to be difficult to address the fundamental issue properly. The deficiencies are most pronounced in the representations of rare words, as we will show in Section 4. Performance of pretrained language models is inconsistent and tends to decrease when input contains rare words (Schick and Schtze, 2020b,a). Schick and Schtze (2020a) observe that replacing a portion of words in the MNLI (Williams et al., 2018) entailment data set with less frequent synonyms leads to decrease in performance of BERT-base and RoBERTa-large by 30% and 21 . 8% respectively. 2 After enriching rare words with surface-form features and additional context, Schick and Schtze (2020a) decrease the performance gap to 20 . 7% for BERT and 17% for RoBERTa, but the gap remains large nonetheless. Why do even the large-scale, pretrained language models struggle to learn good representations of rare words? Consider a language model with an embedding matrix shared between the input and output layers, a standard setup known as weight tying trick (Inan et al., 2017). Intuitively, at any training step t , optimization of the cross-entropy loss can be characterized as pulling" the target embedding, w T , closer to the model's output vector h t , while pushing" all other embeddings, W \ w T , in the same direction, away from the output vector h t . This leads to what we call common enemies effect  the effect of the target words producing gradients of the same direction for all of the nontarget words. Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models 2 Based on the results reported by authors. we have empirically tested; see Figure 1. Although Gao et al. (2019) report a closely related phenomenon and call it representation degeneration , their analysis is based on an assumption that the embedding matrix is learned after all other parameters of the model are well-optimized and fixed, which is not the case in practice. We conduct our analysis in a more realistic setting, and arrive at different conclusions. We show that embeddings do not occupy a narrow cone, but are shifted in one common direction and only appear as a cone when projected to a lower dimensional space (Section 4.1). In fact simply removing the mean vector of all embeddings, thus centering them, shifts the embeddings back onto a more spherical shape. We evaluate embeddings, before and after centering, on four standard benchmarks and observe significant performance improvement across all of them. Why is removing the mean so effective? We find that the common enemies effect applies to most, if not all, words in the vocabulary but in non-uniform manner. As language is known to follow an approximately Zipfian distribution (Zipf, 1949; Manning and Schtze, 2001; Piantadosi, 2014) even common words will not occur frequently in a text corpus, and in result will be often pushed" by other target words in the same direction as rare words. Consequently, all embeddings share a significant common direction. We will focus on the analysis of auto-regressive GPT-2 (Radford et al., 2019) and two masked language models, BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). Our contributions can be summarized as follows:  We show that as word embeddings repeatedly share same direction gradients, they are shifted in one dominant direction in the vector space. The effects are the most evident in representations of rare words, but are also present in representations of frequent words.  The shift causes the distribution of projected embeddings to appear as a narrow cone; we show that simply removing the mean vector is enough to restore the spherical distribution.  We provide empirical evidence of our analyses using state-of-the-art pretrained language models and demonstrate that removing the mean dramatically improves isotropy of the representations. 2 Background 2.1 Distributed Word Representations Distributed representations induce a rich similarity space, in which semantically similar concepts are close in distance (Goodfellow et al., 2016; Bengio et al., 2003; Mikolov et al., 2013c). In a language model, the regularities of embeddings space facilitate generalization, assigning a high probability to a sequence of words that has never been seen before but consists of words that are similar to words forming an already seen sentence (Bengio et al., 2003; Mikolov et al., 2013c). Although models such as BERT or GPT-2 produce representations from a function of the entire input sequence, the representations are a result of a series of transformations applied to the input vectors. Consider an example sentence: The building was dilapidated." , and the sentences resulting from replacing dilapidated" with either ruined" or reconditioned" . If the distance in the embeddings space between the two rather infrequent, but antonymous, words dilapi-dated" and reconditioned" is not larger than the distance between dilapidated" and its relatively frequent synonym ruined" , then by the aforementioned generalization principle there is little to no reason to believe that the distance will become larger in the output layer. 3 2.2 Tokenization Do the subword tokenization methods (Schuster and Nakajima, 2012; Wu et al., 2016; Sennrich et al., 2016; Radford et al., 2019) preserve the word frequency imbalance? Examination of the common tokenization methods, such as Byte-Pair Encoding (Sennrich et al., 2016) and WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016), suggests that subword units induced by tokenization algorithms exhibit similar frequency imbalance to that of full vocabulary. This can be explained by the greedy nature of the vocabulary induction process. Although different methods use different base vocabulary symbols to begin with (i.e., Unicode code points, or bytes), all of the methods construct the vocabulary through iterative merging of the most frequent symbols. As a result, the most frequent units are preserved as words, while the rare words are segmented into subword units. Moreover, the words which are segmented into subword units are 3 In fact, all three sentences are assigned a negative sentiment, with scores between 97% to 100% by RoBERTa finetuned on SST. Masked Language Modeling (MLM) pretraining objective is to maximize the likelihood of masked tokens conditioned on the (noisy) input sequence. Given a sequence of tokens w = [ w 1 , ..., w N ] , a corrupted version  w is constructed by randomly setting a portion of tokens in w to a special [MASK] symbol. Although MLM estimates the token probabilities of all masked positions,  w , simultaneously and renders the factorization from Subsection 3.1 no longer applicable, the mechanism used to un-mask" a token differs only slightly from that in AR, specifically: max  log p  (  w |  w )  N (cid:88) t =1 m t log p  ( w t |  w ) (2) = N (cid:88) t =1 m t log exp (cid:16)(cid:10) h  (  w ) (cid:62) t , e (  w t (cid:11)(cid:17) (cid:80) V w (cid:48) exp (cid:16)(cid:10) h  (  w ) (cid:62) t , e ( w (cid:48) ) (cid:11)(cid:17) = N (cid:88) t =1 m t log softmax (cid:0) h  (  w ) W (cid:62) (cid:1) label t , where m t = 1 indicates w t is masked, and h  (  w ) t is the output representations computed as function of the full, noisy, input sequence. Note, that the main difference between the equations 1 and 2 is the context used to condition the estimation. Models trained with MLM objective, like BERT and RoBERTa, compute the output vector utilizing bidirectional context through the self-attention mechanism, while the unidirectional models use only the context to the left of the target token. Moreover, only the probabilities of masked words, w i such that w i   w, are estimated. Although the two objectives described above differ in terms of the distribution modeled (Yang et al., 2019), both AR and MLM models rely on the softmax function and cross-entropy loss. Using the notation established above, the cross-entropy loss function for an AR model is optimized by minimizing: J (  ) =  E w  data [log p  ( w )] , (3) and for a MLM model it takes a form of: J (  ) =  E w  data [log p  (  w |  w )] . The gradient of the cross-entropy loss with respect to the embedding matrix W is a sum of the gradient flowing through two paths: first one is through the output layer where the embeddings are used to create the targets for the softmax, the second path flows through the encoder stack to the input layer. The gradient flowing through the embedding stack to the input layer is complex, and depends on minute details of a model. Although its contribution is not irrelevant, it is not necessary to illustrate the main point of this section. Thus, we focus on the update rule resulting from the gradient with respect to embeddings in the top layer of a model. For prediction of a token w t , let h  be the output vector of either AR model (at index t  1 ) or MLM model (at index t ), let y = softmax ( f t ) , where f t = h  W (cid:62) , and let  y be the true probability distribution, then: J t  W = h  ( x ) (cid:62) t  ( y   y ) . where  be the learning rate. Since  y is equal to 0 for all the indices except for the index of the target word w t , all the embeddings will become less similar to the representation produced by a model with the exception of the target word embedding. This leads to what we define as the common enemies effect  target words producing gradients of the same direction for all of the non-target words. As the parameters  are updated during the optimization process, the h  changes even when the model is provided with the same input. Therefore, the direction of the gradient for the non-target words changes accordingly, but at a particular step the direction of the update is the same for all the nontarget words. This is fundamentally different from the conclusion of Gao et al. (2019), who states that there exists a uniformly negative direction such that its minimization yields a nearly optimal solution for rare words' embeddings. We find that the common enemies effect is the most pronounced in the representations of rare words, which are less likely to appear as targets, but it is evident in all embeddings nonetheless. Previous studies (Gao et al., 2019; Wang et al., 2020) suggest that word embeddings learned by representations. increasing standard embedding  MEN Test Collection (Bruni et al., measures the relatedness of words.  WordSim353 (Agirre et al., 2009) consists of two parts, one measures similarity, other measures relatedness of words. become more similar, the resulting representations become closer, creating a positive feedback mechanism for the representations to drift collectively. In addition, while isotropy of representations is desirable and has an overall positive impact on performance, the relationships between isotropy and performance in Table 1 and Table 2 suggest that the role of isotropy in model performance needs to be further analyzed. The dynamics of the interactions are being further investigated to pinpoint the root cause and their relationship with the model's performance. Gao et al. (2019) present an insightful derivation of uniformly negative gradients for nonapparent words and formulate the optimization of rare words as an  -strongly convex problem but make strong assumptions that the embedding matrix is learned after all other parameters of the model are well-optimized and fixed, which is not the case in practice. We do not make such assumptions, providing a more realistic explanation for the learning process. Wang et al. (2020) propose to reparametrize the embedding matrix using SVD and propose directly controlling the decay rate of singular values. Our paper's purpose is inherently different from that of Wang et al. (2020); we recognize that the fundamental understanding of the problem is missing and provide an explanation for the observations made in previous studies. Another line of work focuses on limitations of the softmax. Yang et al. (2018) suggest that softmax does not have sufficient capacity to model the complexity of language. Zhang et al. (2019) analyze the skip-gram model to show that optimization based on cross-entropy loss and softmax resembles competitive learning in which words compete among each other for the context vector. This idea is closely related to the common enemies effect reported in this paper, however, skip-gram seems to mitigate this through negative sampling (Mikolov et al., 2013b) but similar approaches do not seem to help Transformer pretraining (Clark et al., 2020). A considerable effort has been made to improve performance of language systems on rare words, but the focus has been on either injecting subword information in non-contextual representations (Lu-ong et al., 2013; Lazaridou et al., 2017; Pinter et al., 2017; Bojanowski et al., 2017), replacing rare words' representations through exploiting their context (Khodak et al., 2018; Liu et al., 2019a), or both (Schick and Schtze, 2019, 2020a). In comparison, we strive to provide an explanation of the underlying problem, which is necessary to render such post-hoc fixes no longer necessary. We find that the embeddings learned by GPT-2, BERT, and RoBERTa do not degenerate into a narrow cone, as has been suggested in the past, but instead drift in one shared direction. We recognize that target words produce gradients in the same direction for all the non-target words at each training step. Combined with the unbalanced distribution of word frequencies, any two words' embeddings will be repeatedly updated with gradients of the same direction. As such updates accumulate, the embeddings drift and share common components. Our experiments show that simply centering the embeddings restores a nearly perfectly isotropic distribution of tested models' embeddings and simultaneously improves embeddings' ability to reflect semantic relations. This understanding of the learning process dynamics opens exciting avenues for future work, such as improving the most affected embeddings of rare words and formulation of more computationally efficient training objectives.
abstain result abstain abstain result other abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method abstain result abstain method result abstain result abstain result other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other method other abstain other other other method other method result method abstain abstain result abstain