How do masked language models (MLMs) such as BERT learn contextual representations? In this work, we analyze the learning dynamics of MLMs. We find that MLMs adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations, which limits the efficiency and effectiveness of MLMs. To address these issues, we propose TACO, a simple yet effective representation learning approach to directly model global semantics. TACO extracts and aligns contextual semantics hidden in contextualized representations to encourage models to attend global semantics when generating contextualized representations. Experiments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over existing MLMs. The code is available at https:// github.com/FUZHIYI/TACO . In the age of deep learning, the basis of representation learning is to learn distributional semantics. The target of distributional semantics can be summed up in the so-called distributional hypothesis (Harris, 1954): Linguistic items with similar distributions have similar meanings . To model similar meanings, traditional representation approaches (Mikolov et al., 2013; Pennington et al., 2014) (e.g., Word2Vec) model distributional semantics by defining tokens using context-independent (CI) dense vectors, i.e., word embeddings, and directly aligning the representations of tokens in the same context. Nowadays, pre-trained language models (PTMs) (Devlin et al., 2019; Radford et al., 2018; Qiu et al., 2020) expand static embeddings into contextualized representations where each token has two kinds of representations: context-independent embedding, and context-dependent  Equal Contribution  This work is done at ByteDance AI Lab. (CD) dense representation that stems from its embedding and contains context information. Although language modeling and representation learning have distinct targets, masked language modeling is still the prime choice to learn token representations with access to large scale of raw texts (Pe-ters et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020). It naturally raises a question: How do masked language models learn contextual representations? Following the widely-accepted understanding (Wang and Isola, 2020), MLM optimizes two properties, the alignment of contextualized representations with the static embeddings of masked tokens, and the uniformity of static embeddings in the representation space. In the alignment property, sampled embeddings of masked tokens play as an anchor to align contextualized representations. We find that although such local anchor is essential to model local dependencies, the lack of global anchors brings several limitations. First, experiments show that the learning of contextual representations is sensitive to embedding quality, which harms the efficiency of MLM at the early stage of 2701 training. Second, MLM typically masks multiple target words in a sentence, resulting in multiple embedding anchors in the same context. This pushes contextualized representations into different clusters and thus harms modeling global dependencies. To address these challenges, we propose a novel T okenA lignment C ontrastive O bjective ( TACO ) to directly build global anchors. By combing local anchors and global anchors together, TACO achieves better performance and faster convergence than MLM. Motivated by the widely-accepted belief that contextualized representation of a token should be the mapping of its static embedding on the contextual space given global information, we propose to directly align global information hidden in contextualized representations at all positions of a natural sentence to encourage models to attend same global semantics when generating contextualized representations. Concerning possible relationships between context-dependent and context-independent representations, we adopt the simplest probing method to extract global information via the gap between context-dependent and context-independent representations of a token for simplification, as shown in Figure 1. To be specific, we define tokens in the same context (text span) as positive pairs and tokens in different contexts as negative pairs, to encourage the global information among tokens within the same context to be more similar compared to that from different contexts. We evaluate TACO on GLUE benchmark. Experiment results show that TACO outperforms MLM with average 1.2 point improvement and 5x speedup (in terms of sample efficiency) on BERT-small, and with average 0.9 point improvement and 2x speedup on BERT-base. The contributions of this paper are as follows. We analyze the limitation of MLM and propose a simple yet efficient method TACO to directly model global semantics. Experiments show that TACO outperforms MLM with up to 1.2 point improvement and up to 5x speedup on GLUE benchmark. The key idea of MLM is to randomly replace a few tokens in a sentence with the special token [MASK] and ask a neural network to recover the original tokens. Formally, we define a corrupted sentence as x 1 , x 2 ,    , x L , and feed it into a Transformers encoder (Vaswani et al., 2017), the hidden states from the final layer are denoted as h 1 , h 2 ,    , h L . We denote the embeddings of the corresponding original tokens as e 1 , e 2 ,    , e L . The MLM objective can be formulated as: LMLM ( x ) =  1 |M| (cid:88) i M log exp( m i  e i ) (cid:80) |V| k =1 exp( m i  e k ) (1) where M denotes the set of masked tokens and |V| is the size of vocabulary V . m i is hidden state of the last layer at the masked position, and can be regarded as a fusion of contextualized representations of surrounding tokens. Following the widely-accepted understanding (Wang and Isola, 2020), Eq.1 optimizes: (1) the alignment between contextualized representations of surrounding tokens and the context-independent embedding of the target token and (2) the uniformity of representations in the representation space. In the alignment part, MLM relies on sampled contextual-independent embeddings of masked tokens as anchors to align contextualized representations in contexts, as shown in Figure 2. Local anchor is the key feature of MLM. Therefore, the learning of contextualized representations heavily relies on embedding quality. In addition, multiple local anchors in a sentence tend to pushing contextualized representations of surrounding tokens closer to different clusters, encouraging models to attend local dependencies where global semantics are neglected. To verify our understanding, we conduct comprehensive experiments to investigate: How does embedding anchor affect the learning dynamics of MLM? We re-train a BERT-small (Devlin et al., 2019) model with the MLM objective solely and analyze the changes in its semantic space during 2702 pre-training. Contextualized representation evaluation. In general, if contextualized representations are well learned, the contextualized representations in a same context will have higher similarity than that of in different contexts. Naturally, we use the gap between intra-sentence similarity and inter-sentence similarity to evaluate contextual information in contextualized representations. We call this gap as contextual score . The similarity can be evaluated via probing methods like L2 distance, cosine similarity, etc. We observe similar findings on different probing methods and only report cosine similarity here for simplification. Figure 3(b) shows how contextual score changes during training. Other statistical results are listed in Appendix A. Embedding similarity evaluation. To observe how sampled embeddings affect contextualized representation learning, we evaluate the embedding similarity between co-occurrent tokens. Motivated by the target that co-occurrent tokens should have similar representations, we use the similarity score calculated by cosine similarity between co-occurrent words labeled by humans (sampled from the WordSim353 dataset (Agirre et al., 2009)) as the evaluation metric. Figure 3(a) shows how embedding similarity between co-occurrent tokens changes during training. The learning of contextualized representations heavily relies on embeddings similarity. As we can see from Figure 3(a), the embedding similarity between co-occurrent tokens first decreases during the earliest stage of pre-training. It is because all embeddings are randomly initialized with the same distribution and the uniformity feature in MLM pushes tokens far away from each other, thus resulting in the decrease of embedding similarity. Meanwhile, the contextual score, i.e., the gap between intra-context similarity and inter-context similarity in Figure 3(b), does not increase at the earliest stage of training. It shows that random embeddings provide little help to learn contextual semantics. During 5K-10K iterations, only when embeddings become closer, contextualized representations in the same context begin to have similar features. At this stage, the randomly sampled embeddings from the same sentence, i.e., the same context, usually have similar representations and thus MLM can push contextualized tokens closer to each other. We further verify the effects of embedding quality in Figure 4. To this end, we train two BERT models whose embedding matrices are frozen and initialized with the ones from different pre-training stage. We can see the model initialized with random embedding fails to teach contextualized representations to attend sentence meanings and representations from different contexts have almost the same similarity. However, the variant with well-trained but frozen embeddings learns to distinguish different contexts early at around 4k steps. These statistical observations verify that embedding anchors bring the efficiency and effectiveness problem. Surprisingly, embedding anchors reduce global contextual information in contextualized representation at the later stage of training. Figure 3(a) shows that embedding similarity begins to drop after 8k steps. It shows that the model learns the specific meanings of co-occurrent tokens and begins to push them a little bit far away. Since MLM adopts local anchors, these local em-2703 beddings push contextualized representations into different clusters. The contextual score begins to decrease too. This phenomenon proves the embedding bias problem where the learning of contextualized representations is decided by the selected embeddings where the global contextual semantics are neglected. To address the challenges of MLM, we propose a new method TACO to combine global anchors and local anchors. We first introduce TC, a token-alignment contrastive loss which explicitly models global semantics in Section 3.1, and combine TC with MLM to get the overall objective for training our TACO model in Section 3.2. To model global semantics, the objective is expected to be capable of explicitly capturing information shared between contextualized representation of tokens within the same context. Therefore, a natural solution is to maximize the mutual information of contextual information hidden in contextualized representations in the same context. To extract shared contextual information, we first define a rule to generate contextual representations of tokens by combining embeddings and global information. Formally, h i = f ( e i , g ) . where f is a probing algorithm and e i is the embedding and g is the global bias of a concrete context. In this paper, we adopt a straightforward probing method to get global information hidden in contextualized representations, where g i = h i  e i . Given contextualized representations of an token x and its nearby tokens c in the same context, we use g x and g c to represent global semantics hidden in these representations. The mutual information between the two global bias g x and g c is I ( g x , g c ) = (cid:88) g x , g c p ( g x , g c ) log p ( g x | g c ) p ( g x ) (4) According to van den Oord et al. 2019, the InfoNCE loss serves as an estimator of mutual information of x and c : I ( g x , g c )  log( K )  L ( g x , g c ) (5) where L ( g x , g c ) is defined as: L ( g x , g c ) =  E   log f ( g x , g c ) f ( g x , g c ) + (cid:80) Kk =1 f ( g x , g c  k )   (6) where c  k is the k -th negative sample of x and K is the size of negative samples. Hence minimizing the objective L ( g x , g c ) is equivalent to maximizing the lower bound on the mutual information I ( g x , g c ) . This objective contains two parts: positive pairs f ( g x , g c ) and negative pairs f ( g x , g c  k ) . Previous study (Chen et al., 2020) has shown that cosine similarity with temperature performs well as the score function f in InfoNCE loss. Following them, we take f ( g x , g c ) = 1  g x  g c (cid:107) g x (cid:107)(cid:107) g c (cid:107) (7) where  is the temperature hyper-parameter and (cid:107)  (cid:107) is (cid:96) 2 -norm function. Contextualized representation : To get global bias g x and g c following Eq. 3, we adopt the widely-used Transformer (Vaswani et al., 2017) as the encoder and take the last hidden states as 2704 the contextualized representations h x and h c . Formally, suppose a batch of sequences { s i } where i  { 1 ,    , N } . We feed it into the Transformer encoder to obtain contextualized representations, h i 1 , h i 2 ,    , h i | s i | where h ij  R d . Positive pairs : Given each token x , we randomly sample a positive sample c from nearby tokens in the same context (sequence) within a window span where W is the window size. Negative pairs : Given each token x , we randomly sample K tokens from other sequences in this batch as negative samples c  k . To sum up, the T oken-alignment C ontrastive ( TC ) loss is applied to every token in a batch as: LTC = 1 NN (cid:88) i =1 1 | s i | | s i | (cid:88) j =1 L ( g ij , g ij c ) (8) where N is the number of sequences of this batch; s i is the i -th sequence; j and jc are tokens in s i where jc (cid:54) = j ; g i is the global semantics hidden in contextualized representation of token s i . g ij and g ij c are generated via: g ij = h ij  e ij (9) g ij c = h ij c  e ij c (10) where h ij and e ij are the contextualized representation and static embedding of the anchor token, respectively. h ij c and e ij c are the contextualized representation and static embedding of the sampled positive token in the same context. As described before, the token-alignment contrastive loss LTC is designed to model global dependencies while MLM is able to capture local dependencies. Therefore, we can better model contextualized representations by combining the token-alignment contrastive loss LTC and the MLM loss to get our overall objective LTACO : LTACO = LTC + LMLM (11) We implement it in a multi-task learning manner where all objectives are calculated within one for-ward propagation, which only introduces negligible extra computations. WordPiece tokenization) (Zhu et al., 2015) and English Wikipedia (4B words) as pre-training corpus. We pre-train two variants of BERT models: BERT-small and BERT-base. All models are equipped with the vocabulary of size 30,522, trained with 15% masked positions for MLM. The maximum sequence length is 256 and batch size is 1,280. We adopt optimizer AdamW (Loshchilov and Hut-ter, 2019) with learning rate 1e-4. All models are trained until convergence. To be specific, the small model is trained up to 250k steps with a warm-up of 2.5k steps. The base model is trained up to 500k steps with a warm-up of 10k steps. For TACO, we set the positive sample window size W to 5, the negative sample number K to 50, and the temperature parameter  to 0.07 after a slight grid-search via preliminary experiments. More pre-training details can be found in Appendix A. During fine-tuning models, we conduct a grid search over batch sizes of {16, 32, 64, 128}, learning rates of {1e-5, 2e-5, 3e-5, 5e-5}, and training epochs of {4, 6} with an Adam optimizer (Kingma and Ba, 2015). We use the open-source packages for implementation, including HuggingFace Datasets 1 and Transformers 2 . All the experiments are conducted on 16 GPU chips (32 GB V100). Evaluation We evaluate methods on the GLUE benchmark (Wang et al., 2019). Specifically, we test on Microsoft Research Paraphrase Matching (MRPC) (Dolan and Brockett, 2005), Quora Question Pairs (QQP) 3 and STS-B (Conneau and Kiela, 2018) for Paraphrase Similarity Matching; Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) for Sentiment Classification; Multi-Genre Natural Language Inference Matched (MNLI-m), Multi-Genre Natural Language Inference Mismatched (MNLI-mm) (Williams et al., 2018), Question Natural Language Inference (QNLI) (Ra-jpurkar et al., 2016) and Recognizing Textual Entailment (RTE) (Wang et al., 2019) for the Natural Language Inference (NLI) task; The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) for Linguistic Acceptability. Following Devlin et al. (2019), we exclude WNLI (Levesque, 2011). We report F1 scores for QQP and MRPC, Spearman correlations for STS-B, and accuracy scores for the other tasks. For evaluation results on validation sets, we report the 1 https://github.com/huggingface/datasets 2 https://github.com/huggingface/transformers 3 https://www.quora.com/q/quoradata/ First-Quora-Dataset-Release-Question-Pairs 2705 Approach MNLI(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE Avg. average score of 4 fine-tunings with different random seeds. For results on test sets, we select the best model on the validation set to evaluate. Baselines We mainly compare TACO with MLM on BERT-small and BERT-base models. In addition, we also compare TACO with related contrastive methods: a sentence-level contrastive method BERT-NCE and a span-based contrastive learning method INFOWORD, both from Kong et al. (2020). We directly compare TACO with the results reported in their paper. Table 1 and Figure 5 show the results of TACO on BERT-small. As we can see, compared with MLM with 250k training steps ( convergence steps), TACO achieves comparable performance with only 1/5 computation budget. By modeling global dependencies, TACO can significantly improve the efficiency of contextualized representation learning. In addition, when pre-trained with the same steps, TACO outperforms MLM with 1.2 average score improvement on the validation set. In addition to convergence, we also compare TACO and MLM on fewer training data. The results are shown in Table 2. We sample 4 tasks with the largest amount of training data for evaluation. As we can see, TACO trained on 25% data can achieve competitive results with MLM trained on full data. These results also verify the data efficiency of our method, TACO. We also compare TACO with MLM on base-sized models, which are the most commonly used models according to the download data from Huggingface 4 (Wolf et al., 2020). First, from Table 3, we can see that TACO consistently outperforms 4 https://huggingface.co/models Approach MNLI QQP QNLI SST-2 Avg. MLM under all pre-training computation budgets. Notably, TACO-250 k achieves comparable performance with MLM-500 k , which saves 2x computations. Similar results are observed on TACO-100 k and BERT-250 k . These results demonstrate that TACO can achieve better acceleration over MLM. It is also a significant improvement compared to previous methods (Gong et al., 2019) focusing on accelerating BERT but only with slight speedups. In addition, as shown in Table 4, TACO achieves competitive results compared to BERT-NCE and INFOWORD, two similar contrastive methods. To better understand how TACO works, we conduct a quantitative comparison on the learning dynamic for BERT and TACO. Similar to Section 2.2, we plot the Cosine similarity among contextualized representations of tokens in the same context (intra-context) and different contexts (inter-context) in Figure 6. We find that the learning dynamic of TACO significantly differs from that of MLM. Specifically, for TACO, the intra-context representation similarity remains high and the gap between intra-context similarity and inter-context similarity remains large at the later stage of training. This confirms that TACO can better fulfill global semantics, which may contribute to the superior downstream performance. TACO is implemented as a token-level contrastive (TC) loss along with the MLM loss. Therefore, the improvement of TACO might come from two aspects, including 1) denser supervision signals from the all-token objective and 2) the benefits of the contrastive loss to strengthen global dependencies. It is helpful to figure out which factor is more important. To this end, we design two variants for ablation. One is a concentrated TACO, where the contrastive loss is built on the 15% masked positions only, keeping the same density of supervision signal with MLM. The other is an extended MLM, where not only 15% masked positions are asked to predict the original token, so do the rest 85% unmasked positions. The extended MLM has the same dense supervision with TACO but loses the benefits of modeling the global dependencies. The results on small models are shown in Figure 6. As we can see, the performance of TACO decreases if we sample a part of token positions to implement TC objectives. It shows that more supervision signals benefit the final performance of TACO. However, simply adding more supervision signals by predicting unmasked tokens does not help MLM too much. Even equipped with the extra 85% token prediction (TP) loss, MLM+TP does not show significant improvements and it is noticeable that the performance of MLM+TP starts to drop after 150k steps. This further confirms the effectiveness of TC loss by strengthening global dependencies. Classic language representation learning methods (Mikolov et al., 2013; Pennington et al., 2014) aims to learn context-independent representation of words, i.e., word embeddings. They generally follow the distributional hypothesis (Harris, 1954). Recently, the pre-training then fine-tuning paradigm has become a common practice in NLP because of the success of pre-trained language models like BERT (Devlin et al., 2019). Context-dependent (or contextualized) representations are the basic characteristic of these methods. Many 2707 0 5k 10k 15k 20k pretrain checkpoints 0.0 0.2 0.4 0.6 0.8 1.0 MLM intra-context similarity MLM inter-context similarity TACO intra-context similarity TACO inter-context similarity 0 50k 100k 150k 200k 250k pretrain checkpoints 0.68 0.70 0.72 0.74 0.76 a v e r a g e GLUE s c o r e 15% MLM + 100% TC (TACO) 15% MLM + 15% TC 15% MLM + 85% TP 15% MLM Figure 6: The left figure existing contextualized models are based on the masked language modeling objective, which randomly masks a portion of tokens in a text sequence and trains the model to recover the masked tokens. Many previous studies prove that pre-training with the MLM objective helps the models learn syntactical and semantic knowledge (Clark et al., 2019). There have been numerous extensions to MLM. For example, XLNet (Yang et al., 2019) introduced the permutated language modeling objective, which predicts the words one by one in a permutated order. BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) investigated several denoising objectives and pre-trained an encoder-decoder architecture with the mask span infilling objective. In this work, we focus on the key MLM objective and aim to explore how MLM objective helps learn contextualized representation. Apart from denoising-based objectives, contrastive learning is another promising way to obtain self-supervision. In contrastive-based self-supervised learning, the models are asked to distinguish the positive samples from the negative ones for a given anchor. Contrastive-based SSL method was first introduced in NLP for efficient learning of word representations by negative sampling, i.e., SGNS (Word2Vec (Mikolov et al., 2013)). Later, similar ideas were brought into CV field for learning image representation and got prevalent, such as MoCo (He et al., 2020), SimCLR (Chen et al., 2020), BYOL (Caron et al., 2020), etc. In the recent two years, there have been many studies targeting at reviving contrastive learning for contextual representation learning in NLP. Tak-2708 ing MLM as an example, we investigate whether and how current language model pre-training objectives learn contextualized representation. For instance, CERT (Fang et al., 2020) utilized back-translation to generate positive pairs. CAPT (Luo et al., 2020) applied masks to the original sentence and considered the masked sentence and its original version as the positive pair. DeCLUTR (Giorgi et al., 2020) samples nearby even overlapping spans as positive pairs. INFOWORD (Kong et al., 2020) treated two complementary parts of a sentence as the positive pair. However, the aforementioned methods mainly focus on sentence-level or span-level contrast and may not provide dense self-supervision to improve efficiency. Unlike these approaches, TACO regards the global semantics hidden in contextualized token representations as the positive pair. The token-level contrastive loss can be built on all input tokens, which provides a dense self-supervised signal. Another related work is ELECTRA (Clark et al., 2020). ELECTRA samples machine-generated tokens from a separate generator and trains the main model to discriminate between machine-generated tokens and original tokens. ELECTRA implicitly treats the fake tokens as negative samples of the context, and the unchanged tokens as positive samples. Unlike this method, TACO does not require architectural modifications and can serve as a plug-and-play auxiliary objective, largely improving pretraining efficiency. In this paper, we propose a simple yet effective objective to learn contextualized representation. We find that the MLM objective mainly focuses on local anchors to align contextualized representations, which harms global dependencies modeling due to an embedding bias problem. Motivated by these problems, we propose TACO to directly model global semantics. It can be easily combined with existing LM objectives. By combining local and global anchors, TACO achieves up to 5  speedups and up to 1.2 improvements on GLUE score. This demonstrates the potential of TACO to serve as a plug-and-play approach to improve contextualized representation learning. We thank the anonymous reviewers for their helpful feedback. We also thank the colleagues from ByteDance AI Lab for their suggestions on our experiment designing and paper writing.
abstain method result objective abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain result abstain abstain abstain objective abstain objective result method abstain objective objective objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method method other abstain abstain abstain method abstain method abstain method method method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain other abstain result abstain abstain result abstain result abstain abstain abstain result result result abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other objective other other other other other objective other other other other other other other other other other other objective objective objective abstain abstain abstain other other