Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n -gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, n -grams are masked and predicted directly using explicit n -gram identities rather than contiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n -gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive n -gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github. com/PaddlePaddle/ERNIE . Pre-trained on large-scaled text corpora and fine-tuned on downstream tasks, self-supervised representation models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020; Clark et al., 2020) have achieved remarkable improvements in natural language understanding (NLU). As one of the most prominent pre-trained models, BERT (Devlin et al., 2019) employs masked language modeling (MLM) to learn representations by masking individual tokens and predicting them based on their bidirectional context. However, BERT's MLM focuses on the representations of fine-grained text units (e.g. words or subwords in English and characters in Chinese), rarely considering the coarse-grained linguistic information (e.g. named entities or phrases in English and words in Chinese) thus incurring inadequate representation learning. Many efforts have been devoted to integrate coarse-grained semantic information by independently masking and predicting contiguous sequences of n tokens, namely n -grams, such as named entities, phrases (Sun et al., 2019b), whole words (Cui et al., 2019) and text spans (Joshi et al., 2020). We argue that such contiguously masking strategies are less effective and reliable since the prediction of tokens in masked n-grams are independent of each other, which neglects the intra-dependencies of n-grams. Specifically, given a masked n -gram w = { x 1 , ..., x n } , x V F , we maximize p ( w ) = (cid:81) ni =1 p ( x i | c ) for n -gram learning, where models learn to recover w in a huge and sparse prediction space F  R |V F | n . Note that VF is the fine-grained vocabulary 1 and c is the context. We propose ERNIE-Gram, an explicitly n gram masked language modeling method in which n -grams are masked with single [MASK] symbols, and predicted directly using explicit n gram identities rather than sequences of tokens, as depicted in Figure 1(b). The models learn to predict n -gram w in a small and dense prediction space N  R |V N | , where VN indicates a prior n gram lexicon 2 and normally |V N | (cid:28) |V F | n . To 1 VF contains 30 K BPE codes in BERT (Devlin et al., 2019) and 50 K subword units in RoBERTa (Liu et al., 2019). 2 VN contains 300 K n -grams, where n  [2 , 4) in this paper, n -grams are extracted in word-level before tokenization. i i and explicit n -grams respectively. Note that the weights of fine-grained classifier ( WF  R h |V F | ) and N-gram classifier ( WN  R h |(cid:104)V F , VN (cid:105)| ) are not used in fine-tuning stage, where h is the hidden size and L is the layers. learn the semantic of n -grams more adequately, we adopt a comprehensive n -gram prediction mechanism, simultaneously predicting masked n grams in coarse-grained (explicit n -gram identities) and fine-grained (contained token identities) manners with well-designed attention mask metrics, as shown in Figure 1(c). In addition, to model the semantic relationships between n -grams directly, we introduce an enhanced n -gram relation modeling mechanism, masking n -grams with plausible n -grams identities sampled from a generator model, and then recovering them to the original n -grams with the pair relation between plausible and original n -grams. Inspired by ELECTRA (Clark et al., 2020), we incorporate the replaced token detection objective to distinguish original n -grams from plausible ones, which enhances the interactions between explicit n -grams and fine-grained contextual tokens. In this paper, we pre-train ERNIE-Gram on both base-scale and large-scale text corpora (16GB and 160GB respectively) under comparable pre-training setting. Then we fine-tune ERNIE-Gram on 13 English NLU tasks and 6 Chinese NLU tasks. Experimental results show that ERNIE-Gram consistently outperforms previous well-performed pre-training models on various benchmarks by a large margin. Self-supervised pre-training has been used to learn contextualized sentence representations though various training objectives. GPT (Radford et al., 2018) employs unidirectional language modeling (LM) to exploit large-scale corpora. BERT (Devlin et al., 2019) proposes masked language modeling (MLM) to learn bidirectional representations efficiently, which is a representative objective for pre-training and has numerous extensions such as RoBERTa (Liu et al., 2019), UNILM (Dong et al., 2019) and ALBERT (Lan et al., 2020). XLNet (Yang et al., 2019) adopts permutation language modeling (PLM) to model the dependencies among predicted tokens. ELECTRA introduces replaced token detection (RTD) objective to learn all tokens for more compute-efficient pre-training. Coarse-grained linguistic information is indispensable for adequate representation learning. There are lots of studies that implicitly integrate coarse-grained information by extending BERT's MLM to contiguously masking and predicting contiguous sequences of tokens. For example, ERNIE (Sun et al., 2019b) masks named entities and phrases to enhance contextual representations, BERT-wwm (Cui et al., 2019) masks whole Chinese words to achieve better Chinese representations, SpanBERT (Joshi et al., 2020) masks contiguous spans to improve the performance on span selection tasks. A few studies attempt to inject the coarse-grained n -gram representations into fine-grained contextualized representations explicitly, such as ZEN (Diao et al., 2020) and AMBERT (Zhang and Li, 2020), in which additional transformer encoders and computations for explicit n -gram representations are incorporated into both pre-training and fine-tuning. Li et al., 2019 demonstrate that explicit n -gram representations are not sufficiently reliable for NLP tasks because of n -gram data sparsity and the ubiquity of out-of-vocabulary n -grams. Differently, we only incorporate n -gram information by leveraging auxiliary n -gram classifier and embedding weights in pre-training, which will be completely removed during fine-tuning, so our method maintains the same parameters and computations as BERT. In this section, we present the detailed implementation of ERNIE-Gram, including n -gram lexicon VN extraction in Section 3.5, explicitly n -gram MLM pre-training objective in Section 3.2, comprehensive n -gram prediction and relation modeling mechanisms in Section 3.3 and 3.4. To inject n -gram information into pre-training, many works (Sun et al., 2019b; Cui et al., 2019; Joshi et al., 2020) extend BERT's masked language modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. Contiguously MLM. Given input sequence x = { x 1 , ..., x | x | } , x  VF and n -gram starting boundaries b = { b 1 , ..., b | b | } , let z = { z 1 , ..., z | b | 1 } to be the sequence of n -grams, where z i = x [ b i : b i +1 ) , MLM samples 15% of starting boundaries from b to mask n -grams, donating M as the indexes of sampled starting boundaries, z M as the contiguously masked tokens, z \M as the sequence after masking. As shown in Figure 1(a), b = { 1 , 2 , 4 , 5 , 6 , 7 } , z = { x 1 , x [2:4) , x 4 , x 5 , x 6 } , M = { 2 , 4 } , z M = { x [2:4) , x 5 } , and z \M = { x 1 , [M] , [M] , x 4 , [M] , x 6 } . Contiguously MLM is performed by minimizing the negative likelihood:  log p  ( z M | z \M ) =  (cid:88) z  z M (cid:88) x  z log p  ( x | z \M ) . Different from contiguously MLM, we employ explicit n -gram identities as pre-training targets to reduce the prediction space for n -grams. To be specific, let y = { y 1 , ..., y | b | 1 } , y  (cid:104)V F , VN (cid:105) to be the sequence of explicit n -gram identities, y M to be the target n -gram identities, and  z \M to be the sequence after explicitly masking n -grams. As shown in Figure 1(b), y M = { y 2 , y 4 } , and  z \M = { x 1 , [M] , x 4 , [M] , x 6 } . For masked n gram x [2:4) , the prediction space is significantly reduced from R |V F | 2 to R |(cid:104)V F , VN (cid:105)| . Explicitly n gram MLM is performed by minimizing the negative likelihood:  log p  ( y M |  z \M ) =  (cid:88) y  y M log p  ( y |  z \M ) . We propose to simultaneously predict n -grams in fine-grained and coarse-grained manners corresponding to single mask symbol [M] , which helps to extract comprehensive n -gram semantics, as shown in Figure 1(c). Comprehensive n -gram MLM is performed by minimizing the joint negative likelihood:  log p  ( y M , z M |  z \M ) =  (cid:88) y  y M log p  ( y |  z \M )  (cid:88) z  z M (cid:88) x  z log p  ( x |  z \M ) . (3) where the predictions of explicit n -gram y M and fine-grained tokens x M are conditioned on the same context sequence  z \M . In detail, to predict all tokens contained in a n gram from single [M] other than a consecutive sequence of [M] , we adopt distinctive mask symbols [Mi] , i =1 , ..., n to aggregate contextualized representations for predicting the i -th token in n gram. As shown in Figure 2(a), along with the same position as y 2 , symbols [M1] and [M2] are used as queries ( Q ) to aggregate representations from  z \M ( K ) for the predictions of x 2 and x 3 , where Q and K donate the query and key in self-attention operation (Vaswani et al., 2017). As shown in Figure 2(b), the self-attention mask metric M controls what context a token can attend to by modifying the attention weight WA = softmax ( QKT  d k + M ) , M is assigned as: M ij = (cid:40) 0 , allow to attend  , prevent from attending (4) We argue that the length information of n -grams is detrimental to the representations learning, because it will arbitrarily prune a number of semantiL' PredictionDistribution Sampler Sampler the proposed a overhaul of the prime minister public official president 0.05 0.01 0.16 accountant ... not second to nothing short of nothing less than 0.09 0.08 0.26 ... the public official proposed completely a overhaul of the tax system . cally related n -grams with different lengths during predicting. From this viewpoint, for the predictions of n -gram { x 2 , x 3 } , 1) we prevent context  z \M from attending to { [M1] , [M2] } and 2) prevent { [M1] , [M2] } from attending to each other, so that the length information of n -grams will not be leaked in pre-training, as displayed in Figure 2(b). To explicitly learn the semantic relationships between n -grams, we jointly pre-train a small generator model  (cid:48) with explicitly n -gram MLM objective to sample plausible n -gram identities. Then we employ the generated identities to preform masking and train the standard model  to predict the original n -grams from fake ones in coarse-grained and fine-grained manners, as shown in Figure 3(a), which is efficient to model the pair relationships between similar n -grams. The generator model  (cid:48) will not be used during fine-tuning, where the hidden size H  (cid:48) of  (cid:48) has H  (cid:48) = H  / 3 empirically. As shown in Figure 3(b), n -grams of different length can be sampled to mask original n -grams according to the prediction distributions of  (cid:48) , which is more flexible and sufficient for constructing n gram pairs than previous synonym masking methods (Cui et al., 2020) that require synonyms and original words to be of the same length. Note that our method needs a large embedding layer E  R |(cid:104)V F , VN (cid:105)| h to obtain n -gram vectors in pretraining. To keep the number of parameters consistent with that of vanilla BERT, we remove the auxiliary embedding weights of n -grams during fine-tuning ( E  E (cid:48)  R |V F | h ). Specifically, let y (cid:48)M to be the generated n -gram identities,  z (cid:48)M to be the sequence masked by y (cid:48)M , where y (cid:48)M = { y (cid:48) 2 , y (cid:48) 4 } , and  z (cid:48)\M = { x 1 , y (cid:48) 2 , x 4 , y (cid:48) 4 , x 6 } in Figure 3(a). The pre-training objective is to jointly minimize the negative likelihood of  (cid:48) and  :  log p  (cid:48) ( y M |  z \M )  log p  ( y M , z M |  z (cid:48)\M ) . Moreover, we incorporate the replaced token detection objective (RTD) to further distinguish fake n -grams from the mix-grained context  z (cid:48)\M for interactions among explicit n -grams and fine-grained contextual tokens, as shown in the right part of Figure 3(a). Formally, we donate  z \M to be the sequence after replacing masked n -grams with target n -gram identities y M , the RTD objective is performed by minimizing the negative likelihood:  log p  (cid:0) 1 ( z (cid:48)\M =  z \M ) |  z (cid:48)\M (cid:1) =  |  z \M | (cid:88) t =1 log p  (cid:0) 1 ( z (cid:48)\M ,t =  z \M ,t ) |  z (cid:48)\M , t (cid:1) . (6) As the example depicted in Figure N-gram Lexicon Extraction. We employ T-test to extract semantically-complete n -grams statistically from unlabeled text corpora X (Xiao et al., 2020), as described in Algorithm 1. We first calcu-Algorithm 1 N-gram Extraction with T-test Input: Large-scale text corpora X for pre-training Output: Semantic n -gram lexicon VN (cid:46) given initial hypothesis H 0 : a randomly constructed n -gram w = { x 1 , ..., x n } with probability p (cid:48) ( w ) = (cid:81) n i =1 p ( x i ) cannot be a statistically semantic n -gram for l in range (2, n ) do VN l  (cid:104)(cid:105) (cid:46) initialize the lexicon for l -grams for l -gram w in X do s  ( p ( w )  p (cid:48) ( w ))   2 /N l : t -statistic score (cid:46) where statistical probability p ( w ) = Count ( w ) N l , deviation  2 = p ( w )(1  p ( w )) , N l donates the count of l grams in XVN l . append ( { w , s } ) VN l  topk ( VN l , k l ) (cid:46) k l is the number of l -gram VN  (cid:104)V N 2 , ..., VN n (cid:105) (cid:46) merge all lexicons return VN late the t -statistic scores of all n -grams appearing in X since the higher the t -statistic score, the more likely it is a semantically-complete n -gram. Then, we select the l -grams with the top k l t -statistic scores to construct the final n -gram lexicon VN . N-gram Boundary Extraction. To incorporate n -gram information into MLM objective, n -gram boundaries are referred to mask whole n -grams for pre-training. Given an input sequence x = { x 1 , ..., x | x | } , we employ maximum matching algorithm to traverse valid n -gram paths B = { b 1 , ..., b |B| } according to VN , then select the shortest paths as the final n -gram boundaries b , where | b |  | b i | ,  i = 1 , ..., |B| . In this section, we first present the pre-training con-figuration of ERNIE-Gram on Chinese and English text corpora. Then we compare ERNIE-Gram with previous works on various downstream tasks. We also conduct several ablation experiments to access the major components of ERNIE-Gram. Base-scale corpora: 16GB uncompressed text from WIKIPEDIA and BOOKSCORPUS (Zhu et al., 2015), which is the original data for BERT. Large-scale corpora: 160GB uncompressed text from WIKIPEDIA , BOOKSCORPUS , OPENWEBTEXT 3 , CC-NEWS (Liu et al., 2019) and STORIES (Trinh and Le, 2018), which is the original data used in RoBERTa. Chinese Pre-training Data. We adopt the same Chinese text corpora used in ERNIE2.0 (Sun et al., 2020) to pre-train ERNIE-Gram. Before pre-training, we first extract 200 K bi-grams and 100 K tri-grams with Algorithm 1 to construct the semantic n -gram lexicon VN for English and Chinese corpora. and we adopt the sub-word dictionary ( 30 K BPE codes) used in BERT and the character dictionary used in ERNIE2.0 as our fine-grained vocabulary VF in English and Chinese. length of the sequence in each batch up to 512 tokens. We add the relative position bias (Raffel et al., 2020) to attention weights and use Adam (Kingma and Ba, 2015) for optimizing. For pre-training on base-scale English corpora, the batch size is set to 256 sequences, the peak learning rate is 1 e 4 for 1 M training steps, which are the same settings as BERTBASE . As for large-scale English corpora, the batch size is 5112 sequences, the peak learning rate is 4 e 4 for 500 K training steps. For pre-training on Chinese corpora, the batch size is 256 sequences, the peak learning rate is 1 e 4 for 3M training steps. All the pre-training hyper-parameters are supplemented in the Appendix A. In fine-tuning, we remove the auxiliary embedding weights of explicit n -grams identities for fair comparison with previous pre-trained models. The General Language Understanding Evaluation (GLUE; Wang et al., 2018) is a multi-task benchmark consisting of various NLU tasks, which contains 1) pairwise classification tasks like language inference (MNLI; Williams et al., 2018, RTE; Dagan et al., 2006), question answering (QNLI; Ra-jpurkar et al., 2016) and paraphrase detection (QQP, MRPC; Dolan and Brockett, 2005), 2) single-sentence classification tasks like linguistic acceptability (CoLA; Warstadt et al., 2019), sentiment analysis (SST-2; Socher et al., 2013) and 3) text similarity task (STS-B; Cer et al., 2017). The fine-tuning results on GLUE of ERNIE-Gram and various strong baselines are presented in Table 1. For fair comparison, the listed models are all in base size and fine-tuned without any data augmentation. Pre-trained with base-scale text corpora, ERNIE-Gram outperforms recent models such as TUPE and F-TFM by 1 . 7 and 1 . 3 points on average. As for large-scale text corpora, ERNIE-Gram achieves average score increase of 1 . 7 and 0 . 6 over RoBERTa and ELECTRA, demonstrating the effectiveness of ERNIE-Gram. The Stanford Question Answering (SQuAD) tasks are designed to extract the answer span within the given passage conditioned on the question. We conduct experiments on SQuAD1.1 (Rajpurkar et al., 2016) and SQuAD2.0 (Rajpurkar et al., 2018) by adding a classification layer on the sequence outputs of ERNIE-Gram and predicting whether each token is the start or end position of the answer span. Table 2 presents the results on SQuAD for base-size pre-trained models, ERNIE-Gram achieves better performance than current strong baselines on both base-scale and large-scale pre-training text corpora. The ReAding Comprehension from Examinations (RACE; Lai et al., 2017) dataset collects 88 K long passages from English exams at middle and high schools, the task is to select the correct choice from four given options according to the questions and passages. We also evaluate ERNIE-Gram on two large scaled text classification tasks that involve long text and reasoning, including sentiment analysis datasets IMDb (Maas et al., 2011) and topic classification dataset AG's News (Zhang et al., 2015). The results are reported in Table 3. It can be seen that ERNIE-Gram consistently outperforms previous models, showing the advantage of ERNIE-Gram on tasks involving long text and reasoning. We execute extensive experiments on six Chinese language understanding tasks, including natural language inference (XNLI; Conneau et al., 2018), Results on six Chinese tasks are presented in Table 4. It is observed that ERNIE-Gram significantly outperforms previous models across tasks by a large margin and achieves new state-of-the-art results on these Chinese NLU tasks in base-size model group. Besides, ERNIE-Gram BASE are also better than various large-size models on XNLI, LCQMC and CMRC2018 datasets. Effect of Explicitly N-gram MLM. We compare two models pre-trained with contiguously MLM and explicitly n -gram MLM objectives in the same settings (the size of n -gram lexicon is 300 K). The evaluation results for pre-training and fine-tuning are shown in Figure 4. Compared with contiguously MLM, explicitly n -gram MLM objective facilitates the learning of n -gram semantic information with lower n -gram level perplexity in pre-training and better performance on downstream tasks. This verifies the effectiveness of explicitly n -gram MLM objective for injecting n -gram semantic information into pre-training. Size of N-gram Lexicon. To study the impact of n -gram lexicon size on model performance, we extract n -gram lexicons with size from 100 K to 400 K for pre-training, as shown in Figure 5. As the Figure 4: lexicon size enlarges, performance of contiguously MLM becomes worse, presumably because more n -grams are matched and connected as longer consecutive spans for prediction, which is more difficult for representation learning. Explicitly n -gram MLM with lexicon size being 300 K achieves the best results, while the performance significantly declines when the size of lexicon increasing to 400 K because more low-frequent n -grams are learning unnecessarily. See Appendix C for detailed results of different lexicon choices on GLUE and SQuAD. and Enhanced N-gram Relation Modeling. As shown in Table 5, we compare several ERNIE-Gram variants with previous strong baselines under the BERTBASE setting. After removing comprehensive n -gram prediction ( #2 ), ERNIE-Gram degenerates to a variant with explicitly n -gram MLM and n -gram relation modeling and its performance drops slightly by 0 . 3 0 . 6 . When removing enhanced n -gram relation modeling ( #3 ), ERNIE-Gram degenerates to a variant with comprehen-# Models MNLI SST-2 SQuAD1.1 SQuAD2.0 m mm Acc EM F1 EM F1 XLNet a 85.6 85.1 93.4 -78.2 81.0 RoBERTa b 84.7 -92.7 -90.6 -79.7 MPNet c 85.6 -93.6 84.0 90.3 79.5 82.2 UNI LMv2 d 85.6 85.5 93.0 85.0 91.5 78.9 81.8 #1 ERNIE-Gram 86.5 86.4 93.2 85.2 91.7 80.8 84.0 #2  CNP 86.2 86.2 92.7 85.0 91.5 80.4 83.4 #3  ENRM 85.7 85.8 93.5 84.7 91.3 79.7 82.7 #4  CNP  ENRM 85.6 85.7 92.9 84.5 91.2 79.5 82.4 Table 5: Comparisons between comprehensive n -gram prediction ( CNP ) and enhanced n -gram relation modeling ( ENRM ) methods. sive n -gram MLM and the performance drops by 0 . 4 1 . 3 . If removing both comprehensive n -gram prediction and relation modeling ( #4 ), ERNIE-Gram degenerates to a variant with explicitly n gram MLM and the performance drops by 0 . 7 1 . 6 . These results demonstrate the advantage of comprehensive n -gram prediction and n -gram relation modeling methods for efficiently n -gram semantic injecting into pre-training. The detailed results of ablation study are supplemented in Appendix C. 4.8 Case Studies To further understand the effectiveness of our approach for learning n -grams information, we fine-tune ERNIE-Gram, contiguously MLM and lower-cased BERT on CoNLL-2003 named entity recognition task (Sang and De Meulder, 2003) for comparison. We divide the evaluation set into five subsets based on the average length of the named entities in each sentence. As shown in Figure 6(a), it is more difficult to recognize whole named entities as the length of them increases, while the performance of ERNIE-Gram declines slower than contiguously MLM and BERT, which implies that ERNIE-Gram models tighter intra-dependencies of n -grams. As shown in Figure 6(b-d), we visualize the attention patterns in the last self-attention layer of fine-tuned models. For contiguously MLM, there are clear diagonal lines in named entities that tokens prefer to attend to themself in named entities. While for ERNIE-Gram, there are bright blocks over named entities that tokens attend to most of tokens in the same entity adequately to construct tight representation, verifying the effectiveness of ERNIE-Gram for n -gram semantic modeling. In this paper, we present ERNIE-Gram, an explicitly n -gram masking and predicting method to eliminate the limitations of previous contiguously masking strategies and incorporate coarse-grained linguistic information into pre-training sufficiently. ERNIE-Gram conducts comprehensive n -gram prediction and relation modeling to further enhance the learning of semantic n -grams for pre-training. Experimental results on various NLU tasks demonstrate that ERNIE-Gram outperforms XLNet and RoBERTa by a large margin, and achieves state-of-the-art results on various benchmarks. Future work includes constructing more comprehensive n -gram lexicon ( n> 3 ) and pre-training ERNIE-Gram with large-size model for more downstream tasks. We would like to thank Zhen Li for his constructive suggestions, and hope everything goes well with his work. We are also indebted to the NAACL-HLT reviewers for their detailed and insightful comments on our work.
abstain abstain abstain objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain result abstain method abstain abstain abstain abstain other other other other other other other other other other method method abstain other other other other other other method other other other other other method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain method method abstain abstain abstain method abstain method abstain abstain method method other method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain method abstain abstain abstain other other