There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25  parameters of BERT Large , demonstrating its generalizability to different downstream tasks. 1 1 Introduction Language models pretrained on unlabeled texts have substantially advanced the state of the art in various NLP tasks, ranging from natural language understanding (NLU) to text generation (Radford et al., 2018a; Devlin et al., 2019; Yang et al., 2019; Radford et al., 2018b; Raffel et al., 2020; Lewis et al., 2019; Brown et al., 2020). Downstream task performance as well as the scale of the parameters have also constantly increased in the past few years. In general, existing pretraining frameworks can be categorized into three families: autoregressive , autoencoding , and encoder-decoder models. Autoregressive models, such as GPT (Radford et al., 2018a), learn left-to-right language models. While they succeed in long-text generation and show few-shot learning ability when scaled to billions of parameters (Radford et al., 2018b; Brown et al., 2020), the inherent disadvantage is the unidirectional attention mechanism, which cannot fully capture the dependencies between the context words in NLU tasks. Autoencoding models, such as BERT (Devlin et al., 2019), learn bidirectional context encoders via denoising objectives, e.g. Masked Language Model (MLM). The encoders produce contextualized representations that suit natural language understanding tasks, but could not be directly applied for text generation. Encoder-decoder models adopt bidirectional attention for the encoder, unidirectional attention for the decoder, and cross attention between them (Song et al., 2019; Bi et al., 2020; Lewis et al., 2019). They are typically deployed in conditional generation tasks, such as text summarization and response generation. 2 . T5 (Raffel et al., 2020) unifies NLU and conditional generation via encoder-decoder models but requires more parameters to match the performance 2 Unconditional generation refers to generating text as a language model without finetuning, while conditional generation refers to sequence-to-sequence tasks. of BRET-based models such as RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2021). None of these pretraining frameworks is flexible enough to perform competitively across all NLP tasks. Previous works have tried to unify different frameworks by combining their objectives via multi-task learning (Dong et al., 2019; Bao et al., 2020). However, since the autoencoding and autoregressive objectives differ by nature, a simple unification cannot fully inherit the advantages of both frameworks. In this paper, we propose a pretraining framework named GLM (General Language Model), based on autoregressive blank infilling. We randomly blank out continuous spans of tokens from the input text, following the idea of autoencoding, and train the model to sequentially reconstruct the spans, following the idea of autoregressive pretraining (see Figure 1). While blanking filling has been used in T5 (Raffel et al., 2020) for text-to-text pretraining, we propose two improvements, namely span shuffling and 2D positional encoding. Empirically, we show that with the same amount of parameters and computational cost, GLM significantly outperforms BERT on the SuperGLUE benchmark by a large margin of 4.6%  5.0% and outperforms RoBERTa and BART when pretrained on a corpus of similar size (158GB). GLM also significantly outperforms T5 on NLU and generation tasks with fewer parameters and data. Inspired by Pattern-Exploiting Training (PET) (Schick and Schtze, 2020a), we reformulate NLU tasks as manually-crafted cloze questions that mimic human language. Different from the BERT-based models used by PET, GLM can naturally handle multi-token answers to the cloze question via autoregressive blank filling. Furthermore, we show that by varying the number and lengths of missing spans, the autoregressive blank filling objective can pretrain language models for conditional and unconditional generation. Through multi-task learning of different pretraining objectives, a single GLM can excel in both NLU and (conditional and unconditional) text generation. Empirically, compared with standalone baselines, GLM with multi-task pretraining achieves improvements in NLU, conditional text generation, and language modeling tasks altogether by sharing the parameters. We propose a general pretraining framework GLM based on a novel autoregressive blank infilling objective. GLM formulates NLU tasks as cloze questions that contain task descriptions, which can be answered by autoregressive generation. GLM is trained by optimizing an autoregressive blank infilling objective. Given an input text x = [ x 1 ,    , x n ] , multiple text spans { s 1 ,    , s m } are sampled, where each span s i corresponds to a series of consecutive tokens [ s i, 1 ,    , s i,l i ] in x . Each span is replaced with a single [ MASK ] token, forming a corrupted text x corrupt . The model predicts the missing tokens in the spans from the corrupted text in an autoregressive manner, which means when predicting the missing tokens in a span, the model has access to the corrupted text and the previously predicted spans. To fully capture the interdependencies between different spans, we randomly permute the order of the spans, similar to the permutation language model (Yang et al., 2019). Formally, let Z m be the set of all possible permutations of the lengthm index sequence [1 , 2 ,    , m ] , and s z <i be [ s z 1 ,    , s z i  1 ] , we de-fine the pretraining objective as max  E z  Z m (cid:34) m (cid:88) i =1 log p  ( s z i | x corrupt , s z <i ) (cid:35) (1) We always generate the tokens in each blank following a left-to-right order, i.e. the probability of generating the span s i is factorized as: p  ( s i | x corrupt , s z <i ) = l i (cid:89) j =1 p ( s i,j | x corrupt , s z <i , s i,<j ) (2) We implement the autoregressive blank infilling objective with the following techniques. The input x is divided into two parts: Part A is the corrupted text x corrupt , and Part B consists of the masked spans. Part A tokens can attend to each other, but cannot attend to any tokens in B. Part B tokens can attend to Part A and antecedents in B, but cannot attend to any subsequent tokens in B. To enable autoregressive generation, each span is padded with special tokens [ START ] and [ END ] , for input and 321 output respectively. In this way, our model automatically learns a bidirectional encoder (for Part A) and a unidirectional decoder (for Part B) in a unified model. The implementation of GLM is illustrated in Figure 2. x 2 [M] x 4 [M] x 1 x 2 [M] x 4 [M] [S] x 5 x 6 [S] x 3 x 5 x 6 [S] x 3 Position 1 1 2 3 4 5 5 5 5 3 3 [M] x 4 [M] [S] x 5 x 6 x 1 x 2 [M] x 4 [M] [S] x 5 x 6 [S] x 3 5 6 [S] 3 [M] [M] Position 2 0 0 0 0 0 1 2 3 1 Position 1 1 2 3 4 5 5 5 5 3 Position 2 0 0 0 0 0 1 2 3 1 Figure 2: GLM pretraining. (a) The original text is [ x 1 , x 2 , x 3 , x 4 , x 5 , x 6 ] . Two spans [ x 3 ] and [ x 5 , x 6 ] are sampled. (b) Replace the sampled spans with [M] in Part A, and shuffle the spans in Part B. (c) GLM autoregressively generates Part B. Each span is prepended with [S] as input and appended with [E] as output. 2D positional encoding represents interand intra-span positions. (d) Self-attention mask. Grey areas are masked out. Part A tokens can attend to themselves (blue frame) but not B. Part B tokens can attend to A and their antecedents in B (yellow and green frames correspond to the two spans). [ M ] := [ MASK ] , [ S ] := [ START ] , and [ E ] := [ END ] . We randomly sample spans of length drawn from a Poisson distribution with  = 3 . We repeatedly sample new spans until at least 15% of the original tokens are masked. Empirically, we have found that the 15% ratio is critical for good performance on downstream NLU tasks. Both new objectives are defined in the same way as the original objective, i.e. Eq. 1. The only difference is the number of spans and the span lengths. In the previous section, GLM masks short spans and is suited for NLU tasks. However, we are interested in pretraining a single model that can handle both NLU and text generation. We then study a multi-task pretraining setup, in which a second objective of generating longer text is jointly optimized with the blank infilling objective. We consider the following two objectives:  Document-level. We sample a single span whose length is sampled from a uniform distribution over 50%100% of the original length. The objective aims for long text generation. Sentence-level. We restrict that the masked spans must be full sentences. Multiple spans (sentences) are sampled to cover 15% of the original tokens. This objective aims for seq2seq tasks whose predictions are often complete sentences or paragraphs. GLM uses a single Transformer with several mod-ifications to the architecture: (1) we rearrange the order of layer normalization and the residual connection, which has been shown critical for large-scale language models to avoid numerical errors (Shoeybi et al., 2019); (2) we use a single linear layer for the output token prediction; (3) we replace ReLU activation functions with GeLUs (Hendrycks and Gimpel, 2016). One of the challenges of the autoregressive blank infilling task is how to encode the positional information. Transformers rely on positional encodings to inject the absolute and relative positions of the tokens. We propose 2D positional encodings to address the challenge. Specifically, each token is encoded with two positional ids. The first positional id represents the position in the corrupted text x corrupt . For the masked spans, it is the position of the corresponding [ MASK ] token. The second positional id represents the intra-span position. For tokens in Part A, their second positional ids are 0 . For tokens in Part B, they range from 1 to the length of the span. The two positional ids are projected into two vectors via learnable embedding tables, which are both added to the input token embeddings. reconstructing them. It is an important difference as compared to other models. For example, XLNet (Yang et al., 2019) encodes the original position so that it can perceive the number of missing tokens, and SpanBERT (Joshi et al., 2020) replaces the span with multiple [MASK] tokens and keeps the length unchanged. Our design fits downstream tasks as usually the length of the generated text is unknown beforehand. Typically, for downstream NLU tasks, a linear clas-sifier takes the representations of sequences or tokens produced by pretrained models as input and predicts the correct labels. The practices are different from the generative pretraining task, leading to inconsistency between pretraining and finetuning. Instead, we reformulate NLU classification tasks as generation tasks of blank infilling, following PET (Schick and Schtze, 2020a). Specifically, given a labeled example ( x , y ) , we convert the input text x to a cloze question c ( x ) via a pattern containing a single mask token. The pattern is written in natural language to represent the semantics of the task. For example, a sentiment classification task can be formulated as {SENTENCE}. It's really [ MASK ] . The candidate labels y  Y are also mapped to answers to the cloze, called verbalizer v ( y ) . In sentiment classification, the labels positive and negative are mapped to the words good and bad. The conditional probability of predicting y given x is p ( y | x ) = p ( v ( y ) | c ( x )) (cid:80) y (cid:48) Y p ( v ( y (cid:48) ) | c ( x )) (3) where Y is the label set. Therefore the probability of the sentence being positive or negative is proportional to predicting good or bad in the blank. Then we finetune GLM with a cross-entropy loss (see Figure 3). For text generation tasks, the given context constitutes the Part A of the input, with a mask token appended at the end. The model generates the text of Part B autoregressively. We can directly apply the pretrained GLM for unconditional generation, or finetune it on downstream conditional generation tasks. In this section, we discuss the differences between GLM and other pretraining models. We are mainly concerned with how they can be adapted to downstream blank infilling tasks. Comparison with BERT (Devlin et al., 2019). As pointed out by (Yang et al., 2019), BERT fails to capture the interdependencies of masked tokens due to the independence assumption of MLM. Another disadvantage of BERT is that it cannot fill in the blanks of multiple tokens properly. To infer the probability of an answer of length l , BERT needs to perform l consecutive predictions. If the length l is unknown, we may need to enumerate all possible lengths, since BERT needs to change the number of [ MASK ] tokens according to the length. Comparison with XLNet (Yang et al., 2019). Both GLM and XLNet are pretrained with autoregressive objectives, but there are two differences between them. First, XLNet uses the original position encodings before corruption. During inference, we need to either know or enumerate the length of the answer, the same problem as BERT. Second, XLNet uses a two-stream self-attention mechanism, instead of the right-shift, to avoid the information leak within Transformer. It doubles the time cost of pretraining. Comparison with T5 (Raffel et al., 2020). T5 proposes a similar blank infilling objective to pretrain an encoder-decoder Transformer. T5 uses independent positional encodings for the encoder and decoder, and relies on multiple sentinel tokens to differentiate the masked spans. In downstream tasks, only one of the sentinel tokens is used, leading to a waste of model capacity and inconsistency between pretraining and finetuning. Moreover, T5 always predicts spans in a fixed left-to-right order. As a result, GLM can significantly outperform T5 on NLU and seq2seq tasks with fewer parameters and data, as stated in Sections 3.2 and 3.3. attention mask among bidirectional, unidirectional, and cross attention. However, UniLM always replaces masked spans with [MASK] tokens, which limits its ability to model the dependencies between the masked spans and their context. GLM feeds in the previous token and autoregressively generates the next token. Finetuning UniLM on downstream generation tasks also relies on masked language modeling, which is less efficient. UniLMv2 (Bao et al., 2020) adopts partially autoregressive modeling for generation tasks, along with the autoencoding objective for NLU tasks. Instead, GLM unifies NLU and generation tasks with autoregressive pretraining. We now describe our pretraining setup and the evaluation of downstream tasks. For a fair comparison with BERT (Devlin et al., 2019), we use BooksCorpus (Zhu et al., 2015) and English Wikipedia as our pretraining data. We use the uncased wordpiece tokenizer of BERT with 30k vocabulary. We train GLM Base and GLM Large with the same architectures as BERT Base and BERT Large , containing 110M and 340M parameters respectively. For multi-task pretraining, we train two Large-sized models with a mixture of the blank infilling objective and the document-level or sentence-level objective, denoted as GLM Doc and GLM Sent . Additionally, we train two larger GLM models of 410M (30 layers, hidden size 1024, and 16 attention heads) and 515M (30 layers, hidden size 1152, and 18 attention heads) parameters with document-level multi-task pretraining, denoted as GLM 410M and GLM 515M . To compare with SOTA models, we also train a Large-sized model with the same data, tokenization, and hyperparameters as RoBERTa (Liu et al., 2019), denoted as GLM RoBERTa . Due to resource limitations, we only pretrain the model for 250,000 steps, which are half of RoBERTa and BART's training steps and close to T5 in the number of trained tokens. More experiment details can be found in Appendix A. 3.2 SuperGLUE To evaluate our pretrained GLM models, we conduct experiments on the SuperGLUE benchmark (Wang et al., 2019) and report the standard metrics. SuperGLUE consists of 8 challenging NLU tasks. We reformulate the classification tasks as blank infilling with human-crafted cloze questions, following PET (Schick and Schtze, 2020b). Then we finetune the pretrained GLM models on each task as described in Section 2.3. The cloze questions and other details can be found in Appendix B.1. For a fair comparison with GLM Base and GLM Large , we choose BERT Base and BERT Large as our baselines, which are pretrained on the same corpus and for a similar amount of time. We report the performance of standard finetuning (i.e. classification on the [CLS] token representation). The performance of BERT with cloze questions is reported in Section 3.4. To compare with GLM RoBERTa , we choose T5, BART Large , and RoBERTa Large as our baselines. T5 has no direct match in the number of parameters for BERT Large , so we present the results of both T5 Base (220M parameters) and T5 Large (770M parameters). All the other baselines are of similar size to BERT Large . Table 1 shows the results. With the same amount of training data, GLM consistently outperforms BERT on most tasks with either base or large architecture. The only exception is WiC (word sense dis-ambiguation). On average, GLM Base scores 4.6% higher than BERT Base , and GLM Large scores 5.0% higher than BERT Large . It clearly demonstrates the advantage of our method in NLU tasks. In the setting of RoBERTa Large , GLM RoBERTa can still achieve improvements over the baselines, but with a smaller margin. Specifically, GLM RoBERTa outperforms T5 Large but is only half its size. We also find that BART does not perform well on the challenging SuperGLUE benchmark. We conjecture this can be attributed to the low parameter efficiency of the encoder-decoder architecture and the denoising sequence-to-sequence objective. Then we evaluate the GLM's performance in a multi-task setting (Section 2.1). Within one training batch, we sample short spans and longer spans (document-level or sentence-level) with equal chances. We evaluate the multi-task model for NLU, seq2seq, blank infilling, and zero-shot language modeling. are also shown in Table 1. We observe that with multi-task pretraining, GLM Doc and GLM Sent perform slightly worse than GLM Large , but still outperform BERT Large and UniLM Large . Among multitask models, GLM Sent outperforms GLM Doc by 1.1% on average. Increasing GLM Doc 's parameters to 410M (1.25  BERT Large ) leads to better performance than GLM Large . GLM with 515M parameters (1.5  BERT Large ) can perform even better. Sequence-to-Sequence. Considering the available baseline results, we use the Gigaword dataset (Rush et al., 2015) for abstractive summarization and the SQuAD 1.1 dataset (Rajpurkar et al., 2016) for question generation (Du et al., 2017) as the benchmarks for models pretrained on BookCorpus and Wikipedia. Additionally, we use the CNN/DailyMail (See et al., 2017) and XSum (Narayan et al., 2018) datasets for abstractive summarization as the benchmarks for models pretrained on larger corpora. The results for models trained on BookCorpus and Wikipedia are shown in Tables 3 and 4. We observe that GLM Large can achieve performance matching the other pretraining models on the two generation tasks. GLM Sent can perform better than GLM Large , while GLM Doc performs slightly worse than GLM Large . This indicates that the document-level objective, which teaches the model to extend the given contexts, is less helpful to conditional generation, which aims to extract useful information from the context. Increasing GLM Doc 's parameters to 410M leads to the best performance on both tasks. The results for models trained on larger corpora are shown in Table 2. GLM RoBERTa can achieve performance matching the seq2seq BART model, and outperform T5 and UniLMv2. with the surrounding context (Zhu et al., 2019; Donahue et al., 2020; Shen et al., 2020). GLM is trained with an autoregressive blank infilling objective, thus can straightforwardly solve this task. We evaluate GLM on the Yahoo Answers dataset (Yang et al., 2017) and compare it with Blank Language Model (BLM) (Shen et al., 2020), which is a specifically designed model for text infilling. From the results in Table 5, GLM outperforms previous methods by large margins (1.3 to 3.9 BLEU) and achieves the state-of-the-art result on this dataset. We notice that GLM Doc slightly underperforms GLM Large , which is consistent with our observations in the seq2seq experiments. Language Modeling. Most language modeling datasets such as WikiText103 are constructed from Wikipedia documents, which our pretraining dataset already contains. Therefore, we evaluate the language modeling perplexity on a held-out test set of our pretraining dataset, which contains about 20M tokens, denoted as BookWiki. We also evaluate GLM on the LAMBADA dataset (Paperno Unidirectional Bidirectional 20 30 40 50 60 A cc u r a c y LAMBADAGLM Doc GLM Doc  2D GLM 410M GLM 515M GPT Large Unidirectional Bidirectional 8 10 12 14 16 P e r p l e x il y Books&Wiki Test Figure 4: Zero-shot language modeling results. et al., 2016), which tests the ability of systems to model long-range dependencies in text. The task is to predict the final word of a passage. As the baseline, we train a GPT Large model (Radford et al., 2018b; Brown et al., 2020) with the same data and tokenization as GLM Large . The results are shown in Figure 4. All the models are evaluated in the zero-shot setting. Since GLM learns the bidirectional attention, we also evaluate GLM under the setting in which the contexts are encoded with bidirectional attention. Without generative objective during pretraining, GLM Large cannot complete the language modeling tasks, with perplexity larger than 100. With the same amount of parameters, GLM Doc performs worse than GPT Large . This is expected since GLM Doc also optimizes the blank infilling objective. Increasing the model's parameters to 410M (1.25  of GPT Large ) leads to a performance close to GPT Large . GLM 515M (1.5  of GPT Large ) can further outperform GPT Large . With the same amount of parameters, encoding the context with bidirectional attention can improve the performance of language modeling. Under this setting, GLM 410M outperforms GPT Large . This is the advantage of GLM over unidirectional GPT. We also study the contribution of 2D positional encoding to long text generation. We find that removing the 2D positional encoding leads to lower accuracy and higher perplexity in language modeling. Summary. Above all, we conclude that GLM effectively shares model parameters across natural language understanding and generation tasks, achieving better performance than a standalone BERT, encoder-decoder, or GPT model. 3.4 Ablation Study Table 6 shows our ablation analysis for GLM. First, to provide an apple-to-apple comparison with BERT, we train a BERT Large model with our implementation, data, and hyperparameters (row 2). The performance is slightly worse than the official BERT Large and significantly worse than GLM Large . It confirms the superiority of GLM over Masked LM pretraining on NLU tasks. Second, we show the SuperGLUE performance of GLM finetuned as sequence classifiers (row 5) and BERT with cloze-style finetuning (row 3). Compared to BERT with cloze-style finetuning, GLM benefits from the autoregressive pretraining. Especially on ReCoRD and WSC, where the verbalizer consists of multiple tokens, GLM consistently outperforms BERT. This demonstrates GLM's advantage in handling variable-length blank. Another observation is that the cloze formulation is critical for GLM's performance on NLU tasks. For the large model, cloze-style finetuning can improve the performance by 7 points. Finally, we compare GLM variants with different pretraining designs to understand their importance. Row 6 shows that removing the span shuffling (always predicting the masked spans from left to right) leads to a severe performance drop on SuperGLUE. Row 7 uses different sentinel tokens instead of a single [ MASK ] token to represent different masked spans. The model performs worse than the standard GLM. We hypothesize that it wastes some modeling capacity to learn the different sentinel tokens which are not used in downstream tasks with only one blank. In Figure 4, we show that removing the second dimension of 2D positional encoding hurts the performance of long text generation. We note that T5 is pretrained with a similar blank infilling objective. GLM differs in three aspects: (1) GLM consists of a single encoder, (2) GLM shuffles the masked spans, and (3) GLM uses a single [MASK] instead of multiple sentinel tokens. While we cannot directly compare GLM with T5 due to the differences in training data and the number of parameters, the results in Tables 1 and 6 have demonstrated the advantage of GLM. Pretrained Language Models. Pretraining large-scale language models significantly improves the performance of downstream tasks. There are three types of pretrained models. First, autoencoding models learn a bidirectional contextualized encoder for natural language understanding via denoising objectives (Devlin et al., 2019; Joshi et al., 2020; Yang et al., 2019; Liu et al., 2019; Lan et al., 2020; Clark et al., 2020). Second, autoregressive models are trained with a left-to-right language modeling objective (Radford et al., 2018a,b; Brown et al., 2020). Third, encoder-decoder models are pretrained for sequence-to-sequence tasks (Song et al., 2019; Lewis et al., 2019; Bi et al., 2020; Zhang et al., 2020). Among encoder-decoder models, BART (Lewis et al., 2019) conducts NLU tasks by feeding the same input into the encoder and decoder, and taking the final hidden states of the decoder. Instead, T5 (Raffel et al., 2020) formulates most language tasks in the text-to-text framework. However, both models require more parameters to outperform autoencoding models such as RoBERTa (Liu et al., 2019). UniLM (Dong et al., 2019; Bao et al., 2020) unifies three pretraining models under the masked language modeling objective with different attention masks. NLU with linear classifiers on the learned representations. GPT-2 (Radford et al., 2018b) and GPT-3 (Brown et al., 2020) show that generative language models can complete NLU tasks such as question answering by directly predicting the correct answers without finetuning, given task instructions or a few labeled examples. However, generative models require much more parameters to work due to the limit of unidirectional attention. Recently, PET (Schick and Schtze, 2020a,b) proposes to reformulate input examples as cloze questions with patterns similar to the pretraining corpus in the few-shot setting. It has been shown that combined with gradient-based finetuning, PET can achieve better performance in the few-shot setting than GPT-3 while requiring only 0.1% of its parameters. Similarly, Athiwaratkun et al. (2020) and Paolini et al. (2020) convert structured prediction tasks, such as sequence tagging and relation extraction, to sequence generation tasks. Blank Language Modeling. Donahue et al. (2020) and Shen et al. (2020) also study blanking infilling models. Different from their work, we pre-train language models with blank infilling objectives and evaluate their performance in downstream NLU and generation tasks. GLM is a general pretraining framework for natural language understanding and generation. We show that the NLU tasks can be formulated as conditional generation tasks, and therefore solvable by autoregressive models. GLM unifies the pretraining objectives for different tasks as autoregressive blank infilling, with mixed attention masks and the novel 2D position encodings. Empirically we show that GLM outperforms previous methods for NLU tasks and can effectively share parameters for different tasks. The work is supported by the Distinguished Young Scholar(61825602), Academy of Artificial Intelligence NSFC for Distinguished and Beijing (BAAI). Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining Text Encoders as Discriminators Rather Than Generators. In ICLR 2020 . Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , pages 177190. Springer. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL 2019 , pages 41714186. Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. pages 24922501. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS 2019 , pages 13042 13054.
abstain abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain result abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method method method abstain method method abstain method method abstain method method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other other other other other other other other other other other other other other other other other objective abstain result abstain result other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain