We present a simple approach for text infilling , the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modelinga special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling , can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories. Text infilling is the task of predicting missing spans of text which are consistent with the preceding and subsequent text. 1 Systems capable of infilling have the potential to enable rich applications such as assisting humans in editing or revising text (Shih et al., 2019), connecting fragmented ideas (AI21, 2019), and restoring ancient documents (Assael et al., 2019). Rather than targeting a particular application, our goal here is to provide a general, flexible, and simple infilling framework which can convincingly infill in a variety of domains. ing remarkably coherent text (Zellers et al., 2019; See et al., 2019), (2) efficient at generating text, and (3) conceptually simple, but cannot infill effectively as they can only leverage context in a single direction (usually the past). On the other hand, strategies such as BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2019) are able to infill using both preceding and subsequent text. However, their use of bidirectional attention limits their infilling capabilities to fixed-length spans. This is problematic asfor many applicationswe may not know the length of a missing span a priori . Zhu et al. (2019) propose a method capable of infilling variable-length spans, but it uses a specialized architecture and hence cannot easily leverage large-scale pre-trained models. In this work, we present infilling by language modeling (ILM), a simple framework which enables LMs to infill variable-length spans while preserving their aforementioned benefits: generation quality, efficient sampling, and conceptual simplicity. Our framework involves a straightforward formulation of the infilling task which, as we demonstrate, can be learned effectively by existing LM architectures. As shown in Figure 1, our approach concatenates artificially-masked text with the text which was masked, and adopts a standard LM training (or fine-tuning) procedure on such examples. Once trained, infilling can be performed for a document with blanks by using the LM to generate text and then replacing the blanks with this text. In addition to its conceptual simplicity, our experiments show that ILM enables off-the-shelf LMs to infill effectively. Furthermore, we find that infilling performance improves when starting from a large-scale pre-trained LM (as opposed to training from scratch), suggesting an additional benefit of using our model-agnostic framework compared to approaches which require specialized architectures. We provide an interactive web demo of models trained under our framework. This demo can infill multiple variable-length spans with different granularities (e.g. words, n-grams, and sentences) on the domains of short stories, scientific abstracts, and song lyrics: https://chrisdonahue.com/ilm . All code, data, and trained models are available at https://github.com/chrisdonahue/ilm and also on the CodaLab platform at https: //worksheets.codalab.org/worksheets/ 0x9987b5d9cce74cf4b2a5f84b54ee447b . The task of infilling is to take incomplete text  x containing one or more missing spans, and return completed text x . Let [ blank ] be a placeholder for a contiguous sequence (span) of one or more missing tokens. Then, incomplete text  x is a sequence of tokens some of which are [ blank ]. In order to map  x to x , an infilling strategy must specify both how many and which tokens to generate for each [ blank ]. Note that there may be many reasonable x for a given  x . Hence, we are interested in learning a distribution p ( x |  x ) . In this section, we describe our ILM framework. We first outline a simple reparametrization of the infilling task. Then, we define a procedure for automatically generating suitable training examples which can be fed to an off-the-shelf LM. Fedus et al. (2018) explore an infilling framework where LMs are trained on concatenations of  x and x , i.e., they use LMs to directly predict x given  x . While their approach is effective at infilling individual words, it is somewhat redundant as the model must predict the unmasked text in  x . Additionally, a model is not guaranteed to exactly reproduce the unmasked text. Instead, we make the trivial observation that it suffices to predict only the missing spans y which will replace the [ blank ] tokens in  x . We can then construct x by simply replacing [ blank ] tokens in  x with predicted spans y in a deterministic fashion. In order to handle multiple variable-length spans, we pose y as the concatenation of all missing spans separated by special [ answer ] tokens (one [ answer ] per [ blank ]) (Figure 1). We can thus cast infilling as learning p ( y |  x ) without loss of generality. Given a corpus consisting of complete text examples, our framework first manufactures infilling examples and then trains an LM on these examples. To produce an infilling example for a given x , we first sample an  x from a stochastic function Mask ( x ) which randomly replaces some number of spans in x with [ blank ] tokens. Then, we concatenate together the spans which were replaced separated by [ answer ] tokensto form a training target y . Finally, we construct the complete infilling example by concatenating  x , [ sep ], and y (see Figure 2 for a complete example). We train (or fine-tune) LMs on these infilling examples using standard LM training methodology, yielding models of the form p  ( y |  x ) . Specifically, we train GPT-2 (Radford et al., 2019) off the shelf, but any LM can potentially be used. This framework has several advantages. First, it incurs almost no computational overhead compared to language modeling. Specifically, if there are k missing spans in  x , the concatenation of  x and y contains only 2 k +1 more tokens than x (one [ blank ] and one [ answer ] per missing span plus one [ sep ]). As k is usually small (averaging around 2 per example in our experiments), sequence lengths remain similar to those encountered for the same x during language modeling. In contrast, using LMs to directly predict x from  x as in Fedus et al. (2018) effectively doubles the sequence length of x . This is particularly problematic when considering models like GPT-2 whose memory usage grows quadratically with sequence length. Second, our framework requires minimal change (three additional tokens) to an existing LM's vocabulary. Finally, because the entirety of  x is in the past when predicting y , the ILM framework combines the ability to attend to incorporate context on both sides of a blank with the simplicity of decoding from LMs. We design our experiments to determine if training an off-the-shelf LM architecture with our ILM framework can produce effective infilling models for a variety of datasets. Specifically, we train on three datasets of different sizes and semantics (details in Appendix A): short STORIES (Mostafazadeh et al., 2016), CS paper ABSTRACTS , and song LYRICS . A benefit of the ILM framework is that it can be trained to infill spans corrupted by arbitrary mask functions. Here, we explore a mask function which simultaneously trains models to infill different granularities of text; specifically, words, n-grams, sentences, paragraphs, and documents. By using a unique special token per granularity (e.g. [ blank word ]), this mask function offers users coarse but intuitive control over the length of the spans to be infilled. We configure our mask function to mask each token in a given document with around 15 % probability, echoing the configuration of Devlin et al. (2019). However, instead of masking individual tokens uniformly at random, we perform a preorder traversal of the granularity hierarchy tree, randomly masking entire subtrees with 3 % probability. For the datasets we consider, this results in a marginal token mask rate of about 15 % (details in Appendix B). While we train to infill several different granularities, we primarily evaluate and discuss the ability of our models to infill sentences for brevity. Quantitative results of our models on other granularities can be found in Appendix D, and granularity functionality can also be explored in our web demo. (Appendix C) while varying the infilling strategy and dataset. In addition to our proposed ILM strategy for infilling, we consider three baseline strategies: (1) language modeling (LM; infilling based only on past context), (2) reverse language modeling (LM-Rev; infilling based only on future context), and (3) language modeling based on all available context (LM-All). LM-All simply concatenates x and  x together as in Fedus et al. (2018). LM-All represents arguably the simplest way one could conceive of infilling with LMs, but results in long sequence lengths. Training examples for all strategies are depicted in Figure 2. For each strategy, we also vary whether training is initialized from the pre-trained GPT-2 model or from scratch. Despite discrepancies between the pre-training and our fine-tuning for most infilling strategies, all of the infilling experiments initialized from the pre-trained checkpoint performed better than their from-scratch counterparts. This indicates that ILM can effectively leverage large-scale language modeling pre-training to improve infilling performance. Henceforth, we will only discuss the models initialized from the pre-trained checkpoint, though we report quantitative performance for all models in Appendix D. For the models trained on STORIES and ABSTRACTS , we trained models to convergence using early stopping based on the validation set perplexity (PPL) of each model computed only on the masked tokens. These models took about a day to reach STOABSLYR Length LM 18.3 27.9 27.7 1.00 LM-Rev 27.1 46.5 34.3 1.00 LM-All 15.6 22.3 21.4 1.81 ILM 15.6 22.4 22.6 1.01 Table 1: Quantitative evaluation results. their early stopping criteria on a single GPU. For the larger LYRICS dataset, we trained models for 2 epochs (about two days on a single GPU). We evaluate the quantitative performance of our models on the sentence infilling task by measuring PPL on test data. 3 In this setting, a sentence is selected at random and masked out, and we measure the likelihood assigned by a model to the masked sentence in the context of the rest of the document. Regardless of differences in the ordering and number of tokens that each strategy uses to represent a test example, PPL is always computed only for the span of tokens comprising the original sentence (e.g. green tokens in Figure 2). Table 1 shows that across all datasets, ILM outperforms models which see only past or future context (LM and LM-Rev respectively), implying that our proposed framework is able to take advantage of bidirectional context despite using unidirectional models. Additionally, while one might expect LM-All to outperform ILM because its training examples more closely resemble those of standard LMs, ILM achieves similar performance to LM-All. This indicates that GPT-2 is able to effectively learn the syntax of ILM examples and achieve reasonable infilling performance with shorter sequences (and hence with much less memory usage). mod-3 Overlap-based metrics such as BLEU score (Papineni et al., 2002) are not appropriate for evaluating infilling as there are many realistic infills that have no word-level overlap with the original, e.g., a sandwich instead of leftover pasta. eling compared to the models which were trained only on language modeling (Appendix D.1). This suggests that ILM does not just repurpose LMs to infill, but rather extends their capabilities while maintaining their original functionality. In addition to our quantitative evaluation, we seek to evaluate the qualitative performance of ILM. To this end, we sample a story from the STORIES test set and randomly replace one of its five human-written sentences with a model output. Then, we task human annotators on Amazon Mechanical Turk with identifying which of the sentences in a story was machine-generated (details in Appendix E). We compare our ILM model to three baseline infilling strategies: an LM (context beyond the replaced sentence was discarded), the best model (self-attention; SA) from Zhu et al. (2019), and the pre-trained BERT (base) model (Devlin et al., 2019). All approaches except for BERT were first fine-tuned on the STORIES dataset. To infill using BERT, we replace the tokens representing the original sentence with mask tokens, and then generate text by replacing mask tokens one at a time (con-ditioning on previously-generated tokens). While vocabulary differences make it is less useful to compare PPL for the SA and BERT baselines to our GPT-2-based strategies, we can still meaningfully compare them in this human evaluation setting. For each approach we compute a score , which we define as the percentage of examples where the annotator did not correctly identify the machine-generated sentence. Therefore, a higher score implies a better (more natural, human-like) model. We collect 100 responses for each model and report the scores in Table 2, with qualitative examples in Figure 3 and Appendix E. Of the four strategies, ILM achieves the highest score, implying that sentences infilled by ILM are harder for humans to recognize as fake than those produced by other strategies. Somewhat surprisingly, we observed that despite only observing past context the LM model performed better than BERT and SA. BERT may have performed poorly due to the intrinsic difficulty of finding convincing infills with a precise length in tokens. SA may have performed poorly because, unlike LM and ILM, it was not initialized from a large-scaled pre-trained LM. Methodology. A number of systems have the capability to infill but have practical drawbacks. Many systems are unable to automatically determine span length, and thus, can only infill fixed-length spans (Fedus et al., 2018; Devlin et al., 2019; Yang et al., 2019; Joshi et al., 2019; Gu et al., 2019; Liu et al., 2019). Methods such as BERT present additional challenges during inference (Wang and Cho, 2019). Rudinger et al. (2015) frame narrative cloze as a generation task and employ language models, but they only consider one infill of a fixed length. Zhu et al. (2019); Shen et al. (2020) infill multiple variable-length sequences, but these approaches require the masked context to be iteratively updated and reprocessed to fill in blanks one a time. In contrast, our approach appends infilled text to the context and does not require reprocessing the entire input sequence for each blank. AI21 (2019) train an LM which can fill in the middle of a paragraph given the first and last sentencesour work generalizes to such capabilities. Task. The cloze task (Taylor, 1953) evaluates language proficiency by asking systems to fill in randomly-deleted words by examining context. Cloze has been extended in the forms of discourse (Deyes, 1984) and narrative cloze (Cham-bers and Jurafsky, 2008), which remove phrases and narrative events respectively. Recently, cloze has been used not only for evaluation, but also to improve text generation quality (Fedus et al., 2018) and transfer learning (Devlin et al., 2019) (under the name masked language modeling). Text infilling can be thought of as generalizing the cloze task from single words to spans of unknown length. Raf-fel et al. (2019) explore infilling as a pre-training objective to improve downstream performance on inference tasks; our work focuses on generation. Story generation. Recent work seeks to generate stories given a title and storyline (Yao et al., 2019), entities (Clark et al., 2018), premise (Fan et al., 2018), or surrounding context and rare words (Ippolito et al., 2019). Our work differs in that we aim to build systems capable of making predictions based only on text context, rather than aspects specific to stories (e.g. storyline). We presented a simple strategy for the task of infilling which leverages language models. Our approach is capable of infilling sentences which humans have difficulty recognizing as machine-generated. Furthermore, we demonstrated that our infilling framework is effective when starting from large-scale pre-trained LMs, which may be useful in limited data settings. In future work, we plan to incorporate these features into co-creation systems which assist humans in the writing process. We hope that our work encourages more investigation of infilling, which may be a key missing element of current writing assistance tools. This work was funded by DARPA CwC under ARO prime contract no. W911NF-15-1-0462. We thank all reviewers for their helpful comments.
method abstain objective method result result abstain abstain objective abstain abstain abstain abstain abstain method objective result abstain result result method other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain method method method method method abstain method abstain abstain abstain method method abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method method method method abstain method method method abstain abstain abstain abstain abstain other other other other other other objective other other other other other other abstain other other objective method method objective abstain method other other other