Recent work in neural generation has attracted significant interest in controlling the form of text, such as style, persona, and politeness. However, there has been less work on controlling neural text generation for content. This paper introduces the notion of Content Transfer for long-form text generation, where the task is to generate a next sentence in a document that both fits its context and is grounded in a content-rich external textual source such as a news story. Our experiments on Wikipedia data show significant improvements against competitive baselines. As another contribution of this paper, we release a benchmark dataset of 640k Wikipedia referenced sentences paired with the source articles to encourage exploration of this new task. Recent work in neural natural language generation (NLG) has witnessed a growing interest in controlling text for various form-related and linguistic properties, such as style (Ficler and Goldberg, 2017), affect (Ghosh et al., 2017), politeness (Sennrich et al., 2016), persona (Li et al., 2016b) voice (Yamagishi et al., 2016), grammatical correctness (Ji et al., 2017), and length (Kikuchi et al., 2016). This trend offers the promise of empowering existing authoring tools such as Grammarly, Google Smart Compose, and Microsoft Word with the ability to control a much greater variety of textual properties, which are currently mostly limited to grammar, spelling, word choice, and wordiness. What has been relatively less explored in neural NLG research is the ability to control the generation of a current sentence not only in its form , but also its content . 1 Consider for example Fig. 1, which illustrates a situation where an author edits a document (here a Wikipedia article), 1 Historically, NLG has focused on generation from structured content such as a database or semantic representation, but this paper is interested in generation from free-form text. and the goal is to generate or suggest a next sentence (shown in orange) to the author. This type of unconstrained, long-form text generation task (Mostafazadeh et al., 2016; Fan et al., 2018) is of course extremely difficult. Free-form generation can easily go astray due to two opposing factors. On one hand, ensuring that the generated output is of relatively good quality often comes at the cost of making it bland and devoid of factual content (Li et al., 2016a). On the other hand, existing techniques can help steer neural models away from blandness in order to produce more contentful outputs (using temperature sampling (Fan et al., 2018), GAN (Goodfellow et al., 2014), etc.), but often at the cost of hallucinating (Wiseman et al., 2017) words or concepts that are totally irrelevant. Neither situation provides a compelling experience to the user. What is clearly missing from the aforementioned authoring scenario is the notion of grounding : there is often a profusion of online resources that bear at least some relevance to any given document currently being written. Much of the general-purpose world knowledge is available in the form of encyclopedias (e.g., Wikipedia), books (e.g., Project Gutenberg, Google Books), and news articles. While the generation of good quality texts without any conditioning on exter-nal sources (Fan et al., 2018) might be an interesting research endeavor on its own, we argue that grounding can make the generation task much easier, e.g., as shown in Fig. 1 where a passage of a news article (green) can be reformulated considering the current context of the document (yellow) in order to produce a natural next sentence (or-ange). In light of this desideratum, this paper addresses the problem of grounded text generation, where the goal is to infuse the content or knowledge from an external source (e.g., a news article as in Fig. 1) in order to generate a follow-up sentence of an existing document. We see this as a form of Content Transfer , as other characteristics of the external sourcesuch as style and linguistic formare not controlled. In addition to formulating this new task, our work makes the following contributions: We provide a large dataset of 640k instances that contain parallel data of a source document (news articles), a context, and sentence to be produced. The latter two are extracted from Wikipedia, which is an attractive dataset for grounded generation as many of the statements in Wikipedia cite external sources (i.e., grounded in an external article). Finally, we also provide simple yet efficient models that condition both on the external article and the context of the current document. We compare our models against extractive and abstractive baselines, including summarization methods that simply try to condense the external article without considering the context of the document. Our experiments show that our models which incorporate the context gain 7.0 ROUGE-L F1 points in other words, treating our task as a summarization problem is not enough. Our human evaluations also show that models that are aware of the context generate relevant and fluent sentences that are coherent to the context. This research is concerned with the general problem of grounded authorship assistance, i.e., the task of suggesting text to insert or append in an existing document draft, in such a way that all the added content reflects information from external sources, such as news articles and books. This type of grounded generation task could take many forms, so we decided to formalize the task as follows, while still keeping the task both challenging and practically interesting. Given an external document (green in Fig. 1), and some existing curated text (yellow), the task is to generate a single update sentence (orange). This update sentence should be both relevant to the context and reflec-tive of the information contained in the document. This task bears some similarity with automatic summarization (Nenkova and McKeown, 2011), as a nave approach to the above problem is to append a one-sentence summary of the document to the curated text. While indeed related, the two tasks differ in two key points. First, the one-sentence summary must be contextually appropriate given the previous context of the curated text. Second, summarization is mostly concerned with finding salient information, butin the case of our taskinformation relevant to the context might actually only be auxiliary within the external document. Section 6 (Related Work) further contrasts our task with summarization. Formally we define our task as follows: given an existing curated text s and a document d describing novel information relevant to that text, the system must produce a revised text s (cid:48) that incorporates the most salient information from d . We restrict our focus to the cases where the revised text s (cid:48) can be obtained by appending the new information from d to the original curated text s . 2 In particular, we assume we can transform the old curated text s into the new text s (cid:48) by appending one additional update sentence x to s . 2 In general, updated information from d might demand substantial changes to s : perhaps core assumptions of s were contradicted, necessitating many removed and rewritten sentences. We postpone this complex setting to future work. This paper operates in a conventional supervised learning setting. For training data, we rely on a large dataset of existing curated text S = { s 1 , . . . , s n } , corresponding documents with novel information D = { d 1 , . . . , d n } , and the update sentences X = { x 1 , . . . , x n } . Our task is to generate the update sentence x i that could be appended to the curated text s i in order to incorporate the additional information from document d i . The goal would be to identify new information (in particular, d i \ s i ) that is most salient to the topic or focus of the text, then generate a single sentence that represents this information. A natural though difficult means of generating this additional update sentence x is to use a generative model conditioned on the information in the curated text s and the new document d . Recent methods inspired by successful neural machine translation systems have produced impressive results in abstractive summarization (Nalla-pati et al., 2016). Hence, our first step is to use the sequence-to-sequence encoder-decoder model (Bahdanau et al., 2015) with attention (Luong et al., 2015) for our task. This kind of model assumes that the output sentence can be generated word-by-word. Each output word x ti generated is conditioned on all prior words x <ti and an encoded representation of the context z : (cid:89) t p ( x ti |  x <ti , z ) (1) Context Agnostic Generative (CAG) Model: One simple baseline is to train a sequence-to-sequence model for the document d alone that does not directly incorporate information from the curated text s . Here, the algorithm is trained to generate the most likely update sentence  x = arg max p ( x | d ) . In this setting, we consider the reference document d i as the source and the update sentence to be generated x i as the target. The encoder and decoder do not directly see the information from the curated text s , but the update x inherently carries some information about it. The parameters of the model are learned from updates that were authored given the knowledge of the curated text. Hence, the model may capture some generalizations about the kinds of information and locations in d that are most likely to contribute novel information to s . Context Only Generative (COG) Model: This algorithm is trained to generate the most likely update sentence  x = arg max p ( x | s ) . This model is similar to CAG except that we consider the curated s i as the source. In this setting, there is no grounding of the content to be generated. Context Informed Generative (CIG) Model: An obvious next step is to incorporate information from the curated text s as well. We can concatenate the document and the curated text, and produce an encoded representation of this sequence. This approach incorporates information from both sources, though it does not differentiate them clearly. Thus, the model may struggle to identify which pieces of information are novel with respect to the curated text. To clearly identify the information that is already present in the curated text s , a model could encode s and d separately, then incorporate both signals into the generative procedure. Context Receptive Generative (CRG) Model: Our next step was to condition our generative process more concretely on the curated text s . We condition the generative process on the representation of s at each time step. Formally: z d = Encoder d ( d i ,  d ) (4) z s = Encoder s ( s i ,  s ) (5)  x i  (cid:89) t p ( x ti | [ x <ti ; z s ] , z d ) (6) where,  d and  s are the parameters of the encoder for the document d and encoder for the curated text s respectively, z d and z s are the encoded representations of the document d i and curated text s i respectively. At each time step of generation, the output is conditioned on the tokens generated up to the time step t concatenated with z s . Hence, the generative process is receptive of the context at each time step. Generative models that construct new sentences conditioned on the relevant context are compelling but have a number of modeling challenges. Such a model must both select the most relevant content and generate a fluent linguistic realization of this information. We also consider extractive models: approaches that select the most relevant sentence from the document d to append to the curated text s . These approaches can focus solely on the content selection problem and ignore the difficulties of generation. This simplification does come at a cost: the most effective sentence to add might require only a subset of information from some sentence in the document, or incorporate information from more than one sentence. Sum-Basic (SB): One common baseline is Sum-Basic, an extractive summarization technique that relies on word frequency statistics to select salient sentences (Nenkova and Vanderwende, 2005). As an initial step, unigram probabilities are computed from the set of input documents using relative frequency estimation. Then, sentences are selected one-by-one in greedy rounds until the summary budget is saturated. At each round, this model selects the most likely sentence according to the current unigram distribution. The selected sentence is added to the summary and removed from the pool of available sentences. The unigram probabilities of all words in the selected sentence are heuristically discounted (replaced by square root). Select-then-discount operations continue until the summary is written. Discounting is crucial to prevent repetition: once a word (or ideally a concept) has been selected for the summary, it is much less likely to be picked in a subsequent round. We use Sum-Basic as a Context Agnostic extractive model: we provide the document d as an input to the model and run Sum-Basic for exactly one round. The selected sentence is considered to be the update sentence x . Context Informed Sum-Basic (CISB): We developed a simple modification of the Sum-basic technique to incorporate information from the curated text s as context. Initial unigram probabilities are computed using word counts from both the curated text and the document. Next, for each sentence in the curated text, we apply just the discount procedure, updating the probability distribution as if those sentences were selected. Finally, we select the single sentence from the document that is most likely according to the resulting disFigure 2: Dataset creation process counted unigram probabilities. This simple modification of Sum-Basic helps select a sentence that is novel with respect to the curated text by lowering the probability of all words already present. Extractive CAG, CIG, CRG Models: Any generative model of x can also be used as an extractive model: we simply estimate the likelihood of each sentence in the document according to the model, and select the most likely one. Generative models may fail because either they are unable to select the most relevant information, or because the resulting sentence is ill-formed. Extractive ranking circumvents all errors due to generation and can help isolate model issues. Hybrid CAG, CIG, CRG Models: Since the document d can be quite large, a generative model may struggle to pick the most salient information based on the context. To simplify the generative modeling task, we can pre-filter the document toward only the most salient parts. We use the Context Informed Sum-Basic technique to first select the top five sentences from the document. We supply only these five sentences in place of the source document d , then apply the CAG, CIG, and CRG techniques described above. Our ideal dataset would capture the edits made to some curated reference text in light of a stream of new articles describing changes. For instance, one might maintain reference software documentation about a system, making additions or changes in light of incoming emails describing updates or additions. This type of data is unfortunately difficult to obtain due to privacy considerations. However, Wikipedia can provide a naturally-occurring body of text with references to primary sources. A substantial fraction of Wikipedia sentences include citations to supporting documentation, a ripe source of data for content transfer. That Corpus Input Output #Examples Rouge-1 R Gigaword (Graff and Cieri, 2003) 10 1 10 1 10 6 78.7 CNN/DailyMail (Nallapati et al., 2016) 10 2  10 3 10 1 10 5 76.1 WikiSum (Liu et al., 2018) 10 2  10 6 10 1  10 3 10 6 59.2 Content Transfer (this paper) 10 1  10 3 10 1  10 2 10 5 66.9 Table 1: Key characteristics of the dataset: approximate size of input and output instances, approximate dataset size, and recall of reference output against the source material, as a measure of dataset difficulty. said, some of the citations are quite difficult to follow or trust: broken URLs might lead to lost information; citations to books are difficult to consume given the large scope of information; etc. Therefore, we only consider cases where the reference links to some well-known news sources. Based on citation frequency, we selected a list of 86 domains, 3 primarily news outlets. During the data creation process we only considered citations belonging to one of these eighty six domains. We make this simplifying assumption for several reasons. First, our English Wikipedia dump contained approximately 23.7 million citation URLS belonging to 1.6 million domains; fine-grained fil-tering would be a daunting task. Our hand-vetted list of domains is a high-precision (albeit low-recall) means of selecting clean data. Second, we wanted to ground the generated text on credible, consistent, and well-written sources of information. Furthermore, well-known domains are readily available on Common Crawl, 4 leading to an easily-reproducible dataset. Fig. 2 illustrates the procedure used to create a dataset for the task described in Section 2 from Wikipedia. For each Wikipedia article, we extracted the plain text without markdown. When encountering a citation belonging to a selected do-main, we considered the sentence just before the citation to be generated based on the content of the citation. This sentence became our reference update sentence: the additional update sentence x added to the curated text s to produce the new text s (cid:48) . The k sentences prior to the target sentence in the Wikipedia article were considered to be the curated text s . In our case, we used a window of k = 3 sentences to select our context. The cited article acted as the document d , from which the appropriate update x can be generated. The HTML source of the citation was down-3 This list is provided in the data release of this paper. loaded from Common Crawl for reproducibility and consistency. The HTML derived from Common Crawl is then processed to get the plain text of the news article. The resulting dataset C consists of aligned tuples C = (cid:0) d i , s i , x i (cid:1) i  [1 ,n ] , where n is the total number of samples in the dataset. Alternatively, one might rely on Wikipedia edit history to create a dataset. In this setting, edits which include a new citation would act as the update x . Although this has the upside of identifying potentially complex, multi-sentence updates, preliminary analysis suggested that these edits are noisy. Editors may first generate the content in one edit, then add the citation in a subsequent edit, they may only rephrase a part of the text while adding the citation, or they may check in a range of changes across the document in a single edit. Our simpler sentence-based approach leads to an interesting dataset with fewer complications. Dataset Statistics and Analysis Table 1 describes some key statistics of our dataset and how it compares with other datasets used for similar tasks. The ROUGE-1 recall scores of reference output x against document d suggest this task will be difficult for conventional extractive summarization techniques. 5 We hypothesize that during content transfer, the language in document d often undergoes substantial transformations to fit the curated text s . The average unigram overlap (after stopword removal) between the document d and the reference update sentence x is 55 . 79 %; overlap of the curated text s and the reference update sentence x is 30 . 12 %. This suggests the reference update sentence x can be derived from the document d , though not extracted directly. Furthermore, the content of x is very different from the content of s but appears topically related. Our dataset consists of approximately 290k unique Wikipedia articles. Some heavily-cited 5 ROUGE-1 recall was computed on a sample of 50k instances from the entire dataset. articles include Timeline of investigations into Trump and Russia (2017)', List of England Test cricketers', and 2013 in science'. We randomly split the dataset into 580k training instances, 6049 validation instances, and 50k test instances, ensuring that any Wikipedia article appearing in the train set must not appear in validation or test. We evaluate our models using both automated metrics and, for a subset of promising systems, human assessment. One key evaluation is the similarity between the model generated update sentence and reference update sentence. We also ask human judges to assess grammaticality and coherence. Hyper-parameter settings: For all our experiments with generative models, we have used bidirectional encoder, 2 layers in encoder and decoder, RNN size of 128, word vector size of 100. We have used sentencepiece toolkit 6 to use byte-pair-encoding (BPE) with a vocabulary size of 32k. We used stochastic gradient descent optimizer and the stopping criterion was perplexity on the validation set. We filtered our dataset to contain instances which have length of the document between 50 and 2000 tokens, length of the curated text between 20 and 500 tokens and the length of the update sentence between 5 and 200 tokens. Our primary automated evaluation metric for system-generated update sentences is ROUGE-L F1 against reference update sentence, 7 though we also include BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2011) as additional indicators. ROUGE is a standard family of metrics for summarization tasks; ROUGE-L measures the longest common subsequence between the system and the reference, capturing both lexical selection and word order. Table 2 illustrates that this task is quite difficult for extractive techniques. Furthermore, the results emphasize the importance of having curated text as context when generating the update. In all experimental conditions, models aware of context perform much better than models agnostic of it. In contrast to Liu et al. (2018), generative approaches 6 https://github.com/google/sentencepiece 7 We use the pyrouge toolkit along with ROUGE-1.5.5: https://github.com/bheinzerling/pyrouge Model ROUGE-LBLEUMETEORSB 5.6 (5.65.7) 0.6 2.0 CISB 7.0 (7.07.1) 1.0 2.8 CAG 9.1 (9.09.2) 1.2 4.6 COG 13.5 (13.413.6) 1.7 3.5 CIG 16.0 (15.9-16.1) 3.5 5.3 CRG 14.7 (14.614.8) 2.6 4.5 Hybrid CAG 8.0 (7.98.0) 1.0 3.8 Hybrid CIG 15.0 (14.915.1) 2.7 4.7 Hybrid CRG 13.5 (13.413.6) 2.3 4.1 Extractive CAG 9.3 (9.29.3) 1.1 3.2 Extractive CIG 9.3 (9.29.3) 1.1 3.2 Extractive CRG 9.2 (9.19.3) 1.1 3.2 Oracle 28.8 (28.729.0) 11.0 10.9 Table 2: Automated metrics; 95% confidence interval in parentheses. outperformed hybrid, likely because we only had a single input document. Extractive CAG, CIG, and CRG all outperformed both Sum-Basic and the context informed variant. Extractive CAG was on-par with generative CAG, suggesting the generated sentences were of reasonable quality. However, generative CIG and CRG were substantially better: rewriting to match context was beneficial. The Oracle system of Table 2 aims to establish an upper limit attainable by extractive methods, using the following oracle experiment: For each test instance (cid:0) d i , s i , x i (cid:1) , we enumerate each extracted sentence e of document d i and select the one with highest ROUGE-L score as Oracle 's update sentence  x i (i.e.,  x i = arg max e  d i ROUGE-L ( x i , e ) ). Note this yields a very optimistic upper bound, as the same ground truth x i is used both to select an extractive sentence from a large pool of candidates and for final automatic metric scoring. 8 Nevertheless, these oracle results let us draw two conclusions: (1) They give us better perspective to assess the non-oracle systems, and we believe that their seemingly low 8 Previous work has shown that this type of oracle can yield upper bounds that are unrealistically high, and they tend to be above human performance (Och et al., 2004, Table 1). One remedy suggested by Och et al. is a round-robin oracle ensuring that the reference (ground truth) used by the argmax is distinct from that of the final automatic evaluation, but that scheme is only possible with a multi-reference test set. automatic evaluation scores are quite reasonable relative to the optimistic upper bound (e.g., CIGs ROUGE-Ls score is 55% of the oracle). (2) The oracle results suggest that humans are substantially changing the surface realization as they summarize for Wikipedia, as otherwise the oracle results would be much closer to maximum metric scores (i.e., 100%). This shows that extractive methods are not enough for this task, justifying our use of generation techniques. For careful evaluation of the performance of the most promising configurations (CAG and CIG models) we also asked human judges for quality assessments. We solicited several types of evaluation, including two relative comparisons between pairs of system outputs and an absolute quality evaluation of individual system outputs. Close to reference (Relative): The first relative comparison measured how accurately the generated update reflected information in the reference update. Here, the annotators saw only the reference update sentence and the outputs of two systems labeled A and B in a randomized order. We asked the annotators Which system output is closest in meaning to the reference update? The annotators could pick system A , system B , or indicate that neither was preferred. This is a simple evaluation task though potentially biased toward the sole reference update. Coherent to context (Relative): The second relative comparison measured whether the generated output contained salient information from the document written in a manner appropriate to the curated text. The annotators saw the document d , the curated text s , and the outputs of the two systems A and B , again in a random order. They were asked, Which system output is more accurate relative to the background information given in the snippet of the article? Each judge had to consider whether the information fits with the curated text and also whether system-generated content could be supported by the document. Four human judges each annotated 30 unique output pairs for these two relative comparison settings, a total of 240 relative judgments. Table 3 shows the results: the context-aware CIG system was substantially better in both settings. DUC Guidelines (Absolute): In addition, we performed an absolute quality evaluation following the guidelines from DUC 2007. 9 Each judge was presented with a single system output, then they were asked to evaluate five aspects of system output: grammaticality, non-redundancy, referential clarity, focus, and structure/coherence. For each aspect, the judge provided an assessment on a five-point scale: (1) Very Poor, (2) Poor, (3) Barely Acceptable, (4) Good, (5) Very Good. We gathered 120 additional judgments in this setting (4 judges, 30 outputs). Again, context-aware CIG substantially outperforms CAG across the board, as seen in Table 4. Observations: Systems unaware of the curated text s tend to generate long updates with repeated frequent words or phrases. Consider the ratio of unique tokens over the total number of tokens in the generated output, which we denote by R . A small R indicates many repeated tokens. We find that 88 % of the time this ratio R falls below 0 . 5 for the CAG model, i.e. for 88 % instances, more than 50 % of the words in the generated output are repeats. This number is relatively small  14 % for CIG and 20 % for CRG  in context aware models. In the reference updates only 0 . 21 % instances repeat more than 50 % of words. Figs. 3 and 4 show good and bad examples generated by the CIG model along with the document, curated text and the reference update. Table 5 has a set of updates generated by the CIG model as 9 http://duc.nist.gov/duc2007/ quality-questions.txt Document (News Article) sequels are fairly new to bollywood, but director sanjay gadhvi realised there was cash to be made from resurrecting his hit action thriller dhoom, by casting sexy young stars like hrithik rosha, aishwarya rai and abhishek bachchan in an even bigger game of cops and robbes...that the twist in dhoom 2's tail is not explained is yet another shortcoming. it's only roshan's charismatic performance as the criminal mastermind, and the sizzling chemistry he shares with rai's sassy cohort, that rescues this adventure from becoming an elongated tourism commercial. Curated Text (Wikipedia Context) it makes no lasting contributions to world cinema, but if two-and-a-half hours of disposable entertainment are all you're after, you could do far worse. l.a. weekly's david chute stated the film was, a movie meal as satisfying as this one can make you feel that nothing else matters. jaspreet pandohar of the bbc gave it a two-star rating, writing by roping in acclaimed action director alan amin to take care of the thrills and spills, you'd expect gadhvi to have spent time crafting out a sophisticated storyline instead of simply sending his cast on a cat-and-mouse chase around the globe. Reference Update it's only roshan's charismatic performance as the criminal mastermind, and the sizzling chemistry he shares with rai's sassy cohort, that rescues this adventure from becoming an elongated tourism commercial. well as the reference update. As we can see in examples 3 and 4, the CIG model misplaces the date but correctly generates the remaining content. In examples 1 and 2, the CIG model appears to successfully select the correct pronouns for co-reference resolution, though it gets confused as to when to use the pronoun or the named entity. Examples 5 and 6 represent failure cases due to missing words. The proposed content transfer task is clearly related to a long series of papers in summarization, including recent work with neural techniques (Rush et al., 2015; Nallapati et al., 2016). In particular, one recent paper casts the the task of generating an entire Wikipedia article as a multi-document summarization problem (Liu et al., 2018). Their best-performing configuration was a two-stage extractive-abstractive framework; a multi-stage approach helped circumvent the diffi-Document (News Article) anne kirkbride, who portrayed bespectacled, gravelly-voiced deirdre barlow in coronation street for more that four decades, has died. the 60-year-old, whose first appearance in the soap opera was in 1972, died in a manchester hospital after a short illness.... kirkbride had left the soap opera after she was diagnosed with non-hodgkin's lymphoma in 1993 but returned some months later after treatment and spoke candidly about how she had struggled with depression following the diagnosis... Curated Text (Wikipedia Context) in 1993, kirkbride was diagnosis with non-hodgkin's lymphoma. she spoke to the british press about her bout of depression following the diagnosis. she was cured within a year of being diagnosed. she was diagnosed with non-hodgkin's lymphoma. culties of purely abstractive methods given quite large input token sequences. Looking beyond the clear task similarity of authoring Wikipedia style content, there are several crucial differences in our approach. First, the goal of that paper is to author the whole page, starting from nothing more than a set of primary sources, such as news articles. In practice, however, Wikipedia articles often contain information outside these primary sources, including common sense knowledge, framing statements to set the article in context, and inferences made from those primary sources. Our task restricts the focus to content where a human editor explicitly decided to cite some external source. Hence, it is much more likely that the resulting summary can be derived from the external source content. Furthermore, we focus on the act of adding information to existing articles, rather than writing a complete article without any context. These two scenarios are clearly useful yet complementary: sometimes people want to produce a new reference text where nothing existed before; in other cases the goal is to maintain and update an existing reference. Another closely related task is update summarization (Dang and Owczarzak, 2008), where systems attempt to provide a brief summary of the novel information in a new article assuming the user has read a known set of prior documents. Our focus on curating an authoritative resource Reference Update Generated Update is a substantial difference. Also our datasets are substantially larger, enabling generative models to be used in this space, where prior update summarization techniques have been primarily extractive (Fisher and Roark, 2008; Li et al., 2015). For any generation task, it is important to address both the content (what' is being said) as well its style (how' it is being said). Recently, a great deal of research has focused on the how' (Li et al., 2018; Shen et al., 2017), including efforts to collect a parallel dataset that differs in politeness (Rao and Tetreault, 2018), to control author characteristics in the generated sentences (Prabhu-moye et al., 2018), to control the perceived personality traits of dialog responses (Zhang et al., 2018). We believe this research thread is complementary to our efforts on generating the what'. Another form of content transfer bridges across modalities: text generation given schematized or semi-structured information. Recent research has addressed neural natural language generation techniques given a range of structured sources: selecting relevant database records and generating natural language descriptions of them (Mei et al., 2016), selecting and describing slot-value pairs for task-specific dialog response generation (Wen et al., 2015), and even generating Wikipedia biography abstracts given Infobox information (Lebret et al., 2016). Our task, while grounded in external content, is different in that it leverages linguistic grounding as well as prior text context when generating text. This challenging setting enables a huge range of grounded generation tasks: there are vast amounts of unstructured textual data. This article highlights the importance of the task of content transfer : generation guided by an existing curated text to set context and tone, and grounded in a new source providing useful information. information. We demonstrate how multiple models can address this challenging problem on a novel dataset derived from Wikipedia and Common Crawl. This dataset is released to the community along with scripts and models. 10 We find this setting particularly promising given the opportunity for human interaction: in contrast to approaches that do not rely on human-generated context, we establish a collaboration between user and computer. Each newly suggested sentence can be rejected, accepted, or edited before inclusion, and the edits can provide more training data. We believe there are many natural extensions to this work. The models described here are mostly extensions of existing approaches; approaches targeting novelty detection, focus, and document structure could lead to substantial improvements. We could apply models in series to incorporate changes for a set of documents. Future work could also explore changes that modify existing content rather than simply appending. We are grateful to the anonymous reviewers, as well as Alan W. Black, Chris Brockett, Bill Dolan, Sujay Jauhar, Michael Gamon, Jianfeng Gao, Dheeraj Rajagopal, and Xuchao Zhang for their helpful comments and suggestions on this work. We also thank Emily Ahn, Khyati Chandu, Ankush Das, Priyank Lathwal, and Dheeraj Ra-jagopal for their help with the human evaluation.
abstain abstain method result objective other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain result objective abstain method method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain method abstain method abstain abstain abstain method abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain other abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain result abstain abstain other other other other other other other other other other abstain other other abstain other method other other method other method other other method other other method other objective abstain objective abstain objective abstain method abstain method abstain other other