The introduction of immensely large causal language models (CLMs) has rejuvenated the interest in open-ended text generation. However, controlling the generative process for these Transformer-based models is at large an unsolved problem. Earlier work has explored either plug-and-play decoding strategies or more powerful but blunt approaches such as prompting. There hence currently exists a trade-off between fine-grained control and the capability for more expressive high-level instructions. To alleviate this trade-off, we propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. We propose a resource-efficient method for converting a pre-trained CLM into this architecture and demonstrate its potential in various experiments, including the novel task of contextualized word inclusion. Our method provides strong results in multiple experimental settings, proving itself to be both expressive and versatile. 1 1 Introduction A causal language model (CLM) is a language model trained using a simple next-token prediction objective. Current CLMs are typically based on the Transformer architecture (Vaswani et al., 2017), which has resulted in unprecedented text generation capabilities (Radford et al., 2018a,b; Brown et al., 2020). Even so, the generation process of a CLM is difficult to control, as one is forced to gradually decode the next-step prediction one token at a time. This inhibits the applicability of CLMs when one intends for the generated text to fulfill certain criteria, and not only be a linguistically sound continuation in a given context. the generated text to counter the many biases that modern CLMs have been shown to possess (Bor-dia and Bowman, 2019). However, most applications require a greater degree of control, as one often wishes to steer the text generation in a specific direction, such as generating a story to a given plot (Li et al., 2013; Yao et al., 2019; Riedl, 2021), or sticking to a certain topic (Keskar et al., 2019). Some areas require stringent and fine-grained control, as the many data-to-text tasks (Gardent et al., 2017; Leppanen et al., 2017; Koncel-Kedziorski et al., 2019), which necessitates that the generated text mediates very specific information and facts. Due to this apparent need for controllable text generation, recent work (see Section 2.1) has explored different methods to steer and constrain the generation process of a CLM. There are mainly two lines of research in this area. The more traditional approach focuses on fine-grained control and how to steer the generation process at arbitrary points, while still adhering to the current context. This is often achieved by independently modifying the predicted vocabulary distribution at each decoding step. However, this decouples the CLM from the control method, prohibiting the CLM's ability to plan accordingly and thus severely limits the type of control that can be formulated. The second approach instead opts for more expressive and high-level control, letting the CLM itself interpret and incorporate the instruction into the text generation. This is often done via either a fine-tuning objective or, as is currently common, by formulating the instruction as a textual context (referred to as prompting). Although expressive, these approaches are less effective than the previous ones in controlling generation at specific points. This is due to the prompt's influence being negatively correlated with the distance from the prompt to the next predicted token (Zou et al., 2021), making prompting difficult for nonadjacent text. In an attempt to bridge the gap between fine-grained control and the expressiveness of prompts, we propose an architecture that permits long-distance and independent prompting throughout the generation process. This architecture has an encoder-decoder setup, where the encoder influences the decoder via a novel non-residual attention schema. Along with theoretical arguments for the benefits of this architecture, we provide a resource-efficient self-supervised method for converting a pre-trained CLM into this setup. In addition to evaluating on the original CommonGen dataset (Lin et al., 2020), we propose a new contextualized version of CommonGen, called Contextualized CommonGen (C 2 GEN ) and evaluate relevant methods on it. This new dataset extends the task to generating a sentence which includes a given set of words, while simultaneously adhering to a given context. We find that no previous solution is capable of handling this task, either barely including 50% of the target words, or not generating text of satisfactory quality. Our Contributions: (1) An encoder-decoder architecture based on a novel attention module which enables prompting at arbitrary time steps. (2) A resource-efficient method, which requires no labeled data, for converting a pre-trained CLM into this architecture. (3) The introduction of the contextualized word inclusion task, through the C 2 GEN dataset. (4) Extensive testing of related baselines and our proposed method, via both automatic and human evaluation. This section briefly introduces the related work for constrained text generation. A detailed description of each method, their strengths and weaknesses, and how they are configured to form our baselines is available in Appendix D. Decoding strategies operate directly on the CLM's predicted vocabulary distribution at each time step, and are hence often model-agnostic. Dathathri et al. (2020) propose Plug-and-Play-Language-Models (PPLM), which adjust the distribution in accordance with the gradients of an external discriminator model. Pascual et al. (2021) introduce Keyword2Text, which steers the CLM to include target words by directly increasing their sampling probability, along with their GloVe (Pen-nington et al., 2014) neighbours. Training objectives can be set up to grant generative control, such as CTRL (Keskar et al., 2019), which incorporates control codes for textual genre. KG-BART (Liu et al., 2021) utilizes a common sense knowledge graph and fine-tunes BART (Lewis et al., 2020) towards word inclusion. GDC (Khalifa et al., 2021) fine-tunes towards arbitrary discriminator signals using Reinforcement Learning. POINTER (Zhang et al., 2020) tackles word inclusion with a non-autoregressive approach, injecting words around the target words until a sentence is formed. Tailor (Ross et al., 2021) fine-tunes a T5 (Raf-fel et al., 2020) for fine-grained semantically-controlled text generation, with a focus on perturbing text for data augmentation. Prompting acts within the framework of the CLM's pre-training task, as constraints are expressed through natural language. This approach was popularized by the GPT models (Radford et al., 2018b; Brown et al., 2020) and has been shown to work for many different types of constraints (Reif et al., 2021; Clive et al., 2021). There is no standardized evaluation methodology for open-ended text generation (Howcroft et al., 2020). The large number of possible good texts hinders the usage of automatic text-overlap metrics (Papineni et al., 2002; Lin, 2004). And many human evaluations are to vague to be properly reproducible (Belz et al., 2020). To remedy this, van der Lee et al. (2019) propose guidelines for human studies, and Gehrmann et al. (2021) argue that textual quality cannot be described through a single metric. Informed by these arguments, we report relevant metrics for various situations, without necessarily claiming one method to be superior in all aspects. We propose to steer a CLM's generative direction by introducing a separate encoder for prompt instructions, which we refer to as the prompt model . The prompt model interprets textual prompts and produces positional invariant key-values, which the CLM can attend to via the novel non-residual attention schema (Section 3.1). The positional invariance ensures that the instruction is equally applicable at any time step, and is achieved by an additional shift of its key-values (Section 3.2). To allow independent prompts at different time steps we compute two distinct streams of information for the CLM. We refer to these as the textual and non-residual streams. The textual stream ignores the prompt model completely, and is identical to the normal self-attention of the CLM. The non-residual stream is responsible for the prediction at each time step, and instead attends to both the previous steps of the textual stream, and key-values from the prompt model. This is depicted in Figure 1 and formalized in Equation 1. Concretely, at time-step n the textual stream self-attends to the current time step and the previous textual key-values KV i<nT . The nonresidual stream self-attends to the current time step, the previous textual key-values KV i<nT , and the prompt model's key-values KVP . Finally, the next step prediction P ( w n +1 ) is computed from the non-residual stream. Applying the prompt SP to every time step in the text SCLM = { w 1 , w 2 , ..., w n } thus results in: KVP = PromptModel ( SP ) KV nT = CLM ( w n | KV i<nT ) P ( w n +1 ) = CLM ( w n | KV p , KV i<nT ) (1) Non-residual key-values are hence never attended to by either streams from subsequent time steps. A prompt instruction at time step n can therefore only influence future decoding steps via the sampled token at time step n , and not through its key-values. This non-residual property of each prompt assures that the hidden state of the CLM does not deteriorate over time. Appendix C.1 further motivates this with an example. Intuitively, this ensures that the residual key-values are only affected by textual input, allowing the CLM to operate within the limits of its pretraining objective. Furthermore, this means that one can apply different prompts at different time steps, without them disrupting each other through the CLM's internal state. 2 Further intuition on non-residual attention is available in Appendix C. 3.2 Position Invariant Transformation Ideally, prompt instructions should be equally applicable at any time step in the generation process. However, the positional encoding system of Transformers makes this difficult, particularly absolute positional encodings (Vaswani et al., 2017). Overcoming this requires a significant amount of training of the prompt model (See Appendix C.2). To alleviate the computational burden, we propose an architectural add-on where positional invariance is achieved by an additional set of weights, trained after the prompt model is trained on single sentence data. This reduces the overall training time, and allows one to easily fine-tune the prompt model on tasks lacking context, and apply the positional invariant transformation afterwards. This is depicted as step 3 an 4 in Figure 2. The prompt model, being a CLM, uses causal self-attention to process text and generate L sets of key-values per time step, where L refers to the number of layers in the model. We refer to the L key-values at a time step i as kv i . Hence, when the prompt model computes a prompt of length n , it yields the sequence of key-values KV  P = { kv 1 , kv 2 , ..., kv n } . The positional invariant transformation, referred to as C , consists of one parameter for each of the CLMs key-value parameters. 3 The same transformation C is then applied by point-wise addition to the prompt model's output at all time steps, thus yielding the shifted key-values KVP = { kv 1 + C, kv 2 + C, ..., kv n + C } . Given a pre-trained CLM, we propose to train an accompanying prompt model via four distinct phases, 4 as demonstrated in Figure 2. This includes an initialization phase, two pre-training phases, and one optional fine-tuning phase. The weights of the CLM are never updated in any of the training phases. As popularized by Raffel et al. (2020), all training, independent of task, is formulated within the framework of teacher-forced causal language modeling (Williams and Zipser, 1989). The goal is to maximize the likelihood of generating text S given prompt P , in accordance with Equation 1. 4.1 Initialization Prior to any training, the prompt model is created by cloning the pre-trained CLM into a separate new model. The CLM and prompt models hence start with an identical set of weights. This results in an efficient starting point, since the CLM is trained to communicate with itself via self-attention, and thus also the CLM and prompt model. Pre-training is divided into two distinct phases, both relying on the text generation task of word inclusion with a target sentence length. In the first phase, the prompt model is trained to influence the CLM using only single sentence data, without any position invariant transformation. In the second phase only the position invariant transformation is learnt, by training on data with longer context. This is illustrated in Figure 3. For both phases, training data is generated by sampling [ A, B ] unique target words for each sentence S = { w 1 , w 2 , ..., w n } , and incorporating them and the sentence length n into prompt P . The second phase utilizes sequences of multiple sentences, where each sentence is given its own prompt. During this phase, each prompt is computed independently, and the CLM attends only to the relevant prompt for each sentence. Details regarding the corpus and sampling schema used in our experiments are available in Appendix A, and details regarding our randomized prompt template is available in Appendix A.3. Finally, one can optionally fine-tune the prompt model towards another task or dataset. This is done by temporarily removing the positional invariant transformation, and tuning only the prompt model. The positional invariant transformation is then re-inserted afterwards, shifting the now fine-tuned prompt model's key-values. This fine-tuning schema circumvents the problem that many NLP tasks and labeled datasets are formulated without any accompanying context. One can therefore utilize single sentence datasets, and still apply the prompt model at arbitrary time steps. CommonGen (Lin et al., 2020) is a dataset for the constrained text generation task of word inclusion. The objective of the task is to generate text that includes a given set of target words and adhering to common sense. Each sample includes 3-5 target words, taken from various image-caption datasets. The samples in CommonGen are however all formulated without any accompanying context. We argue that this task formulation is too narrow, and that it needlessly incentivizes researchers to focus on methods that do not support context. This is orthogonal to our belief that many application areas necessitates the consideration of surrounding context. Therefore, to complement CommonGen, we provide an extended test set where an additional context is provided for each set of target words. The task is therefore reformulated to both generate commonsensical text which includes the given words, and also have the generated text adhere to the given context. Each context is formulated as three sentences, created by human annotators from Mechanical Turk ( www.mturk.com ), as exemplified in Table 1. The annotators were tasked to create three sentences, so that a subsequent sentence would be likely to include the target words. Details regarding the creation process of C 2 GEN , and its statistical properties are available in Appendix F. Jane was excited when the teacher announced it was career week. Jane signed her dad up to visit the classroom. On the appointed day, Jane's dad showed up dressed in his work gear. We separate word inclusion into two different settings. In the first, the model is tasked to generate exactly 32 tokens. Requiring the model to both satisfy the word inclusion objective, and continue generating contextually relevant text. This allows methods that do not grant sentence level control to participate, such as PPLM and Keyword2Text. In the second setting, the model is only tasked to create a single sentence, elevating the requirement of continued text generation. This setting is suitable for methods specifically trained towards creating a single common sense sentence, such as KG-BART and POINTER. For both of these settings, we run experiments on both CommonGen and C 2 GEN . Since experiments on the contextualized C 2 GEN require the model to adhere to a context regardless of whether the objective is to generate a single sentence or a free text, KG-BART and POINTER are excluded from these experiments all together. Using our proposed method we train a nonresidual prompt model to accompany a pre-trained GPT-2 Large model. This setup is referred to as NRP during experiments, and training details can be found in Appendix A. In order to demonstrate how more sophisticated decoding strategies can be incorporated, we also combine NRP with a slightly modified version of Keyword2Text. Details for this incorporation can be found in Appendix B.3. The inference utilizes a beam size of 4, and any additional parameters were set according to a held-out validation set (See Appendix B). All baseline implementations are taken from their respective code repositories, and if possible the official pre-trained model (See Appendix D). In accordance with the guidelines described in Section 2.2, we provide both quantitative and qualitative evaluation. The qualitative examples in Table 4 are intended to convey the overall style for each algorithm, and more qualitative examples are available in Appendix G. Quantitative metrics are easily comparable, but may be less suited to convey the overall style. Our quantitative metrics are described in detail in Appendix E, and briefly below: Word Inclusion Coverage (Cov): The percentage of target words that are included in the generated text. Both target and generated words are lemmatized, alleviating the need to match the exact form of the target word. Perplexity (Ppl): The mean perplexity of the generated text calculated with GPT-2 XL. Although lower perplexity often indicates better language fluency, degenerate repetitions tend to result in low perplexity as well. Therefore, one should not rely on perplexity alone, but in combination with other metrics and qualitative analysis. Nevertheless, it is a metric that yields a hint of language fluency that does not require human evaluation. In the presence of contexts, as is the case with C 2 GEN , the perplexity is conditioned on the context and typically results in significantly lower perplexity values. Self-BLEU-5 (Self-Bleu): Average BLEU-5 overlap between all generated texts. A lower score is desired as this indicates syntactic diversity. Common Sense (Sense): The average score on how well the generated text adheres to common sense, according to human evaluators. Contextual Relevancy (Ctx): The average score on how well the generated text fits the given context, according to human evaluators. For more information about the human evaluation process, see Appendix E.3. We wish to highlight that NRP, Keyword2Text, and the prompted GPT-2 all control the same underlying CLM model. Differences between these approaches are hence a result of the method, not the model. Unfortunately, all quantitative metrics (including human metrics) are intrinsically correlated with sentence length, making comparisons of single sentences non-trivial (See Appendix E). First, we note that it is only the NRP approaches, and arguably GPT-2, that supports all four experiments. In general we find that the incorporation of Keyword2Text with NRP increases the coverage slightly, but at the cost of a slightly higher self-Bleu. Hence, for brevity, we refer to both of them as NRP throughout the remainder of this section. first sentence are ride , scooter , shirt , wear , and the target words for the second sentence are press , card , place , button , scanner . In the CommonGen Free Text setting (Table 2), NRP achieves the best coverage rate by a large margin, and also the best perplexity. Noticeably, NRP outperform GPT-2 in all metrics, besides common sense where they are virtually equal. Interestingly, GPT-2's coverage is virtually the same as its free text counterpart, indicating that it quickly forgets the intended instruction. Keyword2Text generates the lowest self-Bleu, but has both the worst perplexity and common sense score. PPLM performs the best on common sense but instead fails the task completely, as demonstrated by its poor coverage. In the CommonGen Single Sentence setting (Table 2), NRP fall slightly behind the specialized sentence methods in terms of coverage, but has a noticeably higher coverage than GPT-2. POINTER has the best coverage and self-Bleu, but also the worst common sense and dramatically worst perplexity. KG-BART has as expected the best common sense score, while staying fairly balanced on all other metrics. Again, NRP and GPT-2 show similar common sense scores. For C 2 GEN Free Text (Table 3), NRP performs the best on coverage and perplexity. All methods perform nearly identical on the context score. Both PPLM and Keyword2Text perform better than they did on CommonGen, but Keyword2Text is still worst on perplexity and common sense, and PPLM still performs the worst on coverage. As expected, GPT-2 performs poorly on contextualized word inclusion, demonstrated by its low coverage. This indicates that GPT-2 acts more as a regular CLM, ignoring the instruction prompt, which explains its high common sense score. Finally, NRP performs significantly better on coverage, perplexity, self-Bleu and context with Single Sentences on C 2 GEN (Table 3). GPT-2 performs better on common sense, which is likely due to it focusing less on the word inclusion objective. Again, GPT-2 achieves a similar coverage as its Free Text counterpart. As demonstrated in Table 4, NRP and GPT-2 tend to generate more linguistically complicated sentences, with more flow, compared to that of KG-BART. While stylistic complexity is arguably something desirable, it has the drawback that it increases the chance of generating text that breaks common sense. Our inspection also confirms that POINTER generates long sentences with weird formulations, that often break common sense and being syntactically incorrect. Examples of generated texts from all methods are available in Appendix G. Keyword2Text often inserts multiple line breaks, and sometimes gets stuck repeating a word. The differences between NRP, PPLM and GPT-2 are more subtle, the major distinction being that PPLM comes off as slightly more fluid in its formulations. The inclusion of sentence length in the pretraining objective (Section 4.2), gives an additional level of generative control over the linguistic style. As demonstrated in Table 5, the model incorporates and plans using the prompted sentence length, and changes the wording and content accordingly. We note that the model tends to prioritize textual quality over strictly sticking to the exact number of words. To measure this discrepancy, we generate sentences for all CommonGen validation samples for different prompted sentence lengths. Figure 4 shows the results from this experiment, displaying the expected offset for different prompted sentence lengths. The mean offset is always above 0 and below 1 , meaning the CLM can be expected to generate a slightly longer sentence than intended. The standard deviation increases both as the prompted length approaches long, and short sentences. This matches the sentence distribution of the pretraining dataset, as demonstrated in Table 6 found in Appendix A. 8 Discussion and Future Work We opted to demonstrate our architecture's capabilities on the task of word inclusion, since quantitative comparisons on this task are relatively straight-forward, compared to most other open-ended text generation tasks. While experimental results indicate the versatility of our approach, it is important to note that the method conceptually generalizes to a much wider range of tasks. Our non-residual architecture enables the use of prompt instructions at arbitrary time steps, but is not limited to word inclusion. We hence encourage future work to pursue the incorporation of multi-task prompt learning, as being able to apply flexible prompts with precision would be a big step forward in the many areas striving to use CLMs. Indeed, we consider the ability to control the text generation process while considering context crucial for any tool intended for human editors. Admittedly, our training method for realizing our encoder-decoder architecture has largely been dictated by a lack of resources. We conceptually prefer the more straight-forward training approach of training the prompt model directly on long context data, and removing the positional invariant transformation. Future work could thus increase computational resources and investigate the possibility of different positional encoding schemes. Finally, we stress that nothing in our approach has focused explicitly on common sense. It is hence expected that methods that do, like KG-BART, perform better on this metric. Future work could thus investigate the use of a prompt model to control a CLM fine-tuned towards common sense, or fine-tune a prompt model using common sense data. Results on CommonGen and C 2 GEN dataset still leave ample room for improvements. This paper has introduced the concept of nonresidual attention and demonstrated how it can be used to control a generative text model. Additionally, our work pinpoints the lack of open-ended controllable text generation tasks that require the model to also account for a given context. We set out to remedy this by introducing the humanly created C 2 GEN dataset, introducing the task of contextualized word inclusion. Experimental results on C 2 GEN and CommonGen, clearly demonstrates that using a nonresidual prompt model increases generative control over a CLM. Compared to other methods, our approach stands out as the most versatile, consistently performing well across all tested situations. This work is partly funded by the Swedish innovation agency (Vinnova) under contract 2019-02996. Additionally, we thank the Grace & Thomas C. H. Chan Cambridge Scholarship, which supports Fangyu Liu. Finally, a special thanks goes out to Martin Korling at RISE, who helped attain computational resources for the final submission deadline. Controllable text generation is an important step to unleashing the potential of modern CLMs. Additionally, it is an interesting approach to counter many of the problematic biases that have been found. But an increased level of control also entails an increased risk of malicious use. We hence recognize the possibility that techniques proposed in this paper could be utilized in malevolent scenarios, like guided misinformation or targeted harmful content. This work has utilized computational GPU resources provided by ICE-RISE 5 . The final model training lasted roughly 2 days on a single DGX-100 machine, resulting in about 400 GPU hours. The total number of GPU hours for the whole research endeavour is difficult to estimate, but it can be safe to assume that it is less than 2000 GPU hours.
abstain abstain abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain result objective abstain abstain objective other method other other other other other other other other other other other other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain method other other other abstain abstain abstain abstain method abstain abstain