Long-range semantic coherence remains a challenge in automatic language generation and understanding. We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction. We present coherence boosting , an inference procedure that increases a LM's focus on a long context. We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses. It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training. Language models (LMs) are commonly evaluated for their ability to generate, rank, or classify coherent spans of text. Long-range semantic coherence is a unifying feature of modern NLP benchmarks and applications, whether they are about producing short answers to questions, ranking answer choices by their consistency with world knowledge, or generating long responses. Large nonspecialized LMs, such as GPT-2 and -3 (Radford et al., 2019; Brown et al., 2020), sometimes fail to understand or use the semantic link between a text and its prompt or long-range context (Fig. 1). Samples from these LMs have an unnaturally low density of words that require many tokens of context to predict (4.1), and the scores that the models give to completions of prompts indicate that they are oversensitive to recent context (5). We hypothesize that these failures arise from modeling choices and distribution shift. Specifically, autoregressive LMs are typically fit to a multi-objective problem: simultaneously maximizing token likelihoods conditioned on many lengths of truncated context (2.1). Yet, at generation or Code: github.com/zhenwang9102/coherence-boosting. scoring time, likelihoods are conditioned on the entire prompt or previously generated string, specifically selected to be coherent or even guaranteed to influence the output. The two common solutions  finetuning models on one or multiple tasks (Khashabi et al., 2020; Sanh et al., 2022) and improving models or prompts to facilitate in-context learning (Brown et al., 2020; Schick and Schtze, 2021)  do not directly target the problem of long-range coherence. This paper proposes coherence boosting , a simple inference-time procedure that increases the effect of distant words on predicted token distributions and is applicable in both generation and ranking settings. A pretrained model is viewed as an ensemble of experts that produce token distributions conditioned on varying lengths of context. These experts are log-linearly mixed to form a predictor that is superior to the base model (2). Coherence boosting greatly improves prediction of words that depend on a long context, as evidenced by state-of-the-art results on tasks specially meant to assess models' attention to distant words (3). In generation of generic text and dialog responses, we show that coherence boosting brings the frequency of occurrence of such words close to that seen in natural text (4). Beyond generation, we study diverse multiple-choice tasks (5), in which examples are known to be highly coherent. Coherence boosting does not modify the base model and depends on a single parameter than can be estimated in one pass through a validation set, yet is a competitive adaptation algorithm. Balance between satisfaction of short-range statistical constraints and maintenance of long-range structure was a central question of language generation long before neural language modeling. To compensate for the sparsity of the learning signal for long-range influences,  -gram models and 8214 A: I'm Natasha. Neural language modeling brought a need for recurrent units with better numerical properties for propagating information over long distances (Hochreiter and Schmidhuber, 1997; Cho et al., 2014) and eventually saw the reintroduction of alignment variables (Brown et al., 1993) into generation in the form of attention (Bah-danau et al., 2015; Vaswani et al., 2017). Attention is at the core of Transformer LMs, including GPT. Language models are being trained on and adapted to ever-longer input sequences (Beltagy et al., 2020; Zaheer et al., 2020; Roy et al., 2021; Press et al., 2022), but they remain undersensi-tive to distant content or syntax (Khandelwal et al., 2018; Sun et al., 2021) and are easily fooled by recency bias in few-shot prompts (Zhao et al., 2021) or multi-turn conversations (Sankar et al., 2019). Recent work has continued to study inference-time procedures that prevent text sampled from LMs from degenerating into nonsense. Most of these procedures, such as tempered sampling and top /top truncation (Fan et al., 2018; Holtzman et al., 2019), independently modify the output distribution at each generation step to decrease its entropy and diminish its low-likelihood tail. Holtzman et al. (2019) and Meister and Cotterell (2021) found that such local modifications increase the quality of long generated sequences; we adopt and extend their methodology in 4.1. For dialog systems, Li et al. (2016) propose a decoding scheme that maximizes a mutual information criterion, which explicitly optimizes for dependence of generated text on prompts  a special case of coherence boosting. In multiple-choice tasks, where a model must choose one of several given completions of a prompt, Brown et al. (2020) observe that selecting the completion that maximizes the conditional likelihood of the completion following the prompt often favors completions having high unconditional likelihood (likelihood following an empty or dummy prompt) and, for some tasks, chooses to divide the scores of candidate answers by their unconditional likelihoods. This is also a special case of coherence boosting. Such scoring modifications are more thoroughly studied by Zhao et al. (2021); Holtzman et al. (2021). The latter attributes the problem to sur-face form competition': there are many variants of the correct completion that together may capture a 8215 large part of probability mass, but the form of the given answer choice alone is not the most likely. However, we show that other causes are at play: surface form competition is impossible when the completion is known to be a single token and the range of choices is the whole vocabulary (3), and it is not applicable to open-ended generation (4). In this section,  is an autoregressive LM over a vocabulary  with learnable parameters  , taking as input a variable number of tokens (up to a maximum context length  ) and producing a vector of next-token likelihoods: where  (  ) is the probability simplex over  . We will write the  -th component of this output vector as a conditional likelihood,  (  |  1 , . . . ,   ;  ) . We denote by   the model evaluated on only the last  input tokens, ignoring earlier tokens:   (  1 , . . . ,   ;  ) : =  (     + 1 , . . . ,   ;  ) . Coherence boosting for next-token prediction. Coherence boosting for a model  selects real-valued weights  = (  1 ,  2 , . . . ,   ) and produces a new language model   , defined by   (  1 , . . . ,   ;  ) : = softmax (cid:32)  (cid:213)  = 1   log   (  1 , . . . ,   ;  ) (cid:33) , (1) where log is taken element-wise, or, equivalently,   (  |  1 , . . . ,   ;  )   (cid:214)  = 1   (  |  1 , . . . ,   ;  )   . This is a weighted product-of-experts model, where the experts' are copies of the base model  evaluated on different context lengths. Because evaluating  is expensive, we use sparse weights  , as the expression (1) depends only on those   for which    0 . In Fig. 1 and in the experiments, we allow  to have only two nonzero entries: when computing likelihoods of words following a sequence of length  , we consider weighted products of  max : =   (the full context) and an   with    (a short context, either of fixed length or decided by prompt structure as in 4.2). boosting for multiclass classification (Friedman et al., 2000). However, our weak classifiers are pretrained and share all of their parameters, not obtained by an iterative procedure of training on reweighted data, and we permit negative weights. 1 Coherence boosting for answer selection. In multiple-choice problems, a LM must choose the best answer following a context, which consists of a premise or passage followed by a shorter premise-free context (either a short phrase, such as An-swer:, that incites the LM to generate an answer in the right format, or a hypothesis that depends on the premise). The full context is the concatenation of the premise and the premise-free context (E). By the autoregressive factorization, the model  assigns conditional likelihoods to sequences of tokens following context. A typical model for answer selection ranks the candidate answers   (se-quences of tokens) by  (   | full context ;  ) and outputs the highest-ranked   . Coherence boosting chooses a parameter  and ranks the choices by: log  (   | full context ;  ) + +  log  (   | premise-free context ;  ) . This is a log-linear combination of two models:  evaluated with full context and with a partial context. When  = 0 , ranking by (2) is equivalent to ranking by the base model. When  =  1 , it is equivalent to dividing the base model's score by the score of each answer conditioned on the prompt (short context), and thus to maximizing pointwise mutual information between the premise and the answer conditional on the premise-free context. Unlike Brown et al. (2020); Holtzman et al. (2021), our formulation allows the premise-free context to include information specific to the example, not only a domain-specific dummy prompt. We expect coherence boosting to correct for an oversensitivity to the premise-free context, and thus the optimal  will typically be negative (see 5). 1 As for the first half of the term coherence boosting', Howcroft et al. (2020); Belz et al. (2020) found that very incoherent definitions of the word coherence' abound in the natural language evaluation literature. The reader is asked to forgive us for the loose definition of long-range semantic coherence' adopted in this paper. the predictors   , which share parameters  . Each training iteration samples a sequence (or batch of sequences) of a chosen maximum length  + 1 from the data distribution D and minimizes the average negative log-likelihood (NLL) of all words following the parts of the sequence that precede them: the optimization criterion is: E  1 ...  + 1 D 1   (cid:213)  = 1  log  (   + 1 |  1 , . . . ,   ;  ) . If D is uniform over all length-(  + 1 ) subsequences of a training corpus, any given word is equally to likely to appear in all positions within a sampled sequence 2 , and the criterion is equal to  (cid:213)  = 1 1  E [ log   (   + 1 |  1 , . . . ,   ;  )] (cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:125) L  (  ) , (3) This is a uniform scalarization of an  -task problem: the  -th objective L  (  ) is the expected NLL of a word in the corpus following  context words. This situation is different from that seen at generation time. If the text generated so far is  1  2 . . .   , the distribution from which the next word   + 1 is sampled is   (  1 , . . . ,   ;  )  only the ensemble member using full context is used. However, if the string  1 . . .     + 1 had been seen in training,  would have been trained to predict   + 1 given all partial contexts , with equal weight given to all prediction losses. Thus,  is trained to make predictions on data it never sees in evaluation, and may be prevented from optimally learning to use long context: parameters that locally optimize (3) are locally Pareto-optimal for the set of prediction losses L 1 , . . . , L  , but not necessarily optimal for any individual L  . An ensemble of the   (    ) may be a better predictor than   alone. (See A for further analysis of when this occurs.) Undertraining. The parameters  are shared by the predictors   , and modeling power must be spread among the losses L  (  ) . The short-context predictors are easier to fit, while sequences in which long context affects the prediction are rare. We expect sensitivity to long context, and precision in modeling its effect, to be especially diminished if the model is undertrained. 2 Many authors leave unspecified the way in which training batches are formed from a corpus of input documents. Here we assume that all training documents are concatenated into one (very long) document separated by end-of-text tokens and ignore minute effects near the start and end of this document. Distribution shift. While the training procedure causes a bias against the influence of longer contexts on generation, we see the opposite bias in downstream tasks (question answering, natural language inference, adversarial probes for common sense): Many modern NLP benchmarks try to challenge models to use long context (3, 5). The LAMBADA dataset (Paperno et al., 2016) tests LMs' understanding of long-range dependencies by measuring the prediction of the final words in passages of several sentences. The task explicitly requires reasoning over a broad context: humans can reliably guess the last word when given a whole passage, but not when given only the last sentence. We perform experiments with the GPT family of models, closely replicating the evaluation setting of Radford et al. (2019). 3 We predict the final word as the top-ranked token under the boosted model  max     , where  max is the model taking the full available context and ,   are the chosen length and coefficient of the short context. To choose  and   , we do a grid search on the validation set and apply the best values to the testing set. Results. Table 1 shows the accuracies and optimal parameter values   ,    . Coherence boosting vastly reduces prediction error for all models. In particular, the boosted GPT-2 Small performs better than the original GPT-3 2.7B. The boosted GPT-3 175B achieves a new state of the art. 3 Certain details are omitted by Radford et al. (2019). Based on https://github.com/openai/gpt-2/ issues/131 , we nearly match baseline accuracy by predicting the last subword token, rather than the last word. Other than the impressive performance gain, we highlight two observations. (1) The optimal   is always negative, indicating that the optimal mixture of models penalizes the influence of short-range context relative to long-range context. (2) With increasing model size, the optimal   and  become closer to 0. This means that bigger models capture long-range coherence better than small models, as they have less need to penalize the effect of short context. (Fig. 2 shows the accuracy curves for all models by sweeping   with a fixed  . The peak clearly moves to the left as model size grows.) 4 Experiments: Language generation 4.1 Generic text The experiment in this section extends that of Holtzman et al. (2019). A selection of 5000 articles from WebText (Radford et al., 2019) is taken as a reference corpus of human-written text. A language model (for us, GPT-2 Large) is prompted to generate text conditioned only on the first sentence of each of these articles, up to a maximum of 200 tokens, yielding 5000 machine-generated texts. The human-written and machine-generated texts are compared by four automatic metrics: perplexity under the base LM, self-BLEU-4 (Zhu et al. (2018); the mean BLEU-4 score of a generated text with respect to all other generated texts as references), Zipf coefficient (the linear regression coefficient between log-rank and log-frequency of generated tokens) and repetition (the fraction of generated texts that end in a repeating sequence of tokens). It is desirable for a model and inference procedure to produce text that is as close as possible in these metrics to the human-written reference. To measure long-range semantic coherence in the generated text, we define three new metrics: Long-range repetition (LR  ): For a whole number  and document  , let  (  ) be the number of distinct tokens in  , and let   (  ) be the number of distinct tokens for which the distance between their first and last occurrence in  is at least  positions. The long-range repetition score LR  of a corpus {  1 , . . . ,  5000 } is a macro-average: LR  : = (cid:205) 5000  = 1   (   ) (cid:205) 5000  = 1  (   ) . This simple measure of lexical coherence favors repetition of words long after they are first used, but gives lower weight to documents that degenerate into repetition of a short span. Long-dependent token frequency (LTF): A long-dependent token is one to which the base LM assigns a likelihood of at least 20% given its full context, but a likelihood of less than 5% given only the 20 tokens of context preceding it. We compute the frequency of long-dependent tokens among all generated tokens. Long-short likelihood difference (  ): The mean difference in likelihoods assigned to tokens by the base LM conditioned on full context and conditioned on 20 tokens of context. Although some choices of constants are needed to define LTF and  , we intend them to be intuitive summaries of long-range coherence in the absence of established metrics. In particular, 20 tokens is close to the length of one sentence in typical English text. We sample 5000 document completions from GPT-2 Large following sampling procedures with a range of boosting schemes. We consider models of the form      1    max , for   { 8 , 16 , 32 , 64 } and    { 0 . 4 ,  0 . 2 ,  0 . 1 ,  0 . 05 ,  0 . 025 , 0 } . (Such a parametrization of boosting parameters was chosen to ensure that when the context has length less than   or the distant context has very little effect on the next word  the boosted model becomes equivalent to the untempered  max .) Top truncation with  = 0 . 95 is applied to all models. coherence (4.1). (Nearest-to-human values in bold , boosting models better than top sampling alone in italics .) Results. Metrics of two of the best models, with  = 32 ,   =  0 . 05 and  = 64 ,   =  0 . 1 , are shown in Table 2. In particular, the latter model generates text that is closer to the human reference, or equally close, to the pure top sampling (   = 0 ) baseline in all metrics, with the greatest improvement seen in the coherence measures. Fig. 3 shows the dependence of selected metrics on  and   . Coherence boosting brings all metrics closer to those of human text. As  increases, the optimal   grows in magnitude. This is expected: the predictive effect of tokens more than  positions away decreases with  (   approaches  max ). We also note that a simple sampling with temperature 0.9 performs better than top sampling in most of the coherence metrics. This suggests that the improvements accomplished by top truncation come at the cost of introducing a bias towards tokens that are predictable from a short context. Coherence boosting corrects this bias without sac-rificing the gains in other measures. An example of human, top , and coherence boosting outputs is shown in Table D.1. This experiment is based on the Dialog System Technology Challenge 7 (DSTC7) (Galley et al., 2019), which benchmarks generation of dialog responses responses conditioned on one or more turns of conversation context. As a base model, we use DialoGPT (Zhang et al., 2020c), a GPT-2 Small variant that demonstrated strong results on this task. Dialog systems' responses to the 2208 conversation prompts 4 are scored against human-written reference responses (five for each example). Following Zhang et al. (2020c), we use the  -gram overlap metrics NIST (Doddington, 2002), BLEU (Papineni et al., 2002), and METEOR (Lavie and Agarwal, 2007), as well as two intrinsic measures of  -gram diversity from Li et al. (2016); Zhang et al. (2018): Distinct and Entropy . It is desirable for a dialog system to reach scores close to those of the human responses in all metrics. In addition to the decoding algorithms considered by (Zhang et al., 2020c)  beam search and greedy decoding  we consider greedy decoding with a coherence boosting model. As long and short predictors, we use DialoGPT conditioned on the full conversation context and on only the (context-free) response generated so far . That is, if the conversation context is  and the text generated so far is  1 . . .   , then   + 1 is predicted using the model  max   + 1 , evaluated on the string  (cid:104) sep (cid:105)  1 . . .   , where (cid:104) sep (cid:105) is the turn separa-4 The DSTC7 evaluation data, scraped from Reddit, is undisclosed; we reacquire it using officially released code. tor token. We consider   { 0 ,  0 . 1 , . . . ,  0 . 8 } . Results. Table 3 shows the metrics of the boosting models that reach the peak average NIST and BLEU scores (  =  0 . 3 and  =  0 . 7 ). Increasing the magnitude of  leads to responses that are more relevant to the prompt (higher BLEU and NIST) and more diverse than those from greedy decoding. As   grows large, the boosting model favors creative responses that are relevant to the prompt (high NIST), but simple responses that are common in the reference data become unlikely (low BLEU). 5 We observed that the responses with  =  0 . 7 , despite the superior metrics, are more likely to be ungrammatical and innovate words in an effort to use tokens relevant to the prompt. In practice, improving dialog systems with coherence boosting may require techniques to prevent these side effects, such as repetition penalties or relaxation of greedy decoding to low-temperature sampling. Finally, we note that the learning of DialoGPT was initialized with a pretrained GPT-2 and uses GPT-2's end-of-text token as the turn separator. This choice may reduce DialoGPT's attention to past turns, as tokens preceding the end-of-text token are never informative in GPT-2's training data. We evaluate coherence boosting on zero-shot language understanding and inference tasks, where examples are expected to be highly coherent. We study 15 datasets in 5 categories of tasks. (1) Cloze tasks : StoryCloze (Mostafazadeh et al., 2016), HellaSwag (Zellers et al., 2019), and COPA (Roemmele et al., 2011). (2) Question answering : CommonsenseQA (CsQA) (Talmor et al., 2019), OpenBookQA (OBQA) (Mihaylov et al., 5 Galley et al. (2019) argue that NIST and diversity metrics are more informative measures than BLEU for multi-reference scoring, since BLEU favors systems that often produce responses with little relation to the prompt (e.g., I don't know). 2018), ARC Easy / Challenge (ARC-E/C) (Clark et al., 2018), and PIQA (Bisk et al., 2020). (3) Text classification : SST-2/5 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), AGNews (Zhang et al., 2015). (4) Natural language inference : RTE (Dagan et al., 2005), CB (De Marneffe et al., 2019), and BoolQ (Clark et al., 2019). (5) Fact knowledge retrieval : LAMA (Petroni et al., 2019). All tasks except LAMA are formulated as multiple-choice problems. We convert text classification and inference tasks to multiple-choice tasks by choosing meaningful answer words, e.g., True/False. The prediction is made by selecting the choice with the highest LM likelihood. For in-context learning of GPT models, prompt formats greatly impact performance. We follow previous work (Brown et al., 2020; Zhao et al., 2021; Holtzman et al., 2021) to create natural prompts to enlarge the effectiveness of in-context learning, but we do not aim to optimize the full and context-free prompt format: our goal is to evaluate coherence boosting models with a fixed prompt. The prompt formats we use are listed in Table E.1. As described in 2, within each prompt we identify a premise-free context , which is used as the context for the short-range model in coherence boosting. For each dataset, we pick the optimal value   of the parameter  on the validation set and report the accuracy on testing set. (If no testing set is publicly available, we choose  on a subset of the training set and report the final number on the validation set.) Across all experiments, we do not put any few-shot examples in the prompt. For the knowledge retrieval task, we follow Zhao et al. (2021)'s data split of LAMA and evaluate GPT models on facts whose missing answers are at the end of the sentence (to fit the nature of autoregressive language models). We limit the prompt length to be larger than 5 tokens and rerun the model from Zhao et al. (2021) on the new data. Results: Multiple-choice tasks. Results of three representative base models on all multiple-choice tasks are presented in Table 4. (Results for all models are in Tables F.1 and F.2.) We compare our best model with two baselines,  = 0 (  max ) and  =  1 . The former one is the original full-context model, while the latter is, for most tasks, a form of unconditional probability normalization as performed by Brown et al. (2020); Holtzman et al. (2021). We also compare our best model with other inference methods (Holtzman et al., 2021; Min et al., 2021) in Tables F.3 and F.4. By comparing the third column with the first two columns within each model in Table 4, we can see that our method with the selected  generally improves the accuracy on all tasks. Some of the improvements are dramatic, where boosted GPT-2 Small outperforms GPT-2 XL's base model (e.g., CsQA, OBQA, ARC-C) and is even comparable with GPT-3 175B's base model (e.g., SST-2, SST-5, RTE). We make similar conclusions when comparing coherence boosting with other inference methods in Tables F.3 and F.4. We observe that the optimal  depends on tasks and models (fourth column within each model), which means that  cannot be heuristically set to 0 or  1 as in past work. This finding suggests the necessity of searching for an optimal  . We visualize the accuracy curve by varying  in the testing set of all datasets. We show the curve for StoryCloze in Fig. 4 and present similar figures for all tasks in Figs. F.1 and F.2. Consistent with the results on LAMBADA (3), the optimal  is usually negative, and its absolute value tends to decrease with the model size. We selected the optimal  by the validation set, but future work may explore automatic and adaptive methods for setting this parameter. Notice that all experiments required only a single pass through the data to compute answer likelihoods conditioned 8221 GPT-2 GPT-3 125M 350M 760M 1.6B 2.7B 6.7B 13B 175B  max 8.48 14.78 13.88 14.29 17.33 19.42 22.06 26.76 Zhao et al. (2021) 17.45 22.87 23.90 23.97 26.30 30.57 31.96 34.78 CB (   =    ) 19.85 22.87 25.74 25.43 28.75 32.25 35.02 37.57     0.5  0.5  0.5  0.5  0.5  0.5  0.5  0.4   1 2 3 3 1 1 1 2 Table 5: Accuracies (%) of GPT models on LAMA. Results: Knowledge retrieval. Unlike LAMBADA, where long contexts are required for inferring the last word, LAMA contains much shorter sentences for knowledge facts, i.e., (subject, relation, object). A recent study (Cao et al., 2021) shows that the prediction is biased by the relation in the short context, i.e., the answer to a prompt (e.g., Dante was born in ___) can be induced by the relation (was born in) without the subject. Coherence boosting mitigates the influence of those short contexts by making the prediction dependent on a longer context containing the subject. We present results for all models on LAMA in Table 5. We also compare our model with contextual calibration (CC) (Zhao et al., 2021), which processes the LM's output probabilities with a log-linear model. 6 Coherence boosting with the selected  and  outperforms both the base model and CC by significant margins. We suggest three promising research directions: Coherence tuning. The need to evaluate the base LM with multiple contexts in coherence boosting introduces cost and complexity at inference time. It may be desirable instead to modify the weights of the base model to improve long-range coherence properties. In B, we describe a self-tuning' algorithm that achieves this without training on any data created for this purpose . text, but future work should consider other architectures and target domains. In C, we give preliminary results on the text summarization domain. Although we expect recency bias to be less pronounced in LMs that use separate attention modules to process the prompt and the output  such as encoder-decoder models for translation or summarization  procedures inspired by coherence boosting may prove effective in domains where a strong causal link between prompt and output is known to exist. Such domains include language generation conditioned on structured data (Yao et al., 2020; Mager et al., 2020; Moosavi et al., 2021) and model-guided reasoning in formal languages, such as proof or program synthesis (Polu and Sutskever, 2020; Chen et al., 2021; Li et al., 2022). Efficient search proposals. Procedures that force LMs to be more focused on a prompt, or a specific part of it, when generating or ranking tokens can benefit algorithms that search for combinations of words through sampling. It would be interesting to use coherence boosting in non-autoregressive text generation algorithms, such as to accelerate the mixing of MCMC methods for constrained text generation (Miao et al., 2019; Zhang et al., 2020b; Malkin et al., 2021). We have illustrated the hyposensitivity of pretrained language models to long-range context and proposed a simple inference-time remedy. We hope to see coherence boosting used as a simple alternative or complement to finetuning procedures in zero-shot applications of pretrained LMs. The authors are grateful to Sudha Rao, Matt Richardson, and Huan Sun for valuable discussions about this project. We thank the anonymous reviewers for their comments and suggestions. We hope and expect to see a nonnegative net societal impact from better text generation and ranking algorithms in general and from this work in particular. As we have shown, there is room to improve the inference procedures used with small language models, which incur lower costs than training and evaluation of large models. However, researchers should bear in mind the risks and potential misuse of automatic generation of long-form text.
abstain objective method result abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective abstain abstain abstain result method abstain other other other other other other other abstain other other other other other result abstain abstain abstain method abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective result other other abstain abstain abstain