summaries: This paper proposes to model the evolution of sentences in a document via a stochastic process; specifically a Brownian Bridge process. The paper start off by assuming that the generated sequences by autoregressive models like GPT-2 follow Brownian motion in that they tend to get incoherent and "meander" in the semantic space. This paper aims to reduce this random behavior by pinning the endpoints of the trajectory and model the generation by Brownian bridge process instead. The key intuition behind this process is that given two endpoints z0 and zT, the evolution of z along time t is a Gaussian with mean that is some linear combination of z0 and zT. This paper models text by training an encoder for sentences x that produces the embedding z by training over triplets (x0, xt, xT) where 0<t<T that encourages zt to follow Brownian bridge dynamics and uses contrastive loss with a negatively sampled x't for training.  The approach is tested for local coherence, long range order sensitivity, and generation of long sequences and is compared against ablative and external baselines. The proposed approach does lead to learning of embeddings that are obtainable via linear combination and this leads to improved performance on sensitivity to sentence order in documents and document generation.  The authors propose to use a Brownian bridge process to model global coherence of a long piece of text. They show how to train such a model in an encoder-decoder style setup, using a contrastive loss to model the Brownian bridge dynamics. The authors then verify aspects of their model with a series of experiments to show that their model with an underlying generative process outperforms competing approaches on a variety of local and global coherence and generation tasks. This paper introduces a method to enhance the global coherence of text generated from Language models. The proposed method (Time Control). Under the assumption in the latent space of sentence embeddings, the incoherent text can be seen as "Brownian motion" in the latent space. In order to enforce a goal to the generated text authors by fixing a start and end to this Brownian motion the process of text generation can be modeled as a Brownian bridge.   From this assumption, the authors drive a method that consists of three steps (1) training an encoder to map sentences to a latent plan defined as Brownian bridge (2) training a decoder to reconstruct sentences from the given context + the true encoded vector of the target sentence from planning latent space using the trained encoder (3) at inference time: given a start and endpoint, a target trajectory of vectors $z_0, ..., z_t, ..., z_T$ is sampled and use the decoder to generate a sentence based on this bridge.   Authors run several experiments to (1) evaluate the hypothesis that the encoder can capture local text dynamics using sentence order prediction task (2) evaluate the decoder to generate local incoherent text using the text-infilling task. (3) capture global text statistics by measuring the statistics (length of Wikipedia sections for city articles) of the generated text and compare them to the ground truth. (4) Evaluate the overall coherence of the long-generated text. Overall the results look convincing except for some caveats (see the areas of enhancement)   This paper proposes a generation from a language model not only from an initial state, but also using a goal state. Instead of Brownian motion, the authors employ a draw from Brownian bridge by designating initial and end states, called Time Control. Experimental results show the proposed generation from Brownian bridge is more natural and coherent for text-infilling task, and also preserves text  structures both by automatic evaluation and human evaluation.
abstractText: Modern language models have been remarkably successful across a range of text generation tasks. Despite such successes, generated texts from these language models often contain structural flaws such as meandering and incoherent content, which arise from the fact that language models generate text myopically, one token at a time. To address these issues, we introduce Time Control (TC), a language model that implicitly plans via a latent stochastic process. TC does this by learning a representation which maps the dynamics of how text changes in a document to the dynamics of a stochastic process of interest. Using this representation, the language model can generate text by first implicitly generating a document plan via a stochastic process, and then generating text that is consistent with this latent plan. Compared to domain-specific methods and fine-tuning GPT2 across a variety of text domains, TC improves performance on text infilling and discourse coherence. On long text generation settings, TC preserves the text structure both in terms of ordering (up to +40% better) and text length consistency (up to +17% better). Human evaluators also prefer TC’s output 28.6% more.
references.authors: Sebastian Arnold Rudolf Schneider Philippe Cudré-Mauroux Felix A. Gers Alexander Löser
references.title: Sector: A neural model for coherent topic segmentation and classification
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Sanjeev Arora Yuanzhi Li Yingyu Liang Tengyu Ma Andrej Risteski
references.title: A latent variable model approach to pmi-based word embeddings
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Michał Bień Michał Gilski Martyna Maciejewska Wojciech Taisner Dawid Wisniewski Agnieszka Lawrynowicz
references.title: RecipeNLG: A cooking recipes dataset for semi-structured text generation
references.venue: In Proceedings of the 13th International Conference on Natural Language Generation,
references.authors: Samuel Bowman Luke Vilnis Oriol Vinyals Andrew Dai Rafal Jozefowicz Samy Bengio
references.title: Generating sentences from a continuous space
references.venue: In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning,
references.authors: Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell
references.title: Language models are few-shot learners
references.authors: Bill Byrne Karthik Krishnamoorthi Chinnadhurai Sankar Arvind Neelakantan Ben Goodrich Daniel Duckworth Semih Yavuz Amit Dubey Kyu-Young Kim Andy Cedilnik
references.title: Taskmaster1: Toward a realistic and diverse dialog dataset
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Bill Byrne Karthik Krishnamoorthi Saravanan Ganesh Mihir Kale
references.title: TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems
references.venue: In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
references.authors: Mingda Chen Zewei Chu Kevin Gimpel
references.title: Evaluation benchmarks and learning criteria for discourse-aware sentence representations
references.venue: In Proc. of EMNLP,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
references.venue: [cs],
references.authors: Chris Donahue Mina Lee Percy Liang
references.title: Enabling language models to fill in the blanks
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Pablo A. Duboue Kathleen R. McKeown
references.title: Empirically estimating order constraints for content planning in generation
references.venue: In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics,
references.authors: Angela Fan Mike Lewis Yann Dauphin
references.title: Strategies for Structuring Story Generation
references.venue: [cs],
references.authors: Tianyu Gao Xingcheng Yao Danqi Chen
references.title: Simcse: Simple contrastive learning of sentence embeddings
references.venue: arXiv preprint arXiv:2104.08821,
references.authors: Xinyu Hua Lu Wang
references.title: Pair: Planning and iterative refinement in pre-trained transformers for long text generation
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Aapo Hyvarinen Hiroshi Morioka
references.title: Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. arXiv:1605.06336 [cs, stat], May 2016
references.venue: URL http://arxiv. org/abs/1605.06336
references.authors: Aapo Hyvarinen Hiroaki Sasaki Richard Turner
references.title: Nonlinear ica using auxiliary variables and generalized contrastive learning
references.venue: In The 22nd International Conference on Artificial Intelligence and Statistics,
references.authors: Dan Iter Kelvin Guu Larry Lansing Dan Jurafsky
references.title: Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models. arXiv:2005.10389 [cs], May 2020
references.venue: URL http://arxiv.org/abs/2005.10389
references.authors: Dan Jurafsky
references.title: Speech & language processing
references.venue: Pearson Education India,
references.authors: Chloé Kiddon Luke Zettlemoyer Yejin Choi
references.title: Globally coherent text generation with neural checklist models
references.venue: In Proceedings of the 2016 conference on empirical methods in natural language processing,
references.authors: Mahnaz Koupaee William Yang Wang
references.title: Wikihow: A large scale text summarization dataset
references.venue: arXiv preprint arXiv:1810.09305,
references.authors: Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut
references.title: Albert: A lite bert for self-supervised learning of language representations
references.authors: Chin-Yew Lin
references.title: ROUGE: A package for automatic evaluation of summaries
references.venue: In Text Summarization Branches Out,
references.authors: Chu-Cheng Lin Aaron Jaech Xin Li Matthew R. Gormley Jason Eisner
references.title: Limitations of autoregressive models and their alternatives
references.venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5147–5173,
references.authors: Bingbin Liu Pradeep Ravikumar Andrej Risteski
references.title: Contrastive learning of strong-mixing continuous-time stochastic processes. arXiv:2103.02740 [cs, stat], March 2021
references.venue: URL http: //arxiv.org/abs/2103.02740
references.authors: Yixin Liu Pengfei Liu
references.title: SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
references.venue: In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers),
references.authors: Amit Moryossef Yoav Goldberg Ido Dagan
references.title: Step-by-step: Separating planning from realization in neural data-to-text generation
references.venue: In Proceedings of the
references.authors: Nasrin Mostafazadeh Nathanael Chambers Xiaodong He Devi Parikh Dhruv Batra Lucy Vanderwende Pushmeet Kohli James Allen
references.title: A corpus and cloze evaluation for deeper understanding of commonsense stories
references.venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Allen Nie Erin Bennett Noah Goodman
references.title: DisSent: Learning Sentence Representations from Explicit Discourse Relations
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Kishore Papineni Salim Roukos Todd Ward Wei-Jing Zhu
references.title: Bleu: a method for automatic evaluation of machine translation
references.venue: In Proceedings of the 40th annual meeting of the Association for Computational Linguistics,
references.authors: Ben Poole Sherjil Ozair Aaron Van Den Oord Alex Alemi George Tucker
references.title: On variational bounds of mutual information
references.venue: In International Conference on Machine Learning,
references.authors: Ratish Puduppully Li Dong Mirella Lapata
references.title: Data-to-text generation with content selection and planning
references.venue: In Proceedings of the AAAI conference on artificial intelligence,
references.authors: Alec Radford Jeff Wu Rewon Child David Luan Dario Amodei Ilya Sutskever
references.title: Language models are unsupervised multitask learners
references.authors: Nils Reimers Iryna Gurevych Nandan Thakur Johannes Daxenberger
references.title: Sentence-bert: Sentence embeddings using siamese bert-networks
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,
references.authors: Daniel Revuz Marc Yor
references.title: Continuous martingales and Brownian motion, volume 293
references.venue: Springer Science & Business Media,
references.authors: Thibault Sellam Dipanjan Das Ankur P. Parikh
references.title: Bleurt: Learning robust metrics for text generation, 2020
references.authors: Amanda Stent Rashmi Prasad Marilyn Walker
references.title: Trainable sentence planning for complex information presentation in spoken dialog systems
references.venue: In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics,
references.authors: Alex Tamkin Dan Jurafsky Noah Goodman
references.title: Language through a prism: A spectral approach for multiscale language representations
references.venue: Advances in Neural Information Processing Systems,
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding, 2019
references.authors: Noah Weber Leena Shekhar Heeyoung Kwon Niranjan Balasubramanian Nathanael Chambers
references.title: Generating narrative text in a switching dynamical system
references.venue: In Proceedings of the 24th Conference on Computational Natural Language Learning,
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Rémi Louf Morgan Funtowicz Joe Davison Sam Shleifer Patrick von Platen Clara Ma Yacine Jernite Julien Plu Canwen Xu Teven Le Scao Sylvain Gugger Mariama Drame Quentin Lhoest Alexander M. Rush
references.title: Transformers: State-of-the-art natural language processing
references.venue: In Proceedings of the 2020 Conference on Empirical Methods
references.authors: Jiacheng Xu Zhe Gan Yu Cheng Jingjing Liu
references.title: Discourse-aware neural extractive text
references.authors: Vancouver Canada July
references.title: Association for Computational Linguistics
references.venue: doi: 10.18653/v1/
references.authors: T ). Liu
references.title: 2021) characterize properties of an optimal classifier h∗(z, z′) which observes pairs of latents (z, z′) and it outputs in [0, 1], a probability indicating whether the pair comes in order (ie. z′ = zt + ∆t · dzt) or not in order (ie. a randomly sampled latent)
references.authors: Following Liu
references.title: 2021), minimizing L yields the following on each triplet: log p(z
references.authors: van den Oord
references.title: The Implicit Dynamics ablation
references.authors: van den Oord
references.title: We then train both encoders using the InfoNCE loss, as done in prior work. Since genc and gar are trained to align latents up to a linear rotation, we use genc for extracting the sentence embeddings
references.authors: Wikihow (Koupaee Wang
references.title: 2018) contains how-to articles organized by a title, method, and steps. Each article includes multiple methods for completing a multi-step procedural task such as “How to Register to Vote”. We scraped all the available English articles covering a wide range of topics following Koupaee & Wang (2018). We mark each with its own section id tokens: Each article is represented as “[TITLE] text [METHOD] text [STEP] 1 text [STEP] 2 text ...
references.authors: Byrne
references.title: 2019) which contains conversations on finding restaurants between an assistant and a user. The assistant’s turn is marked with an “[ASSISTANT]” tag, and the user’s turn is marked with a “[USER]
references.authors: TicketTalk (Byrne
references.title: 2021) which contains conversations on booking movie tickets between an assistant and a user. The assistant’s and user’s turned are similarly marked as in TM-2
sections.heading: 1 INTRODUCTION
sections.text: Large language models (LLM) such as GPT-2 have been extremely successful in text generation (Radford et al., 2019; Brown et al., 2020). However, LLMs are known to generate incoherent long texts. One reason is that they are unable to plan ahead or represent long-range text dynamics (Kiddon et al., 2016; Fan et al., 2019; Hua & Wang, 2020; Duboue & McKeown, 2001; Stent et al., 2004; Tamkin et al., 2020). As a result, they oftentimes produce wandering content with poor discourse structure and low relevance (Hua & Wang, 2020; Zhao et al., 2017; Xu et al., 2020)—the text reads as if the model has no anchored goal when generating. These problems with coherence are further exacerbated when forcing autoregressive models to generate longer texts as the model struggles to extrapolate beyond its expected text end point. These problems suggest that LLMs currently fail to properly capture how documents evolve from beginning to end and condition generation on goals. Doing so is critical for succeeding in goal-oriented tasks such as story, dialog or recipe generation.
  Prior work has explored the use of planning-based methods for generating globally coherent text (Kiddon et al., 2016; Fan et al., 2019; Hua & Wang, 2020; Duboue & McKeown, 2001; Stent et al., 2004). However, these methods rely on manually defining text dynamics for specific domains. Other work has attempted to use sentence representations for modeling text, such as with variational autoencoders (Bowman et al., 2016) or contrastive learning (Gao et al., 2021; Devlin et al., 2019). Their shortcoming in text generation settings is that the latent representations are static: they capture semantic similarity between sentence neighbors, but don’t capture how sentence embeddings evolve over a document. Methods including van den Oord et al. (2019) have tried to remedy this by learning a model of local latent dynamics. However, it is difficult to use learned local dynamics for generating accurate goal-conditioned trajectories, especially long-horizon ones. We explore an alternative that explicitly assumes a simple, fixed dynamics model with goal-conditioned generation.
  In this work, we propose Time Control as a way to learn a latent space with known, goal-conditioned dynamics. We begin by assuming that meandering text generated without a goal can be represented as Brownian motion in latent space; this motion enforces the embeddings of neighboring sentences to be similar to each other, whereas those of distant sentences to be dissimilar. Goal-directed behavior can be incorporated into this model by conditioning on a fixed start and end point. In this case, the Brownian motion becomes a Brownian bridge. The resulting latent trajectories are controllable and abide by simple, closed-form dynamics.
  In Time Control, we derive a novel contrastive objective for learning a latent space with Brownian bridge dynamics. We can then use this latent space to generate text that retains local coherence and has improved global coherence. To perform text generation, Time Control first plans a latent trajectory via the Brownian bridge process pinned at a start and end point. It then conditionally generates sentences using this latent plan. In our work, we decode latent plans by fine-tuning GPT2 to generate text conditioned on Time Control’s latent trajectory. Trajectories from Time Control act as abstract semantic positions in a document that guide generation of fine-tuned language models.
  In summary, our work’s contributions are the following:
  • We derive Time Control, a language model which explicitly models latent structure with Brownian bridge dynamics learned using a novel contrastive objective.
  • Across a range of text domains, we show that Time Control generates more or equally coherent text on tasks including text infilling and forced long text generation, compared to task-specific methods.
  • We validate that our latent representations capture text dynamics competitively by evaluating discourse coherence with human experiments.
  • We ablate our method to understand the importance of the contrastive objective, enforcing Brownian bridge dynamics, and explicitly modeling latent dynamics.
sections.heading: 2 RELATED WORKS
sections.text: Generating long, coherent text is conceptually difficult for autoregressive models because they lack the ability to model text structure and dynamics (Lin et al., 2021). This means that they struggle to plan and look ahead which leads to generating globally incoherent text. Forcing autoregressive models to generate longer texts exacerbates this incoherence because the models struggle to extrapolate beyond their expected text end point. Prior work has tried to address the problem of generating globally coherent text with planning-based approaches (Puduppully et al., 2019; Moryossef et al., 2019; Fan et al., 2019; Kiddon et al., 2016). However, planning-based approaches rely on domain-specific heuristics for capturing text structure and dynamics.
  Our work uses a contrastive objective to learn latent dynamics in text without domain-specific heuristics. Contrastive objectives have been applied to several domains, including language (Devlin et al., 2019; Iter et al., 2020; Liu & Liu, 2021), vision (Chen et al., 2020), and general time series data (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019). In particular for language, contrastive objectives have been applied to the next-sentence prediction task for improving BERT embeddings (Devlin et al., 2019) and to the discourse coherence setting (Nie et al., 2019; Chen et al., 2019) for evaluating how coherent pairs of sentences are. However, these methods have two shortcomings which we address with our work. One is that the resulting sentence embeddings are often static: they capture semantic similarity between sentence neighbors, but don’t capture how sentence embeddings evolve over a document. Two is that they are not used for generation and are limited to classification tasks like discourse coherence. Prior work has also tried fitting latent variable models, however these generally result in poor language generation (Bowman et al., 2016) or are domain-specific (Weber et al., 2020; Arora et al., 2016) .
  Our work is closely related to Contrastive Predictive Coding (CPC) from van den Oord et al. (2019) which learns the implicit latent dynamics with an autoregressive model. CPC is different from our work in that we impose constraints with known dynamics onto our latent space and assume goalconditioned generation, whereas they learn an unconditioned dynamics model. By assuming that text dynamics follows a well-behaved stochastic process, we benefit significantly during generation. Generating from future time points with CPC results in extrapolation error due to compounding errors from its forward model. In contrast, our approach has an exact, closed-form forward model which allows us to extrapolate substantially further in time.
  Our method builds off of recent findings that contrastive objectives can be used to approximate local transition kernels of stochastic processes (Liu et al., 2021). The main difference between Liu et al. (2021) and our work is that they focus on providing provable conditions for latent recovery, whereas we focus on building empirically effective methods that leverage similar insights for recovering latent representations from language.
sections.heading: 3 TIME CONTROL
sections.text: The intuition behind Time Control is to learn a latent space with smooth temporal dynamics for modeling and generating coherent text. We detail Time Control in three steps: The first step trains an encoder to map sentences to a Brownian bridge (Revuz & Yor, 2013) latent space. The second step trains a decoder to reconstruct sentences from this latent space. The third step generates text by first sampling a latent plan, then generating using a decoder on this latent plan.
sections.heading: 3.1 TRAINING AN ENCODER WITH BROWNIAN BRIDGE DYNAMICS
sections.text: Our encoder is a nonlinear mapping from raw input space to latent space, fθ : X → Z . The objective for the encoder is to map high-dimensional sequential data into low-dimensional latents which follow a stochastic process of interest—in this paper, it is the Brownian bridge process. The density of a Brownian bridge process between an arbitrary start point z0 at t = 0 and end point zT at t = T is,
  p(zt|z0, zT ) = N (
  (1− t T )z0 + t T zT , t(T − t) T
  ) . (1)
  This density is intuitive to understand: It acts like a noisy linear interpolation between the start and end point of the trajectory, where zt should be more like z0 at the start and more like zT at the end of the trajectory. Uncertainty is highest in the middle region, and low near the end points. See Figure 1.
  Consider a set of triplet observations, (x1, x2, x3). The goal of our work is to ensure that fθ(x1), fθ(x2), fθ(x3) follow the Brownian bridge transition density in Equation 1. We ensure this using a contrastive objective. Formally, given multiple sequences of data points, X = {x1, ..., xN}, we draw batches consisting of randomly sampled positive triplets x0, xt, xT where 0 < t < T : B = {(x0, xt, xT )}.1 Our encoder is optimized by,
  LN = EX [ − log exp(d(x0, xt, xT ; fθ))∑
  t′∈B exp(d(x0, xt′ , xT ; fθ))
  ] ,where (2)
  d(x0, xt, xT ; fθ) = − 1 2σ2 ‖ fθ(xt)︸ ︷︷ ︸
  zt
  − (1− t T )fθ(x0)− t
  T fθ(xT )︸ ︷︷ ︸
  mean in Equation 1
  ‖22 (3)
  σ2 is the variance in Equation 1: t(T−t)T . Note that Equation 2 sums over negative middle contrasts, xt′ . This objective can be viewed as maximizing the extent to which true triplets from the data follow the Brownian bridge process while minimizing the extent to which an alternative mid-point sampled from another sequence does so. 2
  1We use indices 0, t, T to denote the start, middle and end point of a Brownian bridge, but these do not correspond to strictly sampling the first, middle and last sentence of a document. x0, xt, xT can be any sentence in a document as long as x0 comes before xt and xt before xT in the document.
  2Empirically, we found Brownian bridge dynamics easier to recover with triplets rather than pairs of contrasts. Appendix M.2 discusses some of the pair-wise contrast results.
  Figure 1 illustrates how the objective translates into the language setting for training the encoder. The objective samples triplet sentences from a document. Sentences drawn from the same document make up a smooth latent trajectory; they should be close to each other and follow the conditional density in latent space. Sentences drawn from different documents should not make up a smooth trajectory and should less likely follow bridge dynamics.
  Connection to mutual information estimation and triplet classification We draw connections between our contrastive loss and the mutual information estimation setup from van den Oord et al. (2019); Poole et al. (2019) (as |B| → ∞) as well as the classification setup from Liu et al. (2021) (|B| = 1). Following van den Oord et al. (2019); Poole et al. (2019), this objective can be seen as a lower bound on the mutual information between the two end points and the middle point: I(Xt, {X0, XT }) ≥ log(N) − LN . Hence, by minimizing the contrastive loss, we are maximizing the amount of information between the trajectory and the linear interpolation of its end points.
  Assuming |B| = 1, we can draw a connection to the classification setup studied in Liu et al. (2021) where they train a classifier to distinguish in- vs. out-of-order pairs of inputs. They show that the Bayes optimal logits for pair-wise classification can be written as a function of the stochastic process transition kernel, which is equivalent to the loss on a single triplet i in our contrastive objective: li = − log exp(d(x0,xt,xT ;fθ))exp(d(x0,xt,xT ;fθ))+exp(d(x0,xt′ ,xT ;fθ)) . We show in Appendix A that the pairwise classification setup is equivalent to the triplet classification setup.
sections.heading: 3.2 TRAINING A DECODER WITH LATENT PLANS
sections.text: Here we discuss how to train a language model to decode latent plans for generation. We first map all the sentences in the training dataset to our learned latent space using the pretrained encoder fθ. This gives us a Brownian bridge trajectory of sentence-level latent codes (z0, . . . , zt, . . . , zT ) for a document in the dataset. Then, rather than learning a decoder from scratch, we fine-tune GPT2 (Radford et al., 2019) to generate text conditioned on past context and the latent plan.
  We fine-tune in the following manner. Let x1 . . . xW be a document with W tokens and S sentences used to train the decoder. Using the encoder fθ, we can obtain embeddings z1 . . . zS for each sentence. The decoder is a standard auto-regressive language model that is modified in the following way: at time t, the decoder must predict xt using all tokens in the past x<t, as well as the sentence embedding zst , where the index st ∈ [S] is a map which takes each token to its corresponding sentence. This is a form of a reconstruction objective, as the identity of xt is encoded in zst .
sections.heading: 3.3 GENERATING TEXT WITH LATENT PLANS AT INFERENCE TIME
sections.text: Figure 2 illustrates how the trained decoder generates text at inference time. Given two end points z0, zT , we sample a trajectory from a latent Brownian bridge, and then generate from the decoder conditioned on this bridge. In many situations, we may not know the endpoints of the Brownian bridge explicitly. In this case, we encode a set of sentences corresponding to start and end points (eg. the first and last sentences of our training set), and fit a Gaussian to these points to form a density estimate. Generating in this case involves first sampling from the Gaussian, and then generating as before from the bridge. More details on training and generation can be found in Appendix C.
sections.heading: 4 EXPERIMENTS
sections.text: We now evaluate the ability of Time Control to capture text dynamics. Specifically, we aim to answer the following research questions (RQ):
  RQ1: Can Time Control model local text dynamics? Section 4.1 investigates this question using a sentence ordering prediction task: given two sentences from the same document, we evaluate whether different models can predict their original order.
  RQ2: Can Time Control generate locally coherent text? Section 4.2 investigates this question using the text-infilling task: given prefix and suffix, we evaluate how well different models can fill in between, evaluated using similarity metrics between generated and ground truth sentences, and human evaluations.
  RQ3: Can Time Control model global text dynamics? Section 4.3 investigates this question on text generation for Wikipedia city articles by examining the length of generated sections.
  RQ4: Can Time Control generate long coherent documents? Section 4.4 investigates this question on forced long text generation: we evaluate how well models preserve global text statistics (such as typical section orders and lengths) when forced to extrapolate during generation.
  We run Time Control with different latent dimensions (d = 8, 16, 32). Our encoder architecture is a frozen, pretrained GPT2 model from Huggingface (Radford et al., 2019; Wolf et al., 2020) and a trainable MLP network. We extract GPT2’s last layer hidden state that corresponds to the end-ofsentence (EOS) token and train the 4-layer MLP on top of the hidden state. The MLP network has intermediate ReLU activations and is trained with stochastic gradient descent with a learning rate of 1e-4 and with momentum 0.9.
  Ablations We perform three ablations on our encoder model. Recall that Time Control (A) explicitly models latent structure with (B) Brownian bridge dynamics using a (C) contrastive loss. (A) replaces explicit dynamics with Implicit Dynamics (ID) where future latents are directly predicted with an autoregressive model (van den Oord et al., 2019). (B) replaces Brownian bridge dynamics with Brownian motion (BM): latents follow the transition density zt|zs ∼ N (zs, t−s) in Equation 3. Note zt is centered at zs and is not conditioned on a goal end-point. (C) replaces the contrastive loss with the Variational Auto-Encoder (VAE) and center the priors over z0 to 0 and zT to 1, as done in our setup. We use GPT2 as the decoder. Appendix E includes more detail on the ablations.
  Datasets We use language datasets that elicit different kinds of structure, from section structure to discourse structure to narrative structure. Time Control does not take in any information about the structure, treating each domain the same under its encoding objective. More information and dataset examples are provided in Appendix F. Wikisection (Arnold et al., 2019) includes Wikipedia articles on cities split by sections. We adapt this dataset such that each article contains four ordered sections (abstract, history, geography, demographics) marked with section id tokens: Each article is represented as, “[ABSTRACT] text [HISTORY] text [GEOGRAPHY] text [DEMOGRAPHICS] text”. Wikihow (WH) (Koupaee & Wang, 2018) contains how-to articles organized by a title, method, and steps. We mark each with its own section id tokens: Each article is represented as “[TITLE] text [METHOD] text [STEP] 1 text [STEP] 2 text ...” Recipe NLG (Bień et al., 2020) contains recipes, each with a title, ingredients and set of directions. A recipe is constructed as “[TITLE] text [INGREDIENTS] text [DIRECTIONS] text”. Taskmaster-2 (TM-2) (Byrne et al., 2019) contains conversations on finding restaurants between an assistant and a user. The assistant’s turn is marked
  Wikisection TM-2 TicketTalk Method k = 5 k = 10 k = 5 k = 10 k = 5 k = 10
  with an “[ASSISTANT]” tag, and the user’s turn is marked with a “[USER]” tag. TicketTalk (Byrne et al., 2021) contains conversations on booking movie tickets between an assistant and a user. The assistant’s and user’s turns are similarly marked as in TM-2. ROC Stories (Mostafazadeh et al., 2016) is a short 5-sentence stories dataset. No additional tokens are added in this dataset.
sections.heading: 4.1 MODELING LOCAL TEXT DYNAMICS
sections.text: We evaluate how well Time Control models local text dynamics (RQ1) on the discourse coherence setting (Jurafsky, 2000). Discourse coherence can be measured by how well representations capture discourse structure by testing for whether a linear classifier can detect in-order and vs. out-oforder sentence pairs. We compare Time Control’s encoder against GPT2’s last layer’s hidden state corresponding to the EOS token (Radford et al., 2019), BERT (Devlin et al., 2019), ALBERT (Lan et al., 2019) , Sentence BERT (Reimers et al., 2019), and SimCSE (Gao et al., 2021). The latter 3 methods are designed as sentence embedding models. We also compare to the ablated methods.
  The setup is the following: The encoder takes in two sentences xt, xt+k to produce their latents zt, zt+k ∈ Rd. At random, the latents are fed either in- or out-of-order. A linear classifier is trained on 100 epochs with stochastic gradient descent with a learning rate of 1e-4 and with momentum 0.9.
  We varied the sentence distance k ∈ {1, 5, 10} and found that on some domains and for k = 1, all methods scored near random accuracy; we have omitted those results in the main paper. Otherwise, the results are summarized in Table 1 where we report the mean and standard error accuracy on a held-out discourse dataset of 3000 examples on 3 runs. Our method is able to outperform and compete with sentence embedding specific methods, like Sentence BERT and SimCSE. Additionally, though GPT2 is used as a base encoder for Time Control, we observe that Time Control greatly improves upon GPT2’s representations with significant gains on TM-2 and TicketTalk (around 20− 25%) and lesser gains on Wikisection (around 7− 15%). Neither VAE nor BM perform better than random accuracy on any of domains. This suggests that the variational lower bound does not recover the latent embeddings as well as contrastive objectives and the choice of stochastic processes matter for learning informative structural embeddings.
  ID does best on the conversation datasets, which indicates that implicitly learning text dynamics yields latent representations that can be used for inferring discourse though not as effectively as Time Control.
  Noticeably, discourse is more easily inferred on task-oriented conversations like TM-2 and TicketTalk. We hypothesize that the ordering of responses matters more in conversations than enumerative, factual settings like Wikisection. Nonetheless, our model performs above chance on both types of domains. This answers RQ1 in the positive: Time Control can model local text dynamics, like in conversations and articles.
sections.heading: 4.2 GENERATING LOCALLY COHERENT TEXT
sections.text: We evaluate how well Time Control generates locally coherent text (RQ2) on the text-infilling setting. Text-infilling requires a model to take an incomplete text, with missing sentences, and complete it. An example input could be, “Patty was excited about having her friends over. [blank] Patty had a great time with her friends.” The challenge in performing text infilling is to generate a sentence that is locally coherent with the left and right neighboring sentences.
  We follow the task setup in Donahue et al. (2020) where they use the ROCStories dataset. Each story in the dataset contains 5 sentences, one of which is randomly masked. Time Control fills in the masked sentence by running the bridge process pinned at the latent for the prefix, z0, and the latent for the suffix, zT . We compare our method against ILM (Donahue et al., 2020), a state-of-the-art method for text-infilling, as well as Donahue et al. (2020)’s LM model, an autoregressive baseline that only sees the prefix. We compare against the ablations; the prefix and suffix are encoded by the ablation’s encoder and the average of the encodings is used for generation. When using the average embeddings, BM and Time Control are equivalent; we omit the BM results.
  We evaluate the text coherence with the BLEU score (Papineni et al., 2002), ROUGE (Lin, 2004), BLEURT (Sellam et al., 2020) and BERTScore (Zhang et al., 2019) between the generated and ground truth infill sentence. Due to space constraints, the last three metrics are in the Appendix, Table 17. We also report human evaluations on how coherent the generated sentence is as a fill-in sentence. Participants were asked to rank the generated fill-in sentence from ILM, LM, and Time Control on a scale of 1-5 (not reasonable to very reasonable). Appendix I includes more details on the human experiment setup.
  The BLEU scores are summarized in Table 2. Time Control generates fill-in sentences that much more closely overlap with the ground truth than ILM and LM. The ablated methods perform worse to varying degrees. On average, VAE performs worse than LM and ID. This suggests that the interpolating embeddings learned via the variational objective yields embeddings which hurt the autoregressive model’s ability to generate locally coherent text.
  Method Wikisection Wikihow TicketTalk Recipe
  GPT2 50.4± 1.0 61.5± 3.5 75.8± 1.6 36.8± 3.7 VAE (8) 57.1± 3.5 66.3± 1.7 66.1± 4.1 71.4± 0.6 VAE (16) 47.3± 3.3 60.5± 2.9 0.8± 0.0 45.6± 0.5 VAE (32) 58.3± 0.3 60.9± 5.6 38.9± 1.7 87.5± 0.2 ID (8) 34.6± 0.0 59.3± 2.9 30.8± 1.6 68.7± 2.3 ID (16) 35.7± 0.0 31.7± 0.9 63.9± 0.4 78.1± 3.1 ID (32) 47.9± 2.0 16.9± 0.6 45.6± 0.6 85.6± 2.3 BM (8) 54.7± 1.9 56.0± 3.5 77.0± 4.7 82.7± 0.4 BM (16) 61.4± 1.0 63.8± 4.7 43.3± 2.4 34.9± 1.5 BM (32) 44.3± 1.4 50.2± 1.2 14.8± 1.0 87.1± 0.1 TC (8) 52.3± 2.5 76.7± 7.5 81.8± 1.0 41.7± 1.0 TC (16) 57.9± 1.0 63.1± 6.2 88.0± 1.3 64.1± 1.0 TC (32) 36.5± 2.8 59.1± 5.5 83.6± 1.3 76.4± 1.0
  Table 5: Ordering in forced long text generation. ROC Stories and TM-2 omitted because they are not applicable. Values are bolded if they overlap with in range with the highest mean score and its corresponding standard error. The highest mean score are marked in gray cells.
  Method Human
  LM 2.4± 0.06 ILM 3.77± 0.07 TC (8) 3.64± 0.07
  Table 6: Human evaluations on text infilling. Scores were ranked between 1 and 5. Higher is better.
  Method Human
  GPT2 2.8± 0.06 TC (8) 3.6± 0.07 TC (16) 3.4± 0.07 TC (32) 3.3± 0.07
  Table 7: Human evaluations on tail end quality in forced long text generation. Scores were ranked between 1 and 5. Higher is better.
  Human evaluations in Table 6 indicate that Time Control performs competitively with ILM.
  Upon inspection of the model output, we find that the ablated models struggle to meaningfully decode from the interpolated latent embeddings (Appendix H.1). Oftentimes, the model would start a sentence coherently, but end the sentence by repeating a word over and over again; examples can be found in Appendix H. This suggests the importance of learning a latent space that supports interpolation by construction. All together, the results provide positive evidence for RQ2: Time Control can generate locally coherent text due to its well-defined latent dynamics.
sections.heading: 4.3 MODELING GLOBAL TEXT DYNAMICS
sections.text: We evaluate how well Time Control models global text dynamics (RQ3) by assessing whether the methods exactly mimic document structure on Wikisection: We check whether the generated section lengths match with the average lengths in the dataset. We focus on Wikisection because it is the only dataset with long, well-defined sections (refer to Appendix F for dataset statistics). Each document contains an abstract, history, geography, and demographics section on a city.
  Time Control plans a latent trajectory by running the bridge process between the start and end latent, z0 ∼ p(z0) and zT ∼ p(zT ). We compare Time Control to fine-tuned GPT2. We also include two oracle methods which are fine-tuned GPT2 models with additional section-embedding supervision. One is “sec-dense GPT2” (SD) where the section embeddings are fed in a dense manner; for every token associated with section i, section i’s embedding is added onto the positional token embeddings. The other is “sec-sparse GPT2” (SS) where the embeddings are fed in a sparse manner, ie. right before the new section starts. For VAE and ID, we calculate the density estimate p(z0) and p(zT ) and run a linear interpolation between the start and end latents. For BM, we calculate the density estimate p(z0) and run Brownian motion.
  Table 3 reports in percentage how much the generated section lengths deviate from the average section lengths. Time Control best matches the section lengths out of all the methods. One surprising observation was that fine-tuned GPT2 mismatches the section lengths significantly by almost 20%. Upon further investigation, we noticed GPT2 overshoots short sections and undershoots long sections; this causes the section length deviations. Adding domain-specific supervision like with SD and SS slightly alleviates this issue. Surprisingly, we find that our method beats the oracle on certain settings, eg. d = 16. ID performs extremely badly in contrast to VAE and BM; this highlights the challenge of interpolating on learned dynamics models which is exacerbated in the long text generation settings. The results affirm the importance of Time Control for modeling global text dynamics, such as in matching document structure, thus answering RQ3 positively.
sections.heading: 4.4 GENERATING GLOBALLY COHERENT TEXT
sections.text: We evaluate how well Time Control generates globally coherent text (RQ4) on forced long text generation. The EOS token is suppressed and forces the model to generate at least 1000 tokens, extrapolating beyond its natural end point in generation. Please refer to Appendix D.2 on how the latent plan is generated. For reference, 1000 tokens is about 50% longer than the average Wikisection document (the longest text domain). This setting is intrinsically difficult for auto-regressive models which do not have the ability to “look ahead” during generation (Lin et al., 2021).
  We evaluate model extrapolation on three metrics. First is ordering: how well do the models maintain the structure of the document (eg. turn-ordering in TicketTalk)? Second is length consistency. Length consistency captures common failure modes such as a model which stops modeling a conversation between two agents, and instead outputs a long monologue from one agent. Third is human evaluations of the long-tail text quality behavior.
  The results for length consistency and ordering are summarized in Table 4 and 5. Overall, our method is able to maintain text flow the best according to these two metrics. A common failure mode of GPT2 is that it delegates the remaining tokens to the last section of each text; this is particularly noticeable on TM-2 and TicketTalk. We show an example of our model’s behavior versus GPT2’s behavior on generating long texts in TicketTalk in Figure 8. The example illustrates how Time Control maintains dialog whereas GPT2 utters nonsensical text.These results are additionally confirmed by our human evaluation experiments in Table 7. We find that human evaluators rank Time Control’s extrapolation ability better on all latent dimension settings than that of GPT2.
  ID performs poorly in forced long text generation, similar to the normal text generation setting: It significantly misorders sections and overshoots section lengths. Although VAE and BM do poorly on length consistency, they perform slightly better than ID on ordering; in Appendix K, we investigate why the VAE baseline performs much better than TC on ordering. This suggests the importance of good latent dynamics for long text generation. These results answer RQ4 positively: Time Control generates globally coherent text thanks to its ability to plan ahead and correctly generate future latent variables following its latent dynamics.
sections.heading: 5 CONCLUSION
sections.text: We propose Time Control, a language model that implicitly plans via a latent stochastic process. Specifically, Time Control looks to Brownian bridge processes as a desirable latent space. Time Control learns to map coherent text to smooth Brownian bridge trajectories. Empirically, we demonstrate this leads to several benefits such as generating more locally and globally coherent text. Although our work focuses on benefits of learning stochastic process latents for language, it can be extended to other domains with sequential data like videos or audio, or extended to handle arbitrary bridge processes without known fixed start and end points.
sections.heading: 6 REPRODUCIBILITY STATEMENT
sections.text: In the supplemental, we include a zip file containing our code and processed datasets. We’ve also included in the appendix how we processed the datasets. Our results on showing equivalence between the triplet classification setting and pairwise classification setting are also included in the appendix.
sections.heading: A SHOWING THE EQUIVALENCE OF THE PAIRWISE AND TRIPLET CLASSIFICATION TASK
sections.text: In this section, we focus on the classification setup where the goal is to train a classifier to detect invs. out-of-order groups of inputs. Specifically, we want to show that the classification setup from Liu et al. (2021) where the inputs come in pairs is equivalent to the setup where the inputs come in triplets. The triplet inputs are what we assume for Time Control.
  Notation We denote latent space as Z; we typically do not observe this space directly. We denote the observation space as X ; this is the space we observe directly. We define the Brownian bridge for t ∈ [0, T ] as
  Bt = B(t) = W (t)− t
  T W (T ), (4)
  where W (t) is a standard Wiener process, N (0, t). We state the assumptions:
  • We assume latents {zt}t≥0 ∈ Rd are drawn from the Brownian bridge process defined by the stochastic differential equation,
  dzt = zT − zt 1− tT dt+ dBt (5)
  The intervals at which latents are sampled are every ∆t step: z′ = zt+∆t. • We denote the transition probabilities as
  p∗(zt|z0, zT ) := N (
  (1− t T )z0 + t T zT , t(T − t) t
  ) . (6)
  We denote the proposal distribution over possible intermediate triplets q(z′0, z ′ t, z ′ T ).
  Liu et al. (2021) characterize properties of an optimal classifier h∗(z, z′) which observes pairs of latents (z, z′) and it outputs in [0, 1], a probability indicating whether the pair comes in order (ie. z′ = zt + ∆t · dzt) or not in order (ie. a randomly sampled latent). They train h∗ using an L2 loss, L(h, {(z, z′), y}). Lemma 1 of their work states the following : The optimum of the contrastive learning objective arg minh E((z,z′),y)[L(h, {(z, z′), y})] satisfies
  h∗(z, z′) = p∆t(z, z′)
  q(z′) + p∆t(z, z′) . (7)
  Manipulating this equality, we observe that the transition kernel has the following relation to the classifier which takes in pairs of observations,
  p∆t(z, z′) = q(z′)h∗(z, z′)
  1− h∗(z, z′) (8)
  log p∆t(z, z′) = log q(z′) + log h∗(z, z′)− log(1− h∗(z, z′)). (9)
  Our setting however assumes that the algorithm receives triplets of data points, zt1 , zt2 , zt3 . We want to show below that minimizing L with triplet contrasts in the classification setting still approximates the transition kernel. In particular, we’re interested in transitions of a Brownian bridge pinned at z0, zT : p∆t(z0, zt, zT ) = Pr(zt|z0, zT ).
  Let’s say we have two positive triplet samples, (zit1 , z i t2 , z i t3) and (z j t1 , z j t2 , z j t3), where t1 < t2 < t3. Following Liu et al. (2021), minimizing L yields the following on each triplet:
  log p∆t(zit1 , z i t2 , z i t3) = log q(z ′) + log h∗(zit1 , z i t2 , z i t3)− log(1− h ∗(zit1 , z i t2 , z i t3)) (10) log p∆t(zjt1 , z j t2 , z j t3) = log q(z ′) + log h∗(zjt1 , z j t2 , z j t3)− log(1− h ∗(zjt1 , z j t2 , z j t3)). (11)
  Taking the difference in log probabilities between Equations 10-11 results in
  log p∆t(zit1 , z i t2 , z i t3)− log p ∆t(zjt1 , z j t2 , z j t3) = [ log h∗(zit1 , z i t2 , z i t3)− log(1− h ∗(zit1 , z i t2 , z i t3)) ]
  − [ h∗(zjt1 , z j t2 , z j t3)− log(1− h ∗(zjt1 , z j t2 , z j t3)) ] .
  (12)
  Similar to the pair-wise classification setting, we’ve shown that minimizing L in the triplet classification setting results in approximating the transition kernel of the Brownian bridge process.
sections.heading: B FIRST ORDER APPROXIMATION OF THE BROWNIAN BRIDGE PROCESS
sections.text: Zt+∆t ≈ Zt(1− 1 T − t )− 1 T − t ∆t · ZT +
  √ ∆tN (0, I) (13)
sections.heading: C TRAINING DETAILS
sections.text: Here we describe the training procedure for the encoder and the fine-tuning procedure for decoding.
  Brownian bridge encoder The encoder architecture is a frozen, pretrained GPT2 model from Huggingface (Radford et al., 2019; Wolf et al., 2020) and a trainable MLP network. We extract the GPT2’s last layer hidden state that corresponds to the end-of-sentence (EOS) token and train the 4-layer MLP on top of the hidden state. The MLP network has intermediate ReLU activations and is trained with stochastic gradient descent with a learning rate of 1e-4 and with momentum 0.9. We train the encoder for 100 epochs on each of the datasets.
  The text fed into GPT2 are fed in on a sentence level. This means that the input xt refers to the t’th sentence of a document. The sentences are separated from each other in the main text as “ . ” which is added to the tokenizer as a separate token for indexing convenience.
  Fine-tuning GPT2 with latent embeddings After training the encoder, we run it on the training dataset to collect an accompanying latent trajectory for each text. The encoder is run on the dataset at a sentence level: we separate the text by sentences and pass the sentences through the encoder.
  The sentence latent embeddings are aligned with the tokens of that sentence and offset by one token before the start of that sentence token. Let’s illustrate by an example. We denote [SOS] as the start of the document token, [s1] as sentence 1 tokens and [s2] as sentence 2 tokens. [ . ] is the period token which we’ve added into the tokenizer. zi denote the latent variable corresponding to the i’th sentence.
  Let’s say that the sequence fed into GPT2 is “[SOS] [s1] [s1] [s1] [ . ] [s2] [s2] [s2]”. Then the corresponding latent trajectory is “z1, z1, z1, z1, z2, z2, z2, z2”. The latent variables are added onto the positional embeddings. We then fine-tune GPT2 as normal.
sections.heading: D GENERATION WITH EMBEDDINGS
sections.text: D.1 NORMAL TEXT GENERATION
  We first sample a start and end latent, z0 ∼ p(z0), zT ∼ p(zT ) where p(z0), p(zT ) are calculated as the density estimates over the training dataset. We pin our trajectory to the start and end latent, and run the Brownian bridge using Equation 13. For normal long text generation, we set T to be the average number of sentences in each of the dataset. For forced long text generation, we set T to be proportional to the number of sentences needed in order to generate 1000 tokens. By the end, we have a trajectory z0, z1, ..., zT .
  Generation starts with feeding the SOS token and the first latent z0. Once GPT2 emits a [ . ] token and terminates sentence t, we transition to the next latent zt+1. This process continues until GPT2 is finished with generation. If GPT2 generates more sentences than there are latents in the trajectory, the last latent zT is used until the end of generation.
  D.2 FORCED LONG TEXT GENERATION
  Let Savg(D) denote the average number of sentences in a document and Tavg(D) denote the average number of tokens in a document. Rather than planning a trajectory of length Savg(D) (the average number of sentences in a document) which is what is done in Section 4.3 for normal text generation, we scale the trajectory length to c · Savg(D). c is determined by how many more tokens we need in order to fill up to GPT-2 maximum context length of 1024: c = 1024−Tavg(D)Tavg(D) .
sections.heading: E ABLATIONS
sections.text: The following methods ablate the encoder model in Time Control; in other words, the ablations dissect the assumptions we make in the first step of Time Control described in Section 3.1. Recall that Time Control (A) explicitly models latent structure with (B) Brownian bridge dynamics using a (C) contrastive loss. (A) replaces explicit dynamics with Implicit Dynamics (ID) where future latents are directly predicted with an autoregressive model (van den Oord et al., 2019). (B) replaces Brownian bridge dynamics with Brownian motion (BM) which doesn’t rely on pinning trajectories. Latents follow the transition density zt|zs ∼ N (zs, t− s). (C) replaces the contrastive loss with the Variational Auto-Encoder (VAE). Below we detail how these ablations are implemented.
  E.1 IMPLICIT DYNAMICS (ID)
  The Implicit Dynamics ablation is where we compare our explicit dynamics objective to van den Oord et al. (2019) which suggests an implicit latent dynamics objective. This ablation thus changes the learned latent dynamics. In van den Oord et al. (2019), they train both a non-linear encoder genc and an autoregressive context model gar. The encoder genc for Implicit Dynamics has the same architecture as Time Control. The context encoder gar using a 2400-hidden-unit GRU, as done in van den Oord et al. (2019). We then train both encoders using the InfoNCE loss, as done in prior work. Since genc and gar are trained to align latents up to a linear rotation, we use genc for extracting the sentence embeddings.
  E.2 BROWNIAN MOTION (BM)
  The Brownian motion ablation is where we remove goal-conditioning in the dynamics. The main change is in distance function in Equation 3. BM instead optimizes the following equation:
  d(xt, xt′ ; fθ) = − 1 2σ2 ‖ fθ(xt′)︸ ︷︷ ︸
  zt′ − fθ(xt)︸ ︷︷ ︸ zt ‖22 (14)
  where σ2 is the variance in of the Wiener process, σ2 = t′ − t.
  E.3 VARIATIONAL AUTO-ENCODER (VAE)
  The Variational Auto-Encoder ablation is where we compare our contrastive objective with a variational one. This ablation changes the encoder objective from Section 3.1 from Equation 2 to the ELBO objective. Below we derive the ELBO objective. Similar to the contrastive objective, we’re deriving the ELBO over the triplet dynamics in the Brownian bridge.
  log p(x) ≥ Eqφ(z|x)[log p(x, z)
  q(z|x) ]
  = Eqφ(z|x)[log p(x0, xt, xT , z0, zt, zT )
  q(z0, zt, zT |x0, xt, xT ) ]
  = Eqφ(z|x)[log p(x0|z0)p(xt|zt)p(xT |zT )p(zt|z0, zT )p(z0)p(zT )
  q(zt|z0, zT , xt)q(z0|x0)q(zT |xT ) ]
  = Eqφ(z|x)[p(x0|z0)p(xt|zt)p(xT |zT )] + Eqφ(z|x)[log p(zt|z0, zT )
  q(zt|z0, zT , xt) ]
  + Eqφ(z|x)[log p(z0)
  q(z0|x0) ] + Eqφ(z|x)[log
  p(zT )
  q(zT |xT ) ]
  = Eqφ(z|x)[log p(x0|z0)p(xt|zt)p(xT |zT )] −DKL(q(zt|z0, zT , xt)‖p(zt|z0, zT ))−DKL(q(z0|x0)‖p(z0))−DKL(q(zT |xT )‖p(zT ))
  We assume the priors over z0 is 0-centered and zT is 1-centered, which is similar to our Brownian Bridge setup . The encoder qφ(z|x) is parameterized with the same architecture as our encoder. The decoder p(x|z) is a fine-tuned GPT2 model.
sections.heading: F DATASET INFORMATION
sections.text: For each dataset, text examples were filtered out if they did not fit within GPT2’s context length of 1024 tokens. We also added the token “ . ” for each setting to mark the end of a sentence. This was done for indexing purposes, eg. when aligning the latent embeddings.
  Wikisection (Arnold et al., 2019) includes Wikipedia articles on cities split by sections. We adapt this dataset such that each article contains four ordered sections (abstract, history, geography, demographics) marked with section id tokens: Each article is represented as, “[ABSTRACT] text [HISTORY] text [GEOGRAPHY] text [DEMOGRAPHICS] text”. These section id tokens are added to the tokenizer.
  The training dataset contains 1420 articles. The section lengths have the following breakdown measured in the number of BPE tokens (GPT2 tokenizer):
  • Abstract: 75.8± 1.4 • History: 191.5± 3.7 • Geography: 83.9± 1.5 • Demographics: 342.6± 4.6
  The test dataset contains 431 articles. The section lengths have a similar breakdown:
  • Abstract: 73.5± 2.6 • History: 180.2± 6.2 • Geography: 85.2± 2.7 • Demographics: 332.5± 8.6
  The ordering metric used in Table 5 is 1 if all four section ids occur exactly once and come in the order as they are listed above.
  The length mismatch in % used in Table 3 is calculated with respect to the training set lengths.
  Wikihow (Koupaee & Wang, 2018) contains how-to articles organized by a title, method, and steps. Each article includes multiple methods for completing a multi-step procedural task such as “How to Register to Vote”. We scraped all the available English articles covering a wide range of topics following Koupaee & Wang (2018). We mark each with its own section id tokens: Each article is represented as “[TITLE] text [METHOD] text [STEP] 1 text [STEP] 2 text ...”
  The training dataset contains 1566 articles. The section lengths are,
  • Title: 10.4± 2.2 • Method: 8.7± 2.2 • Steps (total step length): 480.2± 231.5
  The test dataset contains 243 articles. The section lengths are,
  • Title: 10.7± 2.2 • Method: 8.6± 2.1 • Steps (total step length): 480.1± 224.0
  The ordering metric used in Table 5 is 1 if all the section ids appear in order, the TITLE and METHOD section ids are not repeated, and the step numbers come in order. It’s 0 otherwise.
  The deviation in section length measured in Table 4 is calculated with respect to the training set lengths. We check for whether the models are able to maintain the section lengths when it has to extrapolate. The most common failure mode is that the model generates incoherent text and allocates this text to the last section of what it’s generated thus far, resulting in the deviation in section lengths.
  Recipe NLG (Bień et al., 2020) contains recipes, each with a title, ingredients and set of instructions. A recipe is constructed as “[TITLE] text [INGREDIENTS] text [DIRECTIONS] text”.
  The training dataset contains 4000 recipes. The section lengths are,
  • Title: 9.7± 3.4 • Ingredients: 23.8± 4.5 • Directions (total step length): 62.0± 14.5
  The test dataset contains 1000 recipes. The section lengths are,
  • Title: 9.4± 3.0 • Ingredients: 24.1± 4.5 • Directions (total step length): 63.3± 13.7
  The ordering metric used in Table 5 is 1 if all the section ids appear exactly once and in order. It’s 0 otherwise.
  Taskmaster-2 (TM-2) (Byrne et al., 2019) which contains conversations on finding restaurants between an assistant and a user. The assistant’s turn is marked with an “[ASSISTANT]” tag, and the user’s turn is marked with a “[USER]” tag.
  The training dataset contains 2000 conversations. The section lengths are,
  • User: 11.8± 5.6 • Assistant: 18.0± 3.7
  The test dataset contains 1276 conversations. The section lengths are,
  • User: 11.7± 5.6
  • Assistant: 19.5± 3.0
  The ordering metric used in Table 5 does not apply here because the user and assistant don’t take turns in the dialog.
  The deviation in section length measured in Table 4 is calculated with respect to the training set lengths. We check for whether the models are able to maintain the utterance lengths between the user and assistant when it has to extrapolate. The most common failure mode is that the model generates incoherent text and allocates this text to the last section of what it’s generated thus far, resulting in the deviation in section lengths.
  TicketTalk (Byrne et al., 2021) which contains conversations on booking movie tickets between an assistant and a user. The assistant’s and user’s turned are similarly marked as in TM-2.
  The training dataset contains 2000 conversations. The section lengths are,
  • User: 11.8± 5.6
  • Assistant: 18.0± 3.7
  The test dataset contains 1276 conversations. The section lengths are,
  • User: 11.7± 5.6
  • Assistant: 19.5± 3.0
  The deviation in section length measured in Table 4 is calculated similarly to TM-2.
  The ordering metric used in Table 5 applies because the user and assistant take turns in the dialog. The ordering metric is 1 if the user and assistant take turns in the conversation. It’s 0 otherwise.
  ROC Stories (Mostafazadeh et al., 2016) is a short stories dataset. Each story contains 5 sentences. No additional tokens are added in this dataset. The training dataset contains 2000 stories, and the test dataset contains 1000 stories.
sections.heading: G PERPLEXITY AFTER FINE-TUNING
sections.text: Table 9 reports the final perplexity scores after fine-tuning GPT2 on the different domains with the methods. We fine-tune for 10 epochs and checkpoint the models every 1000 steps; we keep the model checkpoint that scores the lowest PPL on a held-out validation set.
sections.heading: H GENERATION EXAMPLES
sections.text: We provide example outputs from the different models on text infilling. We denote the prefix sentences as “LEFT CONTEXT”, the suffix sentences as “RIGHT SENTENCES”, the generated output from the models as “GENERATED” and the ground truth infill sentence as “GT”.
  H.1 TEXT INFILLING
  LM See Table 10.
  ILM See Table 11.
  Time Control See Table 12.
  Table 13: Example of text infilling with Time Control (d=16) on ROCStories.
  LEFT CONTEXT: Tom was jealous of his brother. His brother was older and stronger. His brother went wherever he wanted. GENERATED: Tom’s brother was very jealous of his brother. GT: One day his brother was grounded for staying out too late. RIGHT CONTEXT: This made Tom really happy.
  LEFT CONTEXT: Jackie was 11 and had to get braces. She was worried about what her friends would think. She tried to hide them when she first got them. GENERATED: But she was too embarrassed to tell them . GT: Eventually her friends saw them and she was embarrassed. RIGHT CONTEXT: Her friends noticed she was embarrassed and decided to comfort her.
  LEFT CONTEXT: Sally was going to surprise the office with a cake. Sally felt that a cake would be a good way to make them smile. She went to the supermarket to pick up the cake. GENERATED: She bought the cake and was very happy with it. GT: At the office she gathered the employees around the conference table. RIGHT CONTEXT: She then brought out a cake and they all felt better.
  LEFT CONTEXT: Lars was playing XBOX. His controller stopped working during a game. Lars didn’t have a car so he had to walk all the way to the store. The store was being remodeled when he got there so he went to another. GENERATED: Video games were all over the floor. GT: Lars wasn’t able to find a controller. RIGHT CONTEXT:
  LEFT CONTEXT: Emma had been working as a dishwasher. Her hands cracked and bled from the hot soapy water. Then her mom noticed and concocted a special salve for her. Emma used the salve every night before bed. GENERATED: Soon she was able to keep her hands clean and happy. GT: Her hands got better. RIGHT CONTEXT:
  LEFT CONTEXT: Jerome dribbled the ball quickly. Sam tried to grab it from him, but wasn’t fast enough. He chased Jerome down the court. GENERATED: He scored a point with a long shot . GT: Jerome pushed Sam backwards and threw the ball. RIGHT CONTEXT: Jerome scored points for his team.
  LEFT CONTEXT: Class started in 10 minutes and I had a math assignment due. My older friend volunteered to do it for me. Unfortunately, my teacher found out about the copying. GENERATED: She was very upset with me for doing the assignment. GT: She crumbled my paper and threw it in the trash. RIGHT CONTEXT: My teacher gave me an F for the assignment.
  Table 14: Example of text infilling with Time Control (d=32) on ROCStories.
  LEFT CONTEXT: I went to the park to play frisbee with my dog. GENERATED: We played all day . GT: I tossed the frisbee to my dog and he would catch it in his mouth. RIGHT CONTEXT: I accidentally threw the frisbee too far. The frisbee landed into the pond. But my dog went and got it.
  LEFT CONTEXT: I was tired of working at Walmart. The hours were bad. The store music was awful. GENERATED: I was very bored . GT: I handed my two weeks in to the manager. RIGHT CONTEXT: I then found another job and was happy.
  LEFT CONTEXT: Sam bought a new SUV. It was all wheel drive. He figured he would take it off road. He hit a few hard bumps and broke his suspension. GENERATED: Unfortunately he had to pay a lot of money for it. GT: Sheepishly, he brought it to the dealership for repair. RIGHT CONTEXT:
  LEFT CONTEXT: Missy got drunk and went to get a tattoo. She decided to get a tattoo on her forehead. The next day, Missy was horrified at what she had done. GENERATED: Her tattoo was on her forehead! GT: Missy scraped up her money to pay for a tattoo removal procedure. RIGHT CONTEXT: After much wasted money, the tattoo was gone.
  LEFT CONTEXT: Jake was going on a road trip to see his family. He Got in the car and drove. GENERATED: Jake’s family was driving down the road . GT: The car’s tires exploded due to too much air. RIGHT CONTEXT: Jake hitchhiked for 30 miles. When Jake got to his family he was happy his trip was over.
  LEFT CONTEXT: Bob decided to start a business. GENERATED: However, he did not know the market at all . GT: He opened up a grocery store and was doing very well. RIGHT CONTEXT: After a year, his profits dropped and he had to declare bankruptcy. Bob was sad to see his business fail. Bob worked hard and reopened his business.
  LEFT CONTEXT: I hadn’t seen my girlfriend in a while. She got a new job so it’s hard to talk. The job takes up all of her time. GENERATED: I had to ask her to go out with me . GT: Finally she called me to hang out. RIGHT CONTEXT: I was really happy to see her and we made plans.
  Variational Auto-Encoder See Table 15.
  Implicit Dynamics See Table 16.
  H.2 FORCED LONG TEXT GENERATION
  We include examples of GPT2 long text generation in Figures 3 and 4.
sections.heading: I HUMAN EXPERIMENTAL SETUP
sections.text: We use Mechanical Turk to collect human evaluation for long text generation and text infilling models. Examples from all the systems are selected randomly and each example is evaluated by 10 annotators.
  I.1 LONG TEXT GENERATION EVALUATION
  The annotators are provided with a dialogue between a movie assistant and a user generated by GPT and Time Control, where the goal of the assistant is to help the user to book a movie ticket. We show the last few generated interactions between the assistant and the user to the annotators, and ask them to assess whether the provided snippet could be a reasonable ending for such a conversation. We use a 5-point Likert scale where 1 corresponds to a “Very Unreasonable Ending” and 5 corresponds to a “Very Reasonable Ending”.
  I.2 TEXT INFILLING EVALUATION
  Given the preceding and following context, we ask annotators to evaluate whether the generated fill-in sentence is a reasonable completion. We evaluate generated fill-in sentence by ILM, LM and Time Control on a scale of 1-5.
sections.heading: J EXAMPLE TRAJECTORIES OF THE LEARNED LATENT SPACE
sections.text: See Figures 7-10 for example latent trajectories over coherent vs. incoherent (randomly scrambled) held-out Wikisection documents. These trajectories show the recovered latent structure by Time Control and the three ablations from our work. Figure 7 are latent trajectories from Time Control. Figure 8 are latent trajectories from Implicit Dynamics. Figure 9 are latent trajectories from Brownian motion. Figure 10 are latent trajectories from Variational Auto-Encoder.
sections.heading: K WHY DOES VARIATIONAL AUTO-ENCODER PERFORM WELL ON THE RECIPE DOMAIN?
sections.text: In Table 5, VAE does much better than TC. Here we investigate potential reasons why.
  We looked into comparing the latent structure recovered by the VAE baseline and TC; see Figure 11. What we found is the Time Control is best at extracting time over the course of the document (ie. its latents recover a Bridge process correlated with time), whereas the VAE doesn’t elicit strong temporal structure. We hypothesize that temporal structure is not all you need to succeed in the Recipe domain.
sections.heading: L ADDITIONAL EXPERIMENTS
sections.text: On discourse coherence, we ran ALBERT on k = 1 since ALBERT is trained on this setting. We found that it got slightly above random chance, unlike the most of the baselines we compare against. The discourse accuracy is 50.1 ± 9.0 on Wikisection, 55.8 ± 4.0 on TicketTalk and 60.8 ± 2.8 on TM-2.
sections.heading: M PREVIOUS EXPLORATORY THREADS
sections.text: M.1 SUCCESS AND FAILURE MODES IN FINE-TUNING GPT2
  We found that fine-tuned GPT2 was able to replicate certain aspects of a document corpus, such as section header ordering ( 92% accurate). However, when going from document-level to sectionlevel statistics, we noticed that GPT-2 seemed to be either undershooting or overshooting the section lengths. These results are reported in Table 3.
  M.2 INVESTIGATING THE IMPORTANCE OF DISTANCES BETWEEN SAMPLED SENTENCES
  In earlier iterations of the work, we did explore learning with pairwise contrasts with fixed tdistances, e.g. distances of length 1, 5, and 10 sentences. We observed that the resulting latent trajectories elicited different fits to Brownian bridge dynamics, and the quality in fit varied in t across domains; we’ve include some examples in Figure 12 on Wikisection.
  BERTScore ROUGE BLEURT Method Latent dim d Precision Recall F1 1-F1 2-F1 L-F1