We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only 50% of the available training data and surpassing BLEU, ROUGE and METEOR with only 40 labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel paraphrases from a single demonstration, which we use to confirm our hypothesis that it learns abstract, generalized paraphrase representations. Representing the relationship between two pieces of text, be it through a simple algorithm or a deep neural network, has a long history and diverse use-cases that include the evaluation of text generation models (Wiseman et al., 2017; Van Der Lee et al., 2019) and the clinical evaluation of human speech (Johnson et al., 2003; Weintraub et al., 2018). One of the earliest examples of such a representation is the Levenshtein distance (Levenshtein, 1966), which describes the number of character-level edits required to transform one piece of text into another. This metric now forms part of a wider fam-ily of edit-distance-based metrics that includes the word error rate (WER) and the translation error rate (TER) (Och, 2003). Other algorithms, such as ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and the widely used BLEU metric (Pa-pineni et al., 2002), perform exact or approximate n -gram matching between the two texts. texts, which benefits from a deep prior understanding of the semantic and syntactic symmetries of language (Novikova et al., 2017). For example, pairs like she was no ordinary burglar and she was an ordinary burglar are close in edit-distance-space but semantically disparate. The goal of an automatic text evaluation metric is typically to be a good proxy for human judgements, which is clearly task-dependent. More recently, neural approaches have begun to close the gap between automatic and human judgements of semantic text similarity using Transformer-based language models such as BERT (Zhang et al., 2019a; Sellam et al., 2020). They aim to leverage the transferable knowledge gained by the model during pretraining on large text corpora. The relationship between two texts is similarly modelled, albeit implicitly, by sequence-to-sequence models such as BART (Lewis et al., 2019) and T5 (Raffel et al., 2019). We consider paraphrase evaluation and paraphrase generation to be two instances of paraphrase representation learning . Linguistically, a paraphrase is a restatement that preserves essential meaning, with arbitrary levels of literality, fidelity and completeness. In practice, what qualifies as a good paraphrase is context-specific. One motivation for considering paraphrase evaluation as a representation learning problem is the varied nature of paraphrase evaluation tasks, which may have an emphasis on semantic equivalence (e.g. PAWS (Zhang et al., 2019b) and MRPC (Dolan and Brockett, 2005)), logical entailment versus contradiction (e.g. MultiNLI (Williams et al., 2017) and SNLI (Bowman et al., 2015)), and the acceptability of the generated text (e.g. the WMT Metrics Shared Task (Bojar et al., 2017)). Considering even broader applications such as clinical speech analysis further motivates learning generalized paraphrase representations. In this paper, we introduce ParaBLEU, a paraphrase representation learning model that predicts a 4052 conditioning factor for sequence-to-sequence paraphrase generation as one of its pretraining objectives, inspired by style transfer in text-to-speech (Skerry-Ryan et al., 2018) and text generation systems (Yang et al., 2018; Lample et al., 2018). ParaBLEU addresses the primary issue with neural paraphrase evaluation models to date: the selection of a sufficiently generalized pretraining objective that primes the model for strong performance on downstream paraphrase evaluation tasks when data is scarce. Previous state-of-the-art neural models have either used a broad multi-task learning approach or eschewed additional pretraining altogether. The former case may encourage the model to learn the biases of inferior or inappropriate metrics, while the latter leaves room for optimization. Non-neural models, such as BLEU, TER, ROUGE and BERTScore (Zhang et al., 2019a), benefit from requiring no training data and thereby avoid do-main shift issues. They cannot, however, learn to exploit task-specific nuances of what defines good' paraphrasing. We evaluate ParaBLEU's ability to predict human judgements of paraphrases using the English subset of the 2017 WMT Metrics Shared Task. A useful neural text similarity metric should be robust to data scarcity, so we assess performance as a function of the fine-tuning dataset size. Finally, using the ParaBLEU pretraining model as a paraphrase generation system, we explore our hypothesis that the model reasons in high-level paraphrastic concepts rather than low-level edits through an explainability study, and demonstrate that ParaBLEU can operate as a conditional paraphrase generation model. In this section, we describe and justify the set of inductive biases we build into ParaBLEU, along with a description of the model architecture and pretraining/fine-tuning strategy. We consider a reference text x and a candidate text  x . We wish to learn a function f : f ( x,  x )  y , where y  RN is a singleor multi-dimensional paraphrase representation, which could be a scalar score. Our approach begins by decomposing paraphrase representation learning into three overlapping factors: Building a representation of high-level syntactic and semantic differences between x and  x , contrasted with the low-level pseudo-syntactic/-semantic operations considered by edit-distance-based and n -gram based metrics. 2. Candidate acceptability judgement: Evaluating the grammaticality, coherence and naturalness of  x in isolation. Perplexity (Jelinek et al., 1977) with respect to a given language model is one proxy for this. 3. Semantic equivalence: Assessing whether x and  x convey the same essential meaning precisely, as opposed to merely being semantically similar. This is related to entailment classification tasks and, more broadly, the interaction between language and formal logic. Using pretrained language models: All three factors require a general understanding of the semantic and syntactic structures of language, making transfer learning from powerful pretrained language models such as BERT (Devlin et al., 2018) appealing. Non-local attention as bitext alignment: Factors (1) and (3) require performing context-aware matching' between x and  x . This is similar to the statistical method of bitext alignment (Tiedemann, 2011). Attention mechanisms within a Transformer (Vaswani et al., 2017) are an obvious candidate for learnable context-aware matching, which has precedent in paraphrasing tasks and the next-sentence-prediction objective of the original BERT pretraining. If the tokens of x and  x are concatenated into one long input sequence, local attention mechanisms, such as those used in T5, may be suboptimal for longer text-pairs. Bottlenecked conditional generation objective: A key insight is that a strong factor (1) representation z  RM where h : h ( x,  x )  z is one that can condition the sampling of  x from x through some generative model g : g ( x | z )   x . One trivial solution to this is h ( x,  x ) =  x . To avoid this case, we introduce a bottleneck on z such that 4053 it is advantageous for the model to learn to represent high-level abstractions, which are cheaper than copying  x through the bottleneck. It is likely advantageous to use a pretrained sequence-to-sequence language model, which can already reason in linguistic concepts. Entailment classification objective: Factor (3) is similar to the classification of whether x logically entails  x . There are a number of sentence-pair datasets with entailment labels that could be used to construct this loss; see Table 4. 2.2 ParaBLEU Inspired by style transfer in text-to-speech (Skerry-Ryan et al., 2018) and text generation systems (Yang et al., 2018; Lample et al., 2018), we propose the architecture shown in Figure 1. The grey box indicates the Transformer encoder we wish to pretrain, which we refer to as the edit encoder'. Masked language modelling objective: Factor (2) can be addressed by an MLM objective, which alone is sufficient for a neural network to learn a language model (Devlin et al., 2018). Performing masked language modelling on a reference-candidate pair also encourages the network to use x to help unmask  x and vice versa, strengthening the alignment bias useful for factors (1) and (2). Factorization of the task leads to three complementary objectives: a cross-entropy masked language modelling loss LMLM (Devlin et al., 2018), a cross-entropy autoregressive causal language modelling loss LAR (Radford et al., 2018) and a binary cross-entropy entailment classification loss LCLS . An additional sequence-to-sequence Transformer model is used during pretraining to provide a learning signal. The proposed bottleneck lies within the feedforward network module (see Figure 1), implemented by restricting the hidden dimension to 64 (down from 768 or 1 , 024 in the cases of ParaBLEU base and ParaBLEU large respectively) before projecting back up to the dimension of the BART decoder. The full pretraining loss is given by: L pre := LAR +   LMLM +   LCLS , (1) where  and  are tunable hyperparameters. the sequence-to-sequence model is discarded and the edit encoder is fine-tuned using a linear projection on top of the pooled output, projecting the pooled output down to a single dimension that constitutes the predicted score. Throughout this work, our pooling layers simply take the beginning-of-sequence token. An MSE loss LMSE is used during fine-tuning. Our architecture places restrictions on valid combinations of pretrained models. We found in practice that using an encoder-only pretrained language model to initialize the edit encoder, and a sequence-to-sequence pretrained language model to initialize the sequence-to-sequence model, works best. This is likely because encoder-only models are encouraged to encode strong representations at the final layer, and these representations have already been directly pretrained with an MLM objective. For technical ease we require that the models use the same tokenizer, and that the pretrained checkpoints are available through the HuggingFace transformers library (Wolf et al., 2019). In this paper, we consider the combination RoBERTa (Liu et al., 2019) + BART, but we note that both multilingual (XLM-R (Conneau et al., 2019) + mBART (Liu et al., 2020)) and long (Longformer + Longformer-Encoder-Decoder (LED) (Beltagy et al., 2020)) combinations exist. We consider both base and large variants, which correspond to RoBERTa base and RoBERTa large . In both cases, we use a BART base checkpoint. Evaluation metrics BERTScore (Zhang et al., 2019a), a non-learned neural metric, uses a matching algorithm on top of contextualized neural word embeddings, similar to n -gram matching approaches. MoverScore (Zhao et al., 2019) is similar to BERTScore but uses an optimal transport algorithm. BLEU, ROUGE, METEOR and chrF++ (Popovic, 2017) are widely used n -gram-based methods, working at the word, subword or character level. TER is an edit-distance-based metric, similar to WER. BLEURT (Sellam et al., 2020) is a neural automatic evaluation metric for text generation. Starting from a pretrained BERT model, it is further pretrained to predict a number of pre-existing metrics, such as BLEU, ROUGE and BERTScore. ParaBLEU, by contrast, does not use pre-existing metrics as training objectives, instead using generative conditioning as a more 4054 Figure 1: general signal for paraphrase representation learning. COMET (Rei et al., 2020) is a framework for training multilingual machine translation (MT) evaluation models where parameters in the regression or ranking layers are optimized using human judgements scores with either an MSE objective or triplet objective respectively. PRISM (Thomp-son and Post, 2020) similar to ParaBLEU formulates evaluation as a paraphrasing task. However it treats paraphrasing as zero-shot translation using a multilingual neural MT model as a paraphraser. BARTScore (Yuan et al., 2021) calculates the log-likelihood of the candidate text conditioned upon the reference text from BART (Lewis et al., 2019), a pretrained sequence-to-sequence model. Paraphrase generation There is a wealth of recent literature on controllable paraphrase generation and linguistic style transfer (Yang et al., 2018; Zhao et al., 2018; Jin et al., 2020), which aims to extract the style of a piece of text and map it onto another piece of text without changing its semantic meaning. T5 leverages a huge text corpus as pretraining for conditional generation using com-mands' encoded as text, which includes paraphrastic tasks such as summarization. FSET (Kazemne-jad et al., 2020) is a retrieval-based paraphrase generation system in which a sentence z is paraphrased by first locating a similar reference sentence from a large bank of reference/candidate pairs, then extracting and replaying similar low-level edits on z . Common to ParaBLEU and FSET is the use of a Transformer for paraphrase style transfer, with differing architectural details. However, FSET is designed to transpose low-level edits and so requires lexically similar examples; whereas ParaBLEU is explicitly designed to learn high-level, reference-invariant paraphrase representations using a factorized objective. The musical style Transformer autoencoder (Choi et al., 2020) uses a similar Transformer-based style transfer architecture to conditionally generate new music in controllable styles. Other examples in text-to-speech systems perform style transfer by encoding the prosody of a source sentence into a bottlenecked reference embedding (Skerry-Ryan et al., 2018) or disentangled style tokens (Wang et al., 2018b). STRAP (Kr-ishna et al., 2020) generates paraphrases in controllable styles by mixing and matching multiple style-specific fine-tuned GPT-2 models. REAP (Goyal and Durrett, 2020) uses a Transformer to generate syntactically diverse paraphrases by including an additional position embedding representing the syntactic tree. DNPG (Li et al., 2019) is a paraphrase generation system that uses a cascade of Transformer encoders/decoders to control whether paraphrasing is sentential/phrasal. In this section, we describe the pretraining and fine-tuning datasets we use in our studies. The WMT Metrics Shared Task is an annual benchmark for automated evaluation metrics for translation systems, where the goal is to predict average human ratings comparing the machine-translated candidate  x with human-translated reference x , both of which have been translated from the same source sentence. We use an identical setup to (Sellam et al., 2020) and (Zhang et al., 2019a), where we use the subset of data for which the candidate and reference are in English, which we will refer to as the to-English subset. The source, which is unused, can be in any non-English language, the set of which varies from year-to-year. We produce results for the WMT Metrics Shared Task 2017 (WMT17), training on the to-English subsets of WMT15 and WMT16. The test set contains 4 , 132 examples and the training set 5 , 360 examples. The distributions of example length in tokens is shown in Figure 2. The WMT data is prepared using the WMT preparation code in the BLEURT repository 1 . The decision to test only on the WMT17 dataset is deliberate. Results from previous state-of-the-art papers (Sellam et al., 2020; Zhang et al., 2019a) demonstrate issues with WMT18 and later datasets: the noise in the test set is high and differentiation between different methods becomes so suppressed for later years that the benchmark becomes uninteresting. This issue is noted in both the BLEURT paper and by the organizers of the 2018 WMT Metrics Shared Task 2 . We report the agreement between the metric and the human scores using two related correlation coefficients: absolute Kendall |  | and absolute Pearson | r | , the latter of which was the official metric of the 2017 task. In our summary results in the main paper, we average these metrics across all source languages but not over reference/candidate language. Full results are provided in Appendix E. 3.2 ParaCorpus In addition to our design choices, we also encourage a robust and generalizable pretraining by using a dataset that covers a variety of styles and lengths. We collate a number of paraphrase datasets to create a single pretraining dataset we call ParaCorpus. The composition of the dataset is shown in Table 4, with a total of  5 . 1 m examples. All examples have reference and candidate texts and around one third additionally have binary entailment labels. Where the source dataset included three-way labels entailment'/contradiction'/neutral', entailment' was mapped to 1 and the others to 0 . A subset of ParaNMT-50M (Wieting and Gimpel, 2017), which includes noisier, speech-like examples, was included to add additional stylistic diversity to the dataset, and to increase the population of the dataset with combined token lengths above 128 , which we hypothesize will make the model more robust to the longer examples seen in the WMT datasets. Token lengths are shown in Figure 2. 4 Experiments In this section, we present results on WMT17, benchmarked against the current state-of-the-art approach, along with widely used neural, n -gram and edit-distance-based metrics. We study ParaBLEU performance as a function of number of pretraining 1 https://github.com/google-research/ bleurt 2 https://www.statmt.org/wmt18/ metrics-task.html 4056 Table 1: Summary results for WMT17. steps and the size of the fine-tuning dataset. Finally, we perform ablations to test the impact of the inductive biases and resultant architectural decisions described in Section 2. We report results for both ParaBLEU base , based on RoBERTa base ( 12 layers, 768 hidden units, 12 heads), and our default model ParaBLEU large , based on RoBERTa large ( 24 layers, 1 , 024 hidden units, 16 heads). Both models are trained near-identically for 4 epochs on ParaCorpus. Further pretraining details can be found in Appendix A. For fine-tuning, we use a batch size of 32 , a learning rate of 1 e5 and train for 40 k steps, with a validation set size of 10% (unless otherwise stated). No reference texts are shared between the train and validation sets, following (Sellam et al., 2020). Pretraining ParaBLEU large takes  10 h on a 16 A100 GPU machine. Fine-tuning takes  8 h on a single A100 GPU machine. ParaBLEU results on WMT17 are given in Table 1, along with a number of baselines described in Section 2.3). ParaBLEU large achieves new state-of-the-art results on WMT17, exceeding the previous state-of-the-art approach, BLEURT, on both correlation metrics. We note that non-neural metrics perform the worst, of which the character-level n -gram-matching algorithm chrF++ performs the best. Non-learned neural metrics (BERTScore and Mover-Figure 3: Performance of ParaBLEU large on WMT17 as a function of number of pretraining steps (top) and the fine-tuning dataset size (bottom). Note that the Pearson r results (blue) use the left y -axis, whereas Kendall  (orange) uses the right y -axis. Score) tend to perform better, and learned neural metrics (BLEURT and ParaBLEU) perform the best. BLEU, the most widely used metric, has the poorest correlation with human judgements. This is consistent with results seen previously in the literature (Zhang et al., 2019a; Sellam et al., 2020). The significant drop in performance from ParaBLEU large to ParaBLEU base highlights the benefit of larger, more expressive pretrained language models. Figure 3 probes performance as a function of number of pretraining steps and the size of the fine-tuning dataset for ParaBLEU large . As expected, pretraining for longer increases downstream task performance. However, we note that 40 k steps, approximately 4 epochs of ParaCorpus, does not yet reach diminishing returns on WMT17 performance. We therefore recommend pretraining for significantly longer. Both BERT and RoBERTa are pretrained for 40 epochs (Liu et al., 2019; Lan et al., 2019); the T5 authors ablate their dataset size at a fixed number of steps and conclude that performance does not significantly degrade up to and including 64 epochs (Raffel et al., 2019); conversely, 4057 Table 2: Ablation results on WMT17. the BLEURT authors see diminishing returns on downstream task performance after 2 pretraining epochs (Sellam et al., 2020). For the fine-tuning dataset size study, we consistently use a validation set size of 25% to facilitate the small-data results. Despite the training set (the to-English subsets of WMT15 and WMT16) forming a relatively small dataset, ParaBLEU large trained on 50% of the available data ( 2 , 010 training examples, 670 validation examples) still beats the previous state-of-the-art, BLEURT, yielding a Pearson correlation of 0 . 823 . The impact of reducing the train size from 100% ( 4 , 020 training examples, 1 , 340 validation examples) to 25% ( 1 , 005 training examples, 335 validation examples) has a relatively small effect on performance, reducing Pearson r from 0 . 832 to 0 . 795 . With a dataset size of only 1% ( 40 training examples, 14 validation examples), ParaBLEU large achieves a Pearson r of 0 . 571 , still correlating significantly more strongly with human judgements than BLEU, TER, ROUGE, METEOR and MoverScore. We attribute this to the suitability of the generalized pretraining objective for priming the model for paraphrase evaluation tasks. To more directly test the hypotheses in Section 2.1, we perform ablations in which we remove each component of the factorized objective in turn. The results of this are shown in Table 2. Each part of the objective is associated with an increase in downstream task performance. The most significant degradation comes from removing the MLM loss. Possible reasons for this include: the MLM loss' contribution to candidate acceptability judgement are crucial; the MLM loss acts as a regularizer, encouraging the edit encoder to represent paraphrases in linguistic concepts rather than low-level edits; and the MLM loss further encourages bitext alignment behaviour, as described in Section 2.1. As our final study, we exploit the generative nature of the pretraining architecture to test our claim that the edit encoder reasons in high-level paraphrastic concepts rather than low-level edits. To do this, we diverge from the pretraining setup, in which the same reference text is passed to both the edit encoder and the sequence-to-sequence model, by passing a different, unseen reference to the sequence-to-sequence model. Akin to (Brown et al., 2020; Gao et al., 2020), the hope is that the demon-stration paraphrase' acts as a conditioning factor for paraphrasing the unseen sentence in a similar way. If the model is reasoning in low-level edits or otherwise cheating', we expect to see:  Thematic/word leakage from the encoder candidate to the generated candidate, caused by the candidate being autoencoded. This is the undesirable behaviour we sought to address using a bottleneck. Ungrammatical or otherwise unacceptable output with made-up words and/or bad word order, caused by the encoding of low-level edits scrambling the generator reference tokens. If the model is reasoning in high-level paraphrastic concepts, we expect to see:  Consistently grammatical, acceptable output. The flavour of the paraphrase mirroring the conditioning, e.g. the altering of a linguistic style, mood or tense. We generate text using beam-search (Medress et al., 1977). We sample references at random from the MRPC dataset. The demonstration candidate is a hand-crafted paraphrase of the demonstration reference that embodies a pre-specified paraphrase type. We report the predicted entailment score of the demonstration reference and candidate, along with the candidate generated by the model. A summarized, random subset of generation results is shown in Appendix C. We include two sets of results for each paraphrastic type (e.g. 4058 negative'): one where the demonstration refer-ence/candidate differ in this concept, and one where both embody the concept. Since we wish to encode the difference between the demonstration reference/candidate texts, the desired behaviour when the demonstration pair is identical is no change. If this is not the case, it is likely that the edit encoder is just autoencoding the candidate using high-level linguistic concepts, similar to linguistic style transfer. Further randomly chosen examples are given in Appendix F. The results present a strong case that the encoder is representing high-level paraphrastic concepts. It is able to successfully identify changes in mood, style and tense between the demonstration reference and candidate, and transpose them onto the unseen reference to make a largely grammatical and appropriately paraphrased sentence. We do not see significant leakage of concepts, words or styles between the demonstration candidate and the generated candidate, instead the expected transfer of paraphrase style. Limitations of this work include the relatively small set of baselines used; there is an ever-increasing number of text similarity metrics and so only a subset is presented here. As demonstrated in Section 4.1, it seems likely that the performance is currently limited by pretraining time and so we have not yet probed the ceiling performance of this method. Swapping out the edit encoder and encoder-decoder for current state-of-the-art models like DeBERTa (He et al., 2020) may offer further performance boosts. Expanding this work to predicting on datasets beyond the 2017 WMT Metrics Shared Task will probe the generalizability of the techniques in this paper. The application of ParaBLEU for paraphrase generation has not been quantitatively explored. Although the results presented in Section 5 are chosen at random, the analysis is qualitative. More rigorous methods for evaluating the quality of the paraphrase generation are left for future work. In this paper, we introduced ParaBLEU, a paraphrase representation learning model and associated paraphrase evaluation metric. We demonstrated that the metric yields state-of-the-art correlation with human paraphrase judgements and is robust to data scarcity. We motivated its pretraining strategy through a set of inductive biases, which we tested through ablation studies. Finally, we reframed the pretraining as a one-shot paraphrase generation model and gathered evidence that ParaBLEU represents meaningful paraphrastic information.
abstain abstain abstain result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain objective method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method method method method other other other other other other other other other other other other other other other other other other other other other other method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain objective objective abstain