Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks. 1 1 Introduction Coherence is a property of a well-written text that makes it different from a random set of sentences: sentences in a coherent text are connected in systematic ways such that each sentence follows naturally from previous ones and leads into the following ones (Halliday and Hasan, 1976; Grosz and Sidner, 1986). Coherence models (Barzilay and Lapata, 2005) that can distinguish a coherent text from incoherent ones have a wide range of applications in language generation, summarization, and coherence assessment tasks such as essay scoring and sentence ordering. 1 Our code and data are available at https://ntunlpsg.github.io/project/coherence-paradigm With recent advancements in neural methods, claims of fluency in summarization (Liu et al., 2017; Celikyilmaz et al., 2018), language modeling (Radford et al., 2019; Brown et al., 2020), response generation (Zhang et al., 2020; Hosseini-Asl et al., 2020) and human parity in machine translation (Hassan et al., 2018) have led to calls for finer-grained discourse-level evaluations (Lubli et al., 2018; Sharma et al., 2019; Popel et al., 2020), since traditional metrics such as BLEU and ROUGE are unable to measure text quality and readability (Paulus et al., 2018; Reiter, 2018). Coherence models that can evaluate machine-generated text have become the need of the hour. A majority of coherence models proposed optimize their learning objectives on the permuted document task using the Penn Treebank (WSJ) corpus. An original article is considered a positive' sample of a coherent document, while a permutation of its sentences is considered a negative' or incoherent sample (see Appendix A.1 for an ex-ample). Models are usually trained in a pairwise ranking fashion to distinguish the two. The basic entity-grid model proposed by Barzilay and Lapata (2005, 2008) was extended to incorporate entity-specific features (Elsner and Char-niak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Lin et al., 2011; Feng et al., 2014). Their neural extensions have also been proposed (Nguyen and Joty, 2017; Mohiuddin et al., 2018). More recent state-of-the-art models like the Transferable Neural model (Xu et al., 2019) consider coherence at a local level by training a forward and backward model only on adjacent sentences, in addition to generative pre-training of the sentence encoders. The Unified Coherence model (Moon et al., 2019) uses bi-linear layer and lightweight convolution-pooling in a Siamese framework to capture discourse relations and topic structures, along with an explicit language model loss to capture syntactic patterns. Mohiuddin et al. (2021) recently tested these state-of-the-art models by conducting coherence evaluations on the WSJ permuted document task, machine translation, summarization and next utterance ranking tasks. They found that while models performed well on the permuted document task, when tested off-the-shelf, models generalized poorly to downstream evaluation tasks. They call for more comprehensive evaluations of coherence models. Pishdad et al. (2020) also reached a similar conclusion. They retrained several neural coherence models for tasks analogous to coherence modeling such as detecting connective substitution and topic switching. They found that performance on the permuted document task is only partially indicative of coherence modeling capabilities. In light of these recent findings, our aim is to propose a coherence model that generalizes well to downstream tasks. We train our model purely through self-supervision , without tailoring the model architecture specifically to the permuted document task or any other form of supervision. Li and Jurafsky (2017) point out that coherence models are exposed to a limited number of incoherent samples in the pairwise setup, since only a small sample of all possible incoherent permutations of a document are used to train models. Learning with more negatives can better maximize the mutual information between their representations (van den Oord et al., 2018). By using a contrastive learning (Gutmann and Hyvrinen, 2010) setup, where each positive' document is compared with multiple negative' documents, we increase the proportion of negative samples that the model is exposed to, and show that the coherence model shows significant improvements in performance. Wu et al. (2020) show that the difficulty of the negative samples used for contrastive training can strongly influence model success for visual representation learning. Guided by this principle, we train the model with automatically mined hard negatives, coupled with a large global negative queue encoded by a momentum encoder (He et al., 2019). In summary, our contributions are:  A neural coherence model trained purely through well-designed self-supervision tasks that generalizes well to downstream applications. Evaluation on multiple independent test sets that are more indicative of real-world performance of the coherence model. Empirical results demonstrating that increase in the density and quality of negative samples leads to better generalization for coherence models. To ensure that our coherence model is useful for evaluation in downstream applications, we use a selection of task-independent test sets that cover a variety of domains and genres, including machine generated text from summarization systems and language models. Following Pishdad et al. (2020), we also evaluate the models on a commonsense reasoning narrative dataset. We train (and validate) the coherence models on standard WSJ data, while using the rest as independent test sets to indicate the generalizability of the trained models. All evaluations on downstream tasks are conducted in a pairwise setting to enable a fair comparison. 2.1 Training Data  WSJ The Wall Street Journal (WSJ) corpus consists of news articles divided into 1240/138/1053 documents for train-ing/development/testing in the standard setup. We exclude documents with < 4 sentences and truncate them to a maximum length of 600 tokens. To maximally utilize documents which are otherwise truncated due to GPU memory constraints, we partition documents with 20+ sentences into blocks of 10 sentences and consider each block as a separate positive document. This increases the number of coherent documents' that we can use to generate a larger training set. Moon et al. (2019) use 20 permutations of a document for training; since their setup is pairwise, it means the original positive document is repeated 20x. We regenerate the permuted documents similarly, sampling a larger set of permutations for our contrastive learning setup. 2 This gives us 46 , 522 instances of positive and corresponding negative documents for training and 4 , 522 instances for development. We use the original pairwise test set used by Moon et al. (2019) with 20 , 411 pairs for testing. SUMMEVAL Fabbri et al. (2020) conduct a manual coherence evaluation of the summaries generated by 16 different summarization systems for 2 We ensure that the generated permuted documents are not repeated. For example, our contrastive learning setup requires 5 negative samples per instance; because each positive document appears 20 times in the original dataset, 100 unique permutations would be generated and divided accordingly. 100 source articles based on the CNN/DailyMail (Hermann et al., 2015) dataset. Likert-style coherence ratings from 3 expert annotators are available for each summarized text. We adapt this to the pairwise setting by creating pairs of summaries from every system for each unique source article. The summary with the higher average coherence rating is designated as the positive document, while the summary with the lower rating is the negative document for that pair. This results in (cid:0) 162 (cid:1)  100 = 12 , 000 pairs for evaluation. LMVLM To cover a wider variety of machine generated text, we generated texts from various language models using prompts taken from the validation and test sets of the WritingPrompts dataset (Fan et al., 2018). Four language models were chosen for this purpose: GPT2-Small, GPT2-XL, CTRL and GPT3. The continuations produced by these models for each prompt were truncated at approximately 150 tokens and paired together. Using these texts, we conducted a user study on Amazon Mechanical Turk. Workers were instructed about the concept of coherence and shown examples of coherent and incoherent texts. Given the prompt, they were asked to choose the more coherent text out of two given language model outputs; they were also given an option to choose neither in case the texts were equally coherent/incoherent (see Appendix A.3 for more details such as the study interface). After removing the samples with low agreements and ties, a total of 1 , 046 pairs with judgments from 3 annotators each were collected. The Krippendorff's alpha coefficient (Krippendorff, 2011) between the annotators was 0.84 . We calculate the agreements of the coherence model ranking with these judgments, designated LMVLM. INSTED Shen et al. (2021) propose a sentence intrusion detection task in order to test the coherence modeling capabilities of pre-trained language models. Incoherent documents are created by substituting a sentence from a document with another sentence from a different document, ensuring that the replacement sentence is similar to the original document to make the task sufficiently hard. We adapt their task to the pairwise setting by pairing the original coherent and the corrupted incoherent document, giving us 7 , 168 instances from their CNN test set (INSTED-CNN) and 3 , 666 instances from their Wikipedia test set (INSTED-WIKI ) for evaluation. Shen et al. (2021) also create a handcrafted linguistic probe test set, where incoherence is manually inserted based on a range of linguistic phenomena; we use this test set for analysis (4). STORYCLOZE The STORYCLOZE dataset (created from ROCSTORIES (Sharma et al., 2018)) consists of a short narrative-style text with two possible endings, one of which is implausible. The test set labels are not public so we use the validation set. We designate the text with the correct ending as the positive document and the text with the incorrect ending as the negative document, resulting in a total of 1 , 571 pairs for evaluation. Previous work on coherence modeling proposed elaborate architectures to capture various aspects of coherence (see 1). However, our key hypothesis is that large-scale pre-trained models are expressive enough to model coherence given the right self-supervision. Effective bi-directional encoding through large Transformer networks (Vaswani et al., 2017) can consider longer language context, while language modeling objectives enforce syntactic and local coherence patterns in the model. In our work, we adopt XLNet (Yang et al., 2019) as the backbone model. It is trained using a permuted language modeling objective, in which the expected log-likelihood of a sequence with respect to all permutations of the factorization order is maximized. This allows the modeling of bi-directional context, while maintaining the auto-regressive property and avoiding the pretrain-finetune discrepancy. In addition, XLNet also incorporates segment recurrence (or memory) and the relative encoding scheme of Transformer-XL (Dai et al., 2019), which makes it effective in modeling longer text sequences. This makes it suitable for our purpose of coherence modeling. Given a document D with n sentences ( s 1 , s 2 , . . . , s n ) as input, our model uses the representations obtained through XLNet (parameter-ized by  ) to assign a coherence score to the model. Specifically, for each sentence s i with k tokens ( w 1 , w 2 . . . w k ) , XLNet maps each token w t to its vector representation v t  R d where d is the dimension of the embedding. In addition, the complete input D is also mapped to a document representation z  R d ( i.e., the representation of the [ CLS ] 6046 token). We simply add a linear layer to convert document representation z to obtain the final coherence score: f  ( D ) = w  z + b , where w and b are the weight and bias of the linear layer with  = { , w , b } being the entire parameter set of the model (see the upper part of Figure 1). Setup. Traditionally, coherence model training has been done in a pairwise ranking setup. In this setup, the model is trained to score the coherent or positive document higher than the incoherent or negative document, using a pairwise ranking loss (Collobert et al., 2011) defined as follows: L  = max (cid:0) 0 ,   f  ( D + ) + f  ( D  ) (cid:1) (1) where f  ( D + ) is the coherence score of the positive document, f  ( D  ) is the coherence score of the negative document and  is the margin. Baselines. We compare our models against all three versions of the L ocal C oherence D iscriminator or LCD model (Xu et al., 2019) 3 : ( i ) LCD-G, that uses GloVe (Pennington et al., 2014) representations, ( ii ) LCD-I, that uses InferSent (Conneau et al., 2017) representations, and ( iii ) LCD-L, that uses representations from an RNN-based language model trained on the training data. We also compare against the Un ified C oherence model or UNC (Moon et al., 2019) 4 , which is the previous SOTA on the WSJ permuted document task. Results from evaluation of existing coherence models by Pishdad et al. (2020) and Mohiuddin et al. (2021) indicate that UNC and LCD are the best-performing models (see Appendix A.4 for a full comparison). We retrain their models with our training data for comparison. In addition, to ascertain the contribution of the pre-trained XLNet embeddings, we train our pairwise model without fine-tuning the representations, i.e., only the score-producing linear layer weights w and b are trained on the pairwise ranking task. Results. The results for the baseline models are given in Table 1 (see top five rows). We see that despite accuracies of more than 90% on the WSJ permuted document task, the LCD models perform only a little above a random baseline of 50% on most of the independent test sets, with LCD-G being the best generalizing model out of the three. 3 https://github.com/BorealisAI/cross_domain_coherence 4 https://github.com/taasnim/unified-coherence-model Similarly, despite a relatively high performance on the WSJ test set (94.11%), UNC's performance on the independent test sets is quite poor, even failing to do better than the random baseline of 50% in two out of five cases. Both the LCD and UNC models have slightly better success on the INSTED-CNN dataset, which is the same domain (news) as the training data, with the UNC model reaching 67.21% accuracy. Our XLNet-Pairwise model trained without fine-tuning the representations (No FT) performs no better than the baseline models. This shows that both the LCD-G and the UNC models are in fact strong baselines despite using GloVe and ELMo (Peters et al., 2018) pre-trained representations respectively. Our fully-trained XLNet-Pairwise model not only outperforms the UNC model on the standard WSJ permuted document task, but also significantly outperforms all baseline models on the independent test sets, showing an absolute improvement of 15-20% on the SUMMEVAL , INSTED-CNN, INSTED-WIKI and the STORYCLOZE datasets. On LMVLM, the UNC model has a better performance; we suspect that its explicit conditional language modeling loss might provide an additional advantage for this particular task. Overall, our results are consistent with observations from Mohiuddin et al. (2021) that show poor generalizability in the previous SOTA model. Setup. In pairwise ranking, each positive sample is only compared to one negative at a time. Contrastive learning (Gutmann and Hyvrinen, 2010) makes it general, where a single positive sample can be compared to multiple negatives, which can be particularly useful in the permuted document task where the number of possible incoherent samples per coherent document can be very large. The number of negatives considered and their quality can affect model performance (Arora et al., 2019). Wu et al. (2020) show that contrastive loss maximizes a lower bound on the mutual information between representations. A larger number of negatives increases the tightness of the bound; learning with more negatives can better maximise the mutual information. We train our model with a margin-based contrastive loss defined as: L  =  log (cid:16) e f  ( D + ) e f  ( D + ) + (cid:80) Nj =1 e ( f  ( D  j )   ) (cid:17) (2) 6047 Model WSJ SUMMEVALLMVLM INSTED-CNN INSTED-WIKISTORYCLOZELCD-G 90 . where f  ( D + ) is the coherence score of the positive document, f  ( D  1 ) ,    , f  ( D  N ) are the scores of the N negative documents, and  is the margin. Training. We use the same training data as the baseline models to train our contrastive model; the positive documents remain the same, while we use 5 negative documents per instance (instead of only 1 in the pairwise setup). Effectively, the model sees the same number of positive or coherent documents, but five times as many negative samples during training compared to the pairwise setting. Appendix A.5 gives the full set of hyperparameters. Results. From the results in Table 1, we see that the contrastive model (second to last row) further improves the results across all the independent test sets; the results on the LMVLM dataset also improve, surpassing the UNC model performance. Although the improvement on the WSJ permuted document task is small, the improvement in the generalizability of the model is more significant. It has been shown that the difficulty of the negative samples used for contrastive training can strongly influence model success (Wu et al., 2020; Huang et al., 2020). We therefore automatically mine hard negatives during training. For the permuted document task, we can take advantage of the fact that the negative sample space can be huge; for a document with n sentences, the candidate pool of permutations has n ! 1 incoherent documents from which we can mine hard negatives. For the problem of dense text retrieval, Xiong et al. (2020) find global hard negatives by computing document encodings using a recent checkpoint to build an asynchronous index of the entire corpus, and sampling negative documents from the index. However, the huge candidate pool for permuted documents also makes it infeasible to mine global negatives in our case. Instead, we perform local negative sample ranking. For each positive instance in the training data, we sample a larger number of permuted documents ( h ) per instance than we need for training ( i.e., h > N ). We score these negative documents using the model updated thus far and use the highest ranking negatives for training. Specifically, the model is first trained with x instances ( x is a hyperparam-eter) of data, by using 5 negative samples randomly chosen out of h . The updated model is then used to score all the h negative samples each for another set of x instances from the training data. The scores of the h negative samples are ranked and the top scoring 5 negative samples for each instance are used to train the model for the next x gradient steps. This process is repeated throughout training; the model therefore iteratively mines harder and harder negative samples as it improves. See Algorithm 1 in Appendix A.2 for the pseudocode. In practice however, we find that using hard negative samples directly leads to instability in model training (see 4.1). We therefore use hard negative training in combination with a momentum encoder, which we describe in the next subsection. While increasing the number of negative samples per instance has been shown to be effective for constrastive learning, resource constraints can limit the number of negatives that can be considered per instance. One solution is to consider other positive instances in the same training batch as nega-6048 Document Representations Negative Positive Encoders Slice Inputs Queue CoherenceScores Loss Momentum Encoder Linear BaseEncoder Inner Product Figure 1: Our coherence model with the auxiliary momentum encoder. tives (Karpukhin et al., 2020; Chen et al., 2020). However, it is not suitable for the permuted document task since the negatives are instance-specific. While a permuted document is still independently incoherent, training with permutations of other documents will not provide the same cues for coherence modeling as the original self-supervision. Another solution is to maintain a large global queue of negative samples that are independent of the current training instance. During training, negative samples (specifically, their representations) from the latest batch are enqueued to build a queue upto some size l . As training continues, the negative samples from the oldest batch are dequeued to accommodate newer samples. However, representations of the documents will evolve through training as the model parameters get updated; this will make the negative samples in the queue inconsistent with each other and the training instances in the current batch. Moreover, the issue of mismatched self-supervision with negatives that are permuted versions of other documents still remains. Momentum Encoder. To address these issues, we add an auxiliary momentum encoder (He et al., 2019), which is also XLNet (Yang et al., 2019). Figure 1 shows the overall architecture. Keeping the base contrastive setup the same (the upper part), we add an additional contrastive objective based on representations from the momentum encoder. Specifically, we re-encode the positive and negative samples through the momentum encoder; the negative samples thus encoded are used to build the queue. We train the model to promote the similarity between the positive representations from the momentum encoder and the positive representations from our base encoder over the similarity with the negative samples from the queue, Q . Specifically, we define a momentum loss L mom  as: c + = ( z + )  ( z + m ) || z + || || z + m || ; c  j = ( z + m )  q j || z + m || || q j || ; L mom  =  log (cid:16) e c + e c + + (cid:80) lj =1 e ( c  j   ) (cid:17) (3) where z + and z + m are the positive representations from the base encoder (  ) and the momentum encoder (   ) respectively, q 1 , . . . , q l indexed by j are the negative representations from   in the queue, and  is the margin. The momentum encoder   is updated based on the base encoder  as:        + (1   )   (4) where   [0 , 1) is the momentum coefficient; only  is updated through backpropagation. Our full model is trained with a combination of the original contrastive objective (Eq. 2) and the momentum encoded contrastive similarity objective (Eq. 3): L  =  L  + (1   ) L mom  (5) where  is a weighting hyperparameter. Note that the momentum encoder can be considered as a temporal ensemble model consisting of exponential-moving-average versions of the base model. Due to this, the gradients from the momentum loss (Eq. 3) also help in stabilising the overall training (4). Length Invariance. In the permuted document task, both the positive and the negative samples have the same number of sentences. This is not necessarily the case for downstream applications. To incorporate length invariance into our model, we encode a random contiguous slice of the positive document through the momentum encoder   . 5 The global negatives queue Q is constructed from the mined hard negative samples used for training. Our model is therefore trained to rely not only on comparative coherence cues from the traditional permuted document setup, but also to recognize more independent cues for coherence through the global queue, which is additionally enhanced by incorporating length invariance and automatically mined hard negative samples. Training. We train the model with the same training data, this time sampling h = 50 negatives 6 per instance for hard negative ranking, and setting the training steps (or instances) x = 200 . We use a queue size of l = 1000 and set our momentum coefficient  = 0 . 9999999 , with loss weighting parameter  = 0 . 85 . Due to GPU memory constraints (24GB, Quadro RTX 6000), we train our model with a batch size of 1. See Appendix A.5 for the full set of hyperparameters. Results. The results in Table 1 (last row) show that our momentum encoder model with hard negative mining outperforms all previous models across the independent testsets. This improvement comes despite a very similar performance on the WSJ test set; we believe that our model truly improves in generalizability without overfitting to the permuted document task. The improvements on the out-of-domain test sets, particularly on LMVLM and STORYCLOZE , support this conclusion. We only train our complete model ( i.e., base contrastive plus momentum model) by mining hard Minimum is 4 and maximum is full document. 6 As previously described in 2, we ensure the sampled negative documents are unique even when the positive documents are repeated. This ensures that a much larger sample of the overall candidate pool is considered during training. Since we sample and rank 50 negative documents per positive instance, accounting for 20 repetitions of the positive documents, 20  50 = 1000 total negative documents are considered for hard negative mining. This is 10 times larger than the contrastive setup (100 unique negatives) and 50 times larger than the pairwise setup (only 20 unique negatives). negative samples (3.5), because we find that training the base contrastive model directly with hard negatives leads to instability during training. Figure 2a plots development set accuracies of our base model trained with and without hard negative mining, and our complete model trained with hard negative mining (evaluated every 1000 steps). As seen in the figure, the contrastive model displays significant volatility when trained with hard negatives only, while the complete model is quite stable. This is inline with the finding of Xuan et al. (2020) who show that training with the hardest negative samples leads to bad local minima. This can be explained with the gradient analysis of such negatives which have a larger gradient norm (Xiong et al., 2020), resulting in abrupt gradient steps. The momentum encoder being a temporal ensemble of the base models has a regularizing effect, addressing this issue and leading to stable and improved results (see 3.5). Number of Ranked Negatives. Figure 2b shows the results across the test sets for different num-bers of negative samples considered for ranking ( h ) during hard negative mining. We see that increasing the number of negatives considered improves results across the board, with results on out-of-domain test sets LMVLM and STORYCLOZE showing particular improvement. Momentum Coefficient. Figure 2c shows the variation in the model performance across the test sets for different values of the momentum coefficient  . We see that apart from a slight drop on the INSTED-WIKI dataset at  = 0 . 9999999 , overall an increasing  value leads to better generalization on the independent test sets, presumably due to a more consistent global negative queue. Queue Size. Figure 2d shows the variation in model performance across different test sets for various sizes of the global negative queue Q . We see that while increasing the queue size generally leads to an improvement in scores, at high queue sizes the improvement is limited to test sets from the same domain (WSJ, SUMMEVAL and INSTED-CNN), and the model's generalizability is affected. So far, we have reported the results of training our model on the permuted document task using documents from the WSJ corpus as was done by most prior work (Elsner and Charniak, 2011; Moon et al., 2019). We now test the effectiveness of other datasets, by varying the task itself and by using a different dataset for the permuted document task. Sentence Intrusion. As described in 2.3, Shen et al. (2021) propose a sentence intrusion task to test coherence modeling capabilities of pre-trained language models. We adapt their dataset to the pairwise setting by pairing the original coherent document (positive) with the corrupted (negative) document; setting aside 10% of the data for development gives us 25,852 positive-negative training pairs for INSTED-CNN and 41,135 pairs for INSTED-WIKI . We train our pairwise (3.2) model on this task. From the results in Table 2 (first two rows), we see that the performance on the same domain/task (as the training) and the performance on the LMVLM dataset is high, but the models trained on this task generalize poorly to the other independent test sets. Permuted Document Task with INSTED. We train our model on the permuted document task using the INSTED datasets. We generate 52,607 and 66,679 positive-negative pairs for INSTED-CNN and INSTED-WIKI respectively by sampling permutations, similar to our training data (see 2.1), and train our pairwise model with this data. Specifically for machine generated texts, results in Table 2 show that the sentence intrusion task training does better on the LMVLM dataset. On the other hand, the permuted document task training does better on SUMMEVAL . This could be because the documents in SUMMEVAL are summaries of the same source article and therefore similar in content (detecting incoherence through permutations might help here), while the text generated by language models even for the same prompt tends to differ in content more significantly (detecting intruder sentences might help here). Additionally, the performance of our WSJ model on the INSTED-CNN 6051 Train Dataset Neg. Type Model WSJ SUMMEVALLMVLM INSTED-CNN INSTED-WIKISTORYCLOZEINSTED-WIKI Intrusion Pairwise 95 . 24  0 . 37 53 . 03  1 . 49 0 . 490  0 . 01 94 . 07  0 . 29 82 . 01  0 . 24 64 . 21  1 . 98 INSTED-CNN Intrusion Pairwise 95 . 48  0 . 47 57 . 85  2 . 47 0 . 502  0 . 01 97 . 83  0 . 15 73 . 52  1 . 17 71 . 75  1 . 81 INSTED-WIKI Permuted Pairwise 96 . 89  0 . 23 64 . 53  0 . 82 0 . 491  0 . 01 84 . 17  1 . 50 71 . 35  0 . 88 69 . 09  2 . 29 INSTED-CNN Permuted Pairwise 97 . 03  0 . 12 66 . 63  0 . 97 0 . 483  0 . 01 92 . 61  0 . 62 69 . 88  0 . 64 68 . 95  1 . 02 WSJ Permuted Pairwise 98 . 23  0 . 20 64 . 83  1 . 03 0 . 458  0 . 02 91 . 96  1 . 09 70 . 85  1 . 85 71 . 84  2 . 33 Table 2: Results on the WSJ permuted document test set and other independent test sets of the pairwise model trained on different datasets. and INSTED-WIKI datasets is comparable to the performance of the respective in-domain pairwise models, while outperforming both the other models on the STORYCLOZE dataset. Overall, the model trained on the WSJ permuted document task generalizes well. Shen et al. (2021) create 8 hand-crafted linguistic probe test sets by manually modifying words in coherent texts based on various linguistic phenomena, ensuring the incoherent text produced as a result remains syntactically correct. Except for the words targeted by the probe, the rest of the text remains identical. Each test set has 100 samples each. 7 We evaluate the best performing LCD-G, UNC and our full models on these test sets. The results are shown in Table 3 along with some examples from the dataset. The LCD-G model has mixed success across the test sets. The UNC model has the most success with the tense agreement test set and is moderately successful on the pronoun test sets. We see that our model has perfect accuracy on all pronoun-related test sets and near-perfect accuracy on the tense agreement test set. This shows that our model is indeed capturing the discourse-level phenomena that constitute coherence. Where our model falters is in cases which may require 7 Except for Single Determiner Flip, which has 95. commonsense knowledge, such as identifying that 6.7 wins is not possible. Overall, our model is quite successful in detecting several kinds of incoherence. We show empirically that increasing the ratio and quality of negative samples improves the generalizability of the coherence model. We also test our model on a wide-ranging collection of independent test sets that resemble downstream applications, including machine generated text, on which our model significantly outperforms the previous SOTA model. Our work thus also sets a new evaluation standard for future research in coherence modeling. We open source our code base to encourage research in a new paradigm of coherence modeling. We would like to thank the Senior Area Chairs of ACL 2022 for evaluating our paper on its merits, and the reviewers and meta-reviewer of ARR for their reviews. We would also like to thank our colleagues Mathieu Ravaut and Han Cheol Moon for their valuable inputs. A description of the data pre-processing is provided in 2.1. Datasets that we created will be open-sourced. In the case of the WSJ dataset, the data is licensed for use only to members by the Linguistic Data Consortium. Consequently, we only release scripts to generate the data we use and not the data itself. We highlight however that the permuted document self-supervision task that we train on is independent of the dataset used and the task can be reproduced on any other corpus; see also 4.3. All other datasets we use are licensed freely for academic use. We conduct a user study to collect pairwise coherence judgments on our language model output dataset. As part of our crowd-sourced user study on Amazon Mechanical Turk to collect these coherence judgements, we do not collect any personal information from the participants. Based on the average time spent to perform the tasks, participants were paid the equivalent of 16 USD per hour for their work. The annotation instructions and interface provided to the participants are included in Appendix A.3. One potential issue is that the language model output that we generate from prompts may lead to malicious text generation by the models. We flagged the task to warn the workers that there may be potentially offensive content, and manually checked the final dataset post curation. All our experiments are conducted using data for the English language. However, as coherence and discourse relations in text are a universal concept, and our training data is automatically generated, we expect the permuted document task to be easily extensible to other languages.
objective abstain result objective result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain result abstain method objective objective objective method method method abstain abstain abstain method method abstain method abstain method abstain method method abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain other abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result result objective objective other other abstain method abstain method abstain method method method abstain abstain method abstain method method