In this paper, we propose an entity-based neural local coherence model which is linguistically more sound than previously proposed neural coherence models. Recent neural coherence models encode the input document using large-scale pretrained language models. Hence their basis for computing local coherence are words and even sub-words. An analysis of their output shows that these models frequently compute coherence on the basis of connections between (sub-)words which, from a linguistic perspective, should not play a role. Still, these models achieve state-of-the-art performance in several end applications. In contrast to these models, we compute coherence on the basis of entities by constraining the input to noun phrases and proper names. This provides us with an explicit representation of the most important items in sentences leading to the notion of focus. This brings our model linguistically in line with pre-neural models of computing coherence. It also gives us better insight into the behaviour of the model thus leading to better explainability. Our approach is also in accord with a recent study (O'Connor and Andreas, 2021), which shows that most usable information is captured by nouns and verbs in transformer-based language models. We evaluate our model on three downstream tasks showing that it is not only linguistically more sound than previous models but also that it outperforms them in end applications 1 . Coherence describes the semantic relation between elements of a text. It recognizes how well a text is organized to convey the information to the reader effectively. Modeling coherence can be beneficial to any system which needs to process a text. Example Sentence 1 Mr. Specter, seeming exasperated, said in an interview Thursday. Focus candidates captured by XLNet _said, _in, day, _interview, _,, er, _an, th, s, _exasperated, ..., spect Example Sentence 2 At the same time, unadvertised products may have almost identical ingredients but less name-recognition. Focus candidates captured by XLNet _name, ition, _products, -, _un, _may, _less, _ingredients, _have, ..., _same Table 1: The pretrained language model, XLNet Yang et al. (2019), captures undesirable (sub-)words as focus (Jeon and Strube, 2020). Recent neural coherence models (Mesgar and Strube, 2018; Moon et al., 2019) encode the input document using large-scale pretrained language models (Peters et al., 2018). These neural models compute local coherence, semantic relations between items in adjacent sentences, on the basis of words and even sub-words. However, it has been unclear on which basis these models compute local coherence. Jeon and Strube (2020) present a neural coherence model, which allows to interpret focus information for the first time. Their investigation reveals that neural models, adopting large-scale pretrained language models, compute coherence on the basis of connections between any (sub-)words or function words (Table 1, 11). In these cases, the model might capture the focus based on spurious information. While such a model might reach or set the state of the art in some end applications, it will do so for 7787 the wrong reasons from a linguistic perspective. This problem did not appear with pre-neural models, since they compute coherence on the basis of entities. Early work about pronoun and anaphora resolution by Sidner (1981, 1983) assumes that there is one single salient entity in a sentence, its focus, which serves as a preferred antecedent for anaphoric expressions. Centering theory (Joshi and Weinstein, 1981; Grosz et al., 1995) builds on these insights and introduces an algorithm for tracking changes in focus. Centering theory serves as basis for many researchers to develop systems computing local coherence by approximating entities (Barzilay and Lapata 2008; Feng and Hirst 2012; Guinaudeau and Strube 2013, inter alia). In this paper, we propose a neural coherence model which is linguistically more sound than previously proposed neural coherence models. We compute coherence on the basis of entities by constraining our model to capture focus on noun phrases and proper names. This provides us with an explicit representation of the most important items in sentences, leading to the notion of focus. This brings our model linguistically in line with pre-neural models of coherence. Our approach is not only linguistically more sound but also is in accord with a recent empirical study by O'Connor and Andreas (2021) who investigate what contextual information contributes to accurate predictions in transformer-based language models. Their experiments show that most usable information is captured by nouns and verbs. Their findings suggest that we can design better neural models by focusing on specific context words. Our work follows their findings by modeling entity-based coherence in an end-to-end framework to improve a neural coherence model. Our model integrates a local coherence module with a component which takes context into account. Our model first encodes a document using a pretrained language model and identifies entities using a linguistic parser. The local coherence module captures the most related representations of entities between adjacent sentences, the local focus. Then it tracks the changes of local foci. The second component captures the context of a text by averaging sentence representations. We evaluate our model on three downstream tasks: automated essay scoring (AES), assessing writing quality (AWQ), and assessing discourse coherence (ADC). AES and AWQ determine text quality for a given text, aiming to replicate human scoring results. Since coherence is an essential fac-tor in assessing text quality, many previous coherence models are evaluated on AES and AWQ. ADC evaluates coherence models on informal texts such as emails and online reviews. In our evaluation, our model achieves state-of-the-art performance. We also perform a series of analyses to investigate how our model works. Our analyses show that capturing focus on entities gives us better insight into the behaviour of the model, leading to better explainability. Using this information, we examine statistical differences of texts assigned to different qualities. From the perspective of local coherence, we find that texts of higher quality are neither semantically too consistent nor too variant. Finally, we inspect error cases to examine how our model works differently compared to previous models. Entity-based modeling has been the prevailing approach to model coherence in pre-neural models. The entity grid is its most well-known implementation (Barzilay and Lapata, 2008). It represents entities in a two-dimensional array to track their transitions between sentences. Many variations have been proposed to improve this model, e.g., projecting the grid into a graph representation (Guin-audeau and Strube, 2013) or converting the grid to a neural model (Tien Nguyen and Joty, 2017). However, the neural version of the entity grid (Tien Nguyen and Joty, 2017) has two limitations. First, Lai and Tetreault (2018) state that entity grids applied to downstream tasks are often extremely sparse. In their evaluation, it is difficult to find meaningful entity transitions between sentences in the grids. Accordingly, this model performs worse than other neural models. More importantly, this neural model cannot provide any clues of how this model works since Tien Nguyen and Joty (2017) apply a convolutional layer on the entity grid. The feature map of the convolutional layer is not interpretable. They cannot examine which entity is assigned more importance than others by their model. In contrast, we constrain our model to capture focus on entities using noun phrases. Then our model tracks the changes of focus. Hence, it provides us with an interpretable focus (Section 5). More recently, Moon et al. (2019) propose a neural coherence model to exploit both local and structural aspects. They evaluate their model on an arti-7788 ficial task only, the shuffle test, which determines whether sentences in a document are shuffled or not. However, recent studies (Pishdad et al., 2020) claim that this artificial task is not suitable to evaluate coherence models. Lai and Tetreault (2018) show that the neural coherence models, which achieve the best performance on this task, do not outperform non-neural models on downstream tasks. More recently, Mohiuddin et al. (2021) find a weak correlation between the model performance in artificial tasks and downstream tasks. In our evaluation, we compare Moon et al. (2019) with ours in an artificial task as well as in three downstream tasks. Moon et al. (2019) perform the best in the artificial task, but do not outperform our model in three downstream tasks (Section 4). Figure 1 presents the architecture of our model. We first introduce our entity representation and sentence encoding using a pretrained language model. Next, we describe a novel local coherence model. We then combine the two representations of local coherence and the context vector, simply averaged sentence representations. Finally, we apply a feed-forward network to produce a score label. We use a pretrained language model (Yang et al., 2019) to encode sentences. XLNet learns bidirectional contexts by maximizing expected likelihood using an autoregressive training objective. Hence it allows to capture the focus in sentences. XLNet outperforms other language models in tasks which require processing long texts. Recent work investigates that pretrained language models learn linguistic features that are helpful for language understanding (Tenney et al., 2019; Warstadt et al., 2020). Inspired by this, we encode two adjacent sentences at once to capture discourse features, such as coreference relations. In this strategy, items are encoded twice except the items included in the first and the last sentence. We interpolate items encoded twice to consider context with regard to the preceding and succeeding sentence. We encode an input document using XLNet to obtain word representations. Sentence representations are means of all word representations in a sentence. We then feed sentence representations and the noun phrase representations into the the coherence modules. In formal definitions, let E e = [ h ( e,i, 1) , ..., h ( e,i,m ) , h ( e,i +1 , 1) , ..., h ( e,i +1 ,m ) ] denote the output of encoding, where e indicates the index of encoding, and m indicates the index of a subword ( w ) in the sentence ( s i ). h indicates the encoded representation of w . This encoding output includes the encoded representations of s i and s i +1 since we encode two adjacent sentences at once. Likewise, E e +1 = [ h ( e +1 ,i +1 , 1) , ..., h ( e +1 ,i +2 ,m ) ] is the output in the next encoding, and it includes the encoded representations of s i +1 and s i +2 . Then, the encoded representation of s i +1 is a sequence of ih ( i +1 ,m ) = avg ( h ( e,i +1 ,m ) , h ( e +1 ,i +1 ,m ) ) , which is the interpolated representation of s i +1 in the two encoding stages ( e and e + 1 ). We iterate this process to encode all adjacent sentences. Pretrained language models encode sequences as sub-words, but to our knowledge, there is no linguistic parser using sub-words as input. Hence, we use a linguistic parser to identify noun phrases in each sentence separately. Kitaev and Klein (2018) present a neural constituency parser which determines the syntactic structure of a sentence. To identify noun phrases and proper names, we ap-7789 ply this parser to the original sentences, then map parsed constituents to sub-word tokens. Since pretrained language models do not have the means to represent phrase meaning composition, we average sub-word representations for phrases which consists of multiple sub-words. While this implementation does not capture the complex meaning of phrases, Yu and Ettinger (2020) report that it shows higher correlation with human annotations than using the last word of phrases, assuming that the last word of a phrase is its head. Let NP i = [ np i, 1 , np i, 2 , ..., np i,j ] denote a sequence of noun phases ( np ) in the i th sentence, and j indicates the index of a noun phrase in the sentence. Each representation of a noun phrase is obtained as np i,j = avg ( ih i, 1 , ..., ih i,k ) , where ih i,k indicates the subword tokens contributing to the same entity. We compare the semantic representations of noun phrases between adjacent sentences. The two most similar representations of noun phrases are taken as local focus of the respective sentences. These two representations are averaged to capture the common context. We use cosine similarity to measure semantic similarity. We notice that some sentences do not include noun phrases, approximately 3.5% in the three datasets used in our evaluation. This mostly occurs when some words are omitted as in cases of ellipsis (Hardt and Romero, 2004). In such cases, we maintain the focus of the previous sentence to preserve the context. A depthwise convolutional layer is applied to the local focus to record its transitions. Unlike a typical convolutional layer, the depthwise convolutional layer captures the patterns of semantic changes between different time-steps for the same spatial information (Chollet, 2017). In our model, this layer captures the semantic changes between local foci considering the context but on the same spatial dimension of each focus. Hence, it does not hurt the explainability of our model. We use the lightweight depthwise convolutional layer (Wu et al., 2019). Then we update the representations of local foci to track the semantic changes between them. We use the Tree-Transformer which updates its hidden representations by inducing a tree-structure from a document (Wang et al., 2019). It generates constituent priors by calculating neighboring attention which represents the probability of whether adjacent items are in the same constituent. The constituent priors constrain the self-attention of the transformer to follow the induced structure. Finally, we apply document attention to produce the weighted sum of all the updated local focus representations. The document attention identifies relative weights of updated representations which enables our model to handle any document length. In formal descriptions, let mnp l,i denote the representations of two noun phrases which have the highest cosine similarity scores between the i th and i + 1 th sentence. Then, we define LocalF = [ localf 1 , ..., localf l ] , where localf l is an averaged representation of mnp l,i and mnp l,i +1 . It represents the sequence of local foci between the i th and i + 1 th sentence, and l indicates the index of the local focus in the document. Finally, the local coherence representation is obtained as lcr = doc _ attn ( tree _ trans ( dconv ( LocalF ))) where dconv indicates the depthwise convolutional layer, tree _ trans indicates the Tree-Transformer, and doc _ attn indicates the document attention. We implement our model using the PyTorch library and use the Stanford Stanza library 2 for sentence tokenization. We employ XLNet for the pretrained language model. For the baselines which do not employ a pretrained language model (Dong et al., 2017; Mesgar and Strube, 2018), GloVe is employed for word embeddings, trained on Google News (Pennington et al., 2014) (see Appendix A for more details). To compare baselines within the same framework, we re-implement all of them in PyTorch. We then use our re-implementation to report the performance of models with 10 runs with different random seeds. We verify statistical significance (p-value < 0.01) with both a one-sample t-test, which verifies the reproducibility of the performance of each model, and a two-sample t-test, which verifies that the performance of our model is statistically significantly different from other models. Within the same framework we compare the size of models used in our experiments. Our neural model uses a number of parameters comparable to the state of the art, the transformer-based model 2 https://stanfordnlp.github.io/stanza 7790 (Moon et al. (2019): 118M < Jeon and Strube (2020): 136M < Our model: 137M). In all three downstream tasks, we compare our model against recent neural coherence models. First, Mesgar and Strube (2018) propose a neural local coherence model, based on Centering theory. This model connects the most related states of a Recurrent Neural Network, then represents the coherence patterns using semantic distances between the states. Second, Moon et al. (2019) propose a unified neural coherence model to consider local and structural aspects. This model consists of two modules when they employ a pretrained language model (Peters et al., 2018): a module of inter-sentence relations using a bilinear layer and a topic structure module applying a depth-wise convolutional layer to the sentence representations. To ensure fair comparison, XLNet is employed for this model as well, instead of ELMo (Peters et al., 2018). More recently, Jeon and Strube (2020) propose a neural coherence model approximating the structure of a document by connecting linguistic insights and a pretrained language model. This model consists of two sub-modules. First, a discourse segment parser constructs structural relationships for discourse segments by tracking the changes of focus between discourse segments. Second, a structure-aware transformer updates sentence representation using this structural information. We first evaluate our model on the artificial setup, the shuffle test, used in earlier works (Table 2). We follow the setup used in Lai and Tetreault (2018). In this setup, our model outperforms a simple neural model relying on the pretrained language model. Moon et al. (2019) evaluate their models only in this setup. It achieves outstanding performance in this setup. However, in the following sections, our results show that this model does not outperform our model in downstream tasks. This result is not surprising. There is a line of recent work which shows that this setup is not capable of evaluating coherence models from diverse perspectives. Laban et al. (2021) show that employing fine-tuned language models simply achieves a near-perfect accuracy on this setup. O'Connor and Andreas (2021) measure usable information by selectively ablating lexical and structural information in transformer-based language models. Their findings show that prediction accuracy depends on information about local word co-occurrences, but not word order or global position. We suspect that exploiting all information of a sentence is sufficient for shuffle tests to capture patterns to distinguish whether sentences in a document are shuffled or not. Based on these findings, we evaluate our model on three downstream tasks used for evaluating coherence models, automated essay scoring, assessing writing quality, and assessing discourse coherence. We advise future work not to evaluate coherence models on the artificial setup solely. Dataset. To evaluate the coherence models on AES, we evaluate them on the Test of English as a Foreign Language (TOEFL) dataset (Blanchard et al., 2013). While the Automated Student Assessment Prize (ASAP) dataset 3 is frequently used for AES, TOEFL has a generally higher quality of essays compared to essays in ASAP. The prompts in ASAP are written by students in grade levels 7 to 10 of US middle schools. Many essays in ASAP consist of only a few sentences. In contrast, the prompts in TOEFL are submitted for the standard English test for the entrance to universities by nonnative students. The prompts in TOEFL do not vary so much, the student population is more controlled, and essays have a similar length. Evaluation Setup. We follow the evaluation setup of previous work on AES (Taghipour and Ng, 2016). For TOEFL, we evaluate performance with accuracy for the 3-class classification problem with 5-fold cross-validation. We use the same split for the cross-validation, used by Jeon and Strube (2020). The cross-entropy loss is deployed for training. The ADAM optimizer is used for our model with a learning rate of 0.003. We evaluate performance for 25 epochs on the validation set with a mini-batch size of 32. The model which reaches the 3 https://kaggle.com/c/asap-aes 7791 Model Prompt Avg 1 2 3 4 5 6 7 8 Dong et al. (2017) 69.30 66.47 65.84 66.38 68.89 64.20 67.11 65.73 66.74 Mesgar and Strube (2018) 56.25 55.94 55.20 57.20 56.57 55.10 56.97 58.39 56.45 Averaged-XLNet-1S 70.73 69.48 68.98 67.52 72.35 70.94 70.14 69.01 69.89 Moon et al. (2019)-XLNet 73.75 72.13 72.92 73.29 75.12 74.69 72.89 72.09 73.36 Jeon and Strube (2020)-1S 75.10 73.35 74.75 74.18 76.38 74.30 73.61 73.44 74.39 Jeon and Strube (2020)-2S 76.35 75.40 75.00 74.85 77.63 74.06 73.71 74.00 75.12 Our Model 78.38 75.70 76.58 76.56 79.10 76.41 75.03 74.57 76.54 Table 3: AES: TOEFL Accuracy performance comparison on the test sets, 1S indicates that sentences are encoded individually and 2S indicates that two adjacent sentences are encoded at once on the pretrained language model (see Table 12, 13 in the Appendix C for more details). best accuracy on the validation set is then applied to the test set. Baselines. We compare against Dong et al. (2017), a neural model proposed for AES. They present a model consisting of a convolutional layer, followed by a recurrent layer, and an attention layer (Bah-danau et al., 2015) between the adjacent tokens. Results. Table 3 reports the performance on TOEFL. Dong et al. (2017) report better performance than the more recent neural model based on Centering theory (Mesgar and Strube, 2018). A simple model relying on the pretrained language model outperforms this model, which averages all sentence representations (henceforth, Avg-XLNet). Moon et al. (2019) show that their unified model outperforms previous models on the artificial task, the shuffle test. However, it does not outperform the previous models on the AES task. Jeon and Strube (2020) outperform previous models. Finally, our model, which integrates local and structural aspects, achieves state-of-the-art performance. We perform an ablation study to investigate the contribution of individual components. We compare with Jeon and Strube (2020) who encode two adjacent sentences using the pretrained language model (2SentsEnc). Our results verify that this encoding improves performance, but our model benefits from the novel local coherence module even more. Dataset. Louis and Nenkova (2013) create a dataset of scientific articles from the New York Times (NYT) for assessing writing quality. They assign each article to one of two classes by a semi-supervised approach: typical or good. Though articles included in both classes are of good quality overall, Louis and Nenkova (2013) show that lin-NYT Liu and Lapata (2018)-reimpl 54.35 (1.00) Averaged-XLNet-1SentEnc 67.53 (3.48) Moon et al. (2019)-XLNet-1Sent 74.75 (1.27) Jeon and Strube (2020)-1Sent 75.12 (1.10) Jeon and Strube (2020)-2Sents 76.43 (0.88) Our Model 77.52 (0.42) Table 4: AWQ: Mean (standard deviation) accuracy of assessing writing quality on the test sets in NYT. Evaluation Setup. For NYT, we follow the setup used in previous work. Louis and Nenkova (2013) and Ferracane et al. (2019) undersample the dataset to mitigate the bias of the uneven label distribution. Following Ferracane et al. (2019), Jeon and Strube (2020) partition the dataset into 80% training, 10% validation, and 10% test set, respectively. We use the ADAM optimizer with a learning rate of 0.001 and a mini-batch size of 32. We evaluate performance for 25 epochs. Baselines. Liu and Lapata (2018) propose a neural model which induces structural information without a labeled resource. It induces a non-projective dependency structure by structured attention. Results. Table 4 shows the performance on NYT. Ferracane et al. (2019) reported the best performance of the latent learning model for discourse structure (Liu and Lapata, 2018) on NYT. However, Jeon and Strube (2020) show that the good results are due to embeddings obtained by training on the target dataset. They also report that Avg-XLNet outperforms this model which employs Glove embeddings. Moon et al. (2019) show better performance than this simple model, but it does 7792 Model Yahoo Clinton Enron Yelp Avg Acc  Li and Jurafsky (2017) 53.5 61.0 54.4 49.1 51.7 Mesgar and Strube (2018) 47.3 (1.8) 57.7 (0.6) 50.6 (1.2) 54.6 (0.3) 52.6  Lai and Tetreault (2018) 54.9 60.2 53.2 54.4 55.7 Avg-XLNet-1Sent 58.0 (3.9) 57.6 (0.3) 54.3 (0.8) 55.9 (0.4) 56.4 Moon et al. (2019)-XLNet-1SentEnc 56.2 (0.5) 61.0 (0.4) 53.6 (0.5) 56.6 (0.4) 56.9 Jeon and Strube (2020)-1SentEnc 56.4 (0.6) 62.5 (0.9) 54.5 (0.4) 56.9 (0.3) 57.6 Jeon and Strube (2020)-2SentsEnc 57.2 (0.5) 63.0 (0.4) 54.4 (0.4) 56.9 (0.2) 57.9 Our Model 58.4 (0.2) 64.2 (0.4) 55.3 (0.3) 57.3 (0.2) 58.9 Table 5: ADC: Mean (standard deviation) accuracy performance on the test sets in GCDC (  : reported performance in Lai and Tetreault (2018)). not outperform Jeon and Strube (2020). Our model achieves state-of-the-art performance. An ablation study of the joint sentence encoding, Jeon and Strube (2020)-2SentsEnc, verifies that our model gains improvements not only from this encoding but also from our local coherence module. Dataset. While previous work evaluates coherence models on formally written texts (Barzilay and Lapata, 2008), GCDC (Lai and Tetreault, 2018) is designed to evaluate coherence models on informal texts, such as emails or online reviews. The dataset contains four domains: Clinton and Enron for emails, Yahoo for questions and answers in an online forum, and Yelp for online reviews of businesses. The quality of the dataset is controlled to have evenly-distributed scores and a low correlation between discourse length and scores 4 . Evaluation Setup. For GCDC, we perform the experiments following previous work (Lai and Tetreault, 2018). We perform 10-fold cross-validation, use accuracy as evaluation measure on the 3-class classification, and use the cross-entropy loss function. Baselines. Li and Jurafsky (2017) propose a neural model based on cliques, that are sets of adjacent sentences. This model uses the cliques taken from the original article as a positive label and uses cliques with randomly permutated ones as a negative label. Lai and Tetreault (2018) show that a simple neural model which uses paragraph information outperforms previous models on GCDC. Results. Table 5 summarizes the performance on GCDC. While Avg-XLNet outperforms previous baselines, other advanced neural models show sim-4 The Pearson correlation between text length and scores is lower than 0.12 in all domains. ilar performance. Our model performs slightly better than Jeon and Strube (2020) with two sentences encoding. This shows that the gains mainly benefit from this encoding strategy. We suspect that Jeon and Strube (2020) do not benefit from structural information since texts on GCDC are not well-organized. The texts mostly consist of a few sentences, and they express the writers' emotion. Based on this, Lai and Tetreault (2018) state that texts of lower quality have sudden topic changes. We also suspect that human annotators recognize important entities in the texts, such as the name of a person in the US government. Since our model consists of several components, we examine the influence of each component on the performance of the AES task. Specifically, we first examine the influence of our local coherence module. Then we examine the influence of the Tree-Transformer compared to a naive Transformer. Lastly, we examine the influence of the depth-wise convolutional layer deployed ahead of the Tree-Transformer. Table 7 shows that each component contributes to the performance meaningfully while the depthwise convolutional layer increases the performance slightly. This suggests that we could design a better component in future work to capture semantic transitions between local foci. 5.1 Capturing Focus Using Entities In Centering theory, the focus is described as the most important item in a sentence. Jeon and Strube (2020) capture the focus using attention scores and analyze texts assigned to different qualities using this focus. They state that the focus is difficult to interpret when it is composed of sub-words. To investigate this further, we compare the focus captured on any (sub-)words and the focus constrained to entities. Table 6 indicates that constraining focus to entities leads to better explainability, in particular on NYT. For example, in the NYT-1516415 news article about String theory, a subword of ein is not an interpretable focus. It may, however, include useful information in the vector space for a neural model. In contrast, our entity-based model leads to better explainability. Instead of ein, it provides the more interpretable focus, Einstein, a theoretical physicist. In TOEFL, broad knowl-edge is a more interpretable focus than a focus consisting of the single subword tokens, broad. Table 6 also shows that our model mainly uses pronouns, and noun phrases are playing an important role to represent focus. This suggests that further investigation is needed to understand how language models work on pronouns to process a text. Using interpretable focus information, we investigate differences in focus transitions of texts assigned to different scores. Motivated by the definition of the continue and the shift transition in Centering theory, we define semantic consistency which represents the degree of semantic changes between local foci. Two adjacent sentences are semantically consistent when the semantic similarity ( sim i ) between the local foci ( lf ) is higher than a semantic threshold (  sem ; score ). This threshold is determined as the average of semantic similarities between local foci of adjacent sentences in texts assigned the same score. Otherwise, a semantic transition ( st ) occurs between the local foci: st i = 1 if sim i <  sem ; score . Finally, the semantic consistency (SC) is defined as follows: SC = 1  ( count ( st i ) / | lf | ) . Figure 2 illustrates the semantic consistency on TOEFL, and Table 8 shows the statistics of the semantic consistency on texts assigned to different scores. Texts assigned a high score show lower semantic consistency on average. This indicates that texts of higher quality are overall more semantically variant than texts of lower quality. Additionally, we observe that texts assigned a low score show significantly larger proportions of an extreme level of semantic consistency. We define the extreme level as either texts whose semantic consistency is lower than 5 % , indicating texts are highly variant, or texts whose semantic consistency is higher than 75 % , indicating texts are highly consistent. Hence, these findings indicate that texts of lower quality are semantically too variant or too consistent. Texts of higher quality are neither too variant nor too consistent. We next inspect the focus of texts assigned to different scores (see Table 15,16, and 17 in the Appendix D for more details). This shows that pronouns more frequently indicate the local focus in texts of lower quality than in texts of higher quality. The essays in TOEFL are argumentative essays, and good essays should use facts and evidence to support their claim (Wingate, 2012). We observe that texts assigned a low score frequently include claims without convincing evidence. This 7794 1 2 3 4 5 6 7 8 N-th pair of local focus 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1.00 C o s i n e s i m il a r i t y b e t w ee n l o c a l f o c i causes our model to capture focus based on pronouns more frequently in these texts. In contrast, texts assigned a high score include convincing evidence to support claims, and this lets our model capture different types of foci in these texts. Finally, we conduct an error analysis to investigate how our model works differently compared to previous coherence models on TOEFL. We first compare the predicted scores with Moon et al. (2019) and a simple model which only considers context, averaged-XLNet. These two baselines show biased predictions in the middle score. We suspect that this is caused by the label bias in TOEFL (Blan-chard et al., 2013). Biased label distributions cause biased predictions, and they benefit from these biased predictions. In contrast, our model benefits more from predicting high scores correctly as well as other scores, indicating that our coherence model assesses text quality better. We then compare with the previous state of the art (Jeon and Strube, 2020). This baseline induces discourse structure to model structural coherence. It captures semantic relations between discourse segments, not just between adjacent sentences. We observe two error cases when this baseline struggles to predict correctly. It predicts scores lower than the ground-truth score for texts which lack support and evidence for claims. However, these texts have a well-organized paragraph for one or two claims. We suspect that this leads human annotators to assign a mid or a high score though the text is not well-organized overall. In contrast, it predicts scores higher than ground-truth scores when unrelated claims are listed or claims are listed S Low S Mid S High Avg SC 55.87 54.45 54.05 (std) (24.53) (21.38) (19.70) Prop of Ext level 17.63 11.54 8.59 Table 8: Semantic consistency statistics (%) for the texts assigned to different scores ( S ). without evidence. Our model, which captures local coherence between adjacent sentences, deals with these cases better (see Table 18 and 19 in the Appendix D for more details). We propose a neural coherence model based on entities by constraining the input to noun phrases. This makes our model better explainable and sets a new state of the art in end applications. It also allows us to reveal that texts of higher quality are neither semantically too consistent nor too variant. Our findings suggest a few interesting directions for future work. Our analysis shows that pretrained language models frequently exploit coreference relations to capture semantic relations. We could design an advanced neural model which exploits these relations explicitly. Lastly, our work could be extended to a multilingual setup. Our model is not tied to a specific pretrained language model but connect a language model with linguistic insights. It can employ a multilingual model (Xue et al., 2021), and our datasets can be translated to other languages. The authors would like to thank the anonymous reviewers for their comments. This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany. The first author has been supported by a Heidelberg Institute for Theoretical Studies Ph.D. scholarship.
objective abstain abstain abstain abstain objective method method abstain result result abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain abstain abstain method result method objective abstain abstain abstain method abstain abstain abstain result objective result method result method other other other other other other other other other other other objective method method other other other other other method abstain method abstain abstain method method method abstain abstain abstain abstain method abstain method abstain abstain abstain other abstain method abstain abstain method method method abstain method abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain method method method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result abstain result method objective abstain method other other other