Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., providing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omitting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened attention in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models. Simplification methods aim to make texts more readable without altering their meaning. This may permit information accessibility to a wide range of audiences, e.g., non-native speakers (Yano et al., 1994), children (De Belder and Moens, 2010), as well as individuals with aphasia (Carroll et al., 1998) and dyslexia (Rello et al., 2013). Simplification may also help laypeople digest technical information that would otherwise be impenetrable (Damay et al., 2006; Devaraj et al., 2021). Recent work has made substantial progress by designing sequence-to-sequence neural models that translate complex sentences into simplified versions (Xu et al., 2016; Alva-Manchego et al., 2020). (1) [Original] There was no difference in operating time or perioperative complication rates. [Model simplified] However, there was not enough evidence to determine if there was an important difference in operative time or complication rates when compared to conventional surgery. (2) [Original] All studies were associated with methodological limitations. [Model simplified] All studies were of poor quality and had limitations in the way they were conducted. (3) [Original] On June 24 1979 (the 750th anniversary of the village), Glinde received its town charter. [Model simplified] On June 24 1979, the 750th anniversary of the village was renamed. (4) [Original] Others agreed with the federal court; they started marrying people in the morning. [Model simplified] Others agreed with the federal court; they started trying in morning. (5) [Original] In 2014, Mary Barra became CEO of General Motors, making her the first female CEO of a major automobile company. [Model simplified] Also, just one woman leads a major automobile company. Omitted main subject. An important but mostly overlooked aspect of automated simplificationespecially in the conditional text generation regimeis whether outputs are faithful to the inputs that they are simplifying. Consider, for example, automatically simplifying medical texts (Devaraj et al., 2021): Presenting individuals with readable medical information that contains factual errors is probably worse than providing no such access at all. Recent work has acknowledged factuality and faithfulness as key issues to be addressed in other conditional generation tasks like summarization (Kryscinski et al., 2020a; Maynez et al., 2020; Pagnoni et al., 2021; Goyal and Durrett, 2021), yet so far little research has thoroughly studied the kinds of errors that simplification datasets and system outputs exhibit. This work seeks to close this research gap. Table 1 shows examples of generated outputs from existing simplification systems, and these clearly illustrate that factuality is an issue. We conduct multi-dimensional analyses based on the edit nature of simplification (Xu et al., 2015; Dong et al., 2019) and define a small typology of (poten-tial) factual errors in the context of simplification. Inserting information can be useful to define jargon and provide explanatory content, but introducing irrelevant or erroneous content (hallucinating) is bad (e.g., examples 1-2 in Table 1). Omitting information related to the main entity or event could lead to a change in how the text is understood (e.g., example 5 in Table 1). Finally, making inappropriate substitutions can result in inconsistencies (e.g., examples 3-4 in Table 1). Together these dimensions represent the precision, recall, and accuracy of information conveyed in simplified texts. We collect human ratings of factuality for these aspects on two widely used simplification corpora: Wikilarge (Zhang and Lapata, 2017) and Newsela (Xu et al., 2015). Automatically aligned sentences from these two datasets are typically used to train and evaluate supervised simplification systems. We find that errors occur frequently in the validation and test sets of both datasets, although they are more common in Newsela (Section 6). We then evaluate outputs from several modern simplification models (Zhang and Lapata, 2017; Dong et al., 2019; Martin et al., 2020; Maddela et al., 2021), as well as a fine-tuned T5 (Raffel et al., 2020) model. Compared to RNN-based models, Transformer-based ones tend to have less severe deletion and substitution errors; however, the pre-trained T5 produced more hallucinations on the more abstractive Newsela dataset. We find that existing quality metrics for simplification such as SARI (Xu et al., 2016) correlate poorly with factuality. Although deletion errors correlate with existing semantic similarity measures, they fail to capture insertion and substitution. As an initial step towards automatic factuality assessment in simplification, we train RoBERTa (Liu et al., 2019)-based classification models using our annotated data, and use synthetically generated data to supplement training. We demonstrate that this is a challenging task. Our code and data can be found at https://github.com/AshOlogn/Evaluating-Factuality-in-Text-Simplification. Factuality (and the lack thereof) has been iden-tified as critical in recent work in unsupservised simplification (Laban et al., 2021) and medical simplification (Devaraj et al., 2021). Guo et al. (2018) incorporated textual entailment into their simplification task via an auxillary loss. They showed that this improved simplifications with respect to standard metrics and human assessments of output fluency, adequacy, and simplicity, but they did not explicitly evaluate the resultant factuality of outputs, which is our focus. Given the paucity of prior work investigating factuality in the context of automated simplification, the most relevant thread of research to the present effort is work on measuring (and sometimes improving) the factuality in outputs from neural summarization systems. Falke et al. (2019a) proposed using textual entailment predictions as a means to identify errors in generated summaries. Elsewhere, Kryscinski et al. (2020a) used weak supervision heuristic transformations used to intentionally introduce factual errorsto train a model to identify inaccuracies in outputs. Maynez et al. (2020) enlisted humans to evaluate hallucinations (content found in a summary but not in its corresponding input) in automatically generated outputs. They report that for models trained on the XSUM dataset (Narayan et al., 2018), over 70% of summaries contain hallucinations. This corroborates other recent work (Falke et al., 2019a; Wallace et al., 2021), which has also found that ROUGE is a weak gauge of factuality. Wang et al. (2020a) proposed QAGS , which uses automated question-answering to measure the consistency between reference and generated summaries. Elsewhere, Xu et al. (2020) proposed evaluating textual factuality independent of surface realization via Semantic Role Labeling (SRL). Finally, Pagnoni et al. (2021) introduced the FRANK (meta-)benchmark for evaluating factuality metrics for summarization. While FRANK is tailored towards summarization-specific error categories including discourse, our ontology broadly reflects the goal of simplification (retaining content with simpler language) from the perspective of information precision, recall, and accuracy. Above we reviewed various recently proposed frameworks and methods for assessing the factual accuracy of automatically-generated summaries . We aim in this work to similarly codify content errors in simplification . Below we describe broad categories of errors 1 we observed in simplification datasets and system outputs, and then use these to design annotation guidelines that formalize accuracy assessment (Sec-tion 5). Our analysis revealed three broad categories, illustrated in Table 2: (1) Information Insertion: This occurs when information not mentioned in the complex sentence is inserted intoor hallucinated inits simplified counterpart. The insertion may be as small as mentioning a proper noun not in the complex sentence, or as large as introducing a new main idea. This category is similar to extrinsic hallucination in the summarization literature (Maynez et al., 2020; Goyal and Durrett, 2021). (2) Information Deletion: This is when information in the complex sentence is omitted from the simplified sentence. A minor example of this is the reverse of the insertion case above, where an entity is mentioned by name in the complex sentence but only by pronoun in the simplified sentence. (3) Information Substitution: This is when information in the complex sentence is modified in the simplified sentence such that it changes the meaning. This category is broad, encompassing both alterations to the simplified sentence that directly contradict information in the complex sentence, and those that do not. Because errors can co-occur, we adopt a multidimensional labeling scheme that requires a different label to be provided for each category. Each category label specifies the severity of the error: 0no/trivial change; 1nontrivial but preserves main idea; 2doesn't preserve main idea; -1 gibberish, specified in Figure 1. Table 1 shows 1 We adapt a graded labeling scheme based on content and meaning preservation. For brevity, we use the word error as a generic term to refer to all the phenomena captured by our labeling scheme, even those that may be considered acceptable in some simplification systems. level-2 examples from system outputs for insertion (examples 1-2), substitution (examples 3-4), and deletion (example 5). Reference examples are discussed in Section 6. Interpretation as Precision and Recall In simplification one attempts to rewrite a given complex sentence to be simpler while preserving most of the information that it contains. The categories above can be interpreted as errors in information precision (the fraction of content that also appears in the complex sentence) and recall (the fraction of content in the complex sentence preserved during simplification). With this interpretation, a false positive (affecting precision ) occurs when the simplified sentence contains information not present in the source, i.e., introduces a hallucination. And a false negative (hindering recall ) is where the simplified sentence omits key information in the source. We annotate data from the simplification datasets themselves (we will call these reference examples), as well as from model-generated text. Thus we assess how the distribution of errors in the references compares to that of errors in system outputs and glean insights that might relate model architecture and training choices to the kinds of errors produced. Lapata, 2017) datasets. These are commonly used in the literature, and so results have been reported on these corpora for a diverse collection of models. Wikilarge comprises 296K roughly-aligned sentences pairs from English Wikipedia and Simple English Wikipedia. Newsela (Xu et al., 2015) consists of 96K sentence pairs extracted from a dataset of news stories rewritten at 4 reading levels by professionals. To make analysis tractable in this work, we examine the simplest level for Newsela. We annotated 400 pairs of (complex, simplified) sentences each from the validation and test sets for Newsela. For Wikilarge, we annotated 400 pairs from the validation set and 359 from the test set (this constitutes the entire test set). Simplification Models. We annotated outputs generated by a collection of models on the same validation and test examples from Wikilarge and Newsela, respectively. We selected a set of models intended to be representative of different architectures and training methods. More specifically, for RNN-based models we considered Dress (Zhang and Lapata, 2017) and EditNTS (Dong et al., 2019). Dress is an LSTM model trained using REINFORCE (Williams, 1992) to minimize a reward function consisting of meaning preservation, simplicity, and fluency terms. EditNTS represents each sentence pair as a sequence of edit operations and directly learns these operations to perform simplification. For Transformer-based architectures we evaluated two previously proposed models: Access (Martin et al., 2020) and ControlTS (Mad-dela et al., 2021). Access trains a randomly-initialized Transformer to generate simplifications parametrized by control tokens influencing traits like lexical complexity and length compression. ControlTS is a hybrid method that generates simplification candidates using grammatical rules and then applies a BERT-based (Devlin et al., 2019) paraphrasing model. In addition, we also fine-tuned T5 (Raffel et al., 2020) for the simplification task, detailed in Appendix A. T5 is a Transformer-based model jointly pretrained both on unsupervised language modeling objectives and a host of supervised tasks including summarization and translation, all framed as text-to-text problems. ples from datasets, and for model-generated simplifications. To ensure that only annotators who understood our labeling scheme would be included, we released a qualification task consisting of 10 sentence pairs with perfect agreement among two of the authors, with detailed explanation of the labeling scheme, and required that annotators achieve at least 75% accuracy on this set. After worker qualification, examples were released to only qualified workers, and each example was annotated by 3 workers. The final label for each category (insertion, deletion, substitution) was set to the majority label if one existed. If every annotator provided a different label for a given category, we removed this example for purposes of this category. For example, if annotators provided insertion labels of { 1 , 1 , 2 } and deletion labels of { 2 , 1 , 0 } for a specific instance, then this would not be assigned a deletion label, but would receive a final insertion label of 1. Workers were compensated $10.00 per hour on the annotation task. Inter-annotator Agreement. We quantified the degree of inter-annotator agreement using 3 metrics, each capturing a different dimension of labeling consistency for each category: First, we report the percentage of examples that had a well-defined majority label for each category. Most annotators agreed on labels for the majority of examples (first column in Table 3), meaning that very few annotations had to be discarded for any category. Because 0 was the most common label for all 3 categories, especially for the reference examples from the datasets, we also recorded the percentage of examples with majority non-zero annotations that also have a well-defined majority label. For example, the labels { 0 , 1 , 2 } are majority non-zero but do not correspond to a well-defined majority label, while { 0 , 1 , 1 } satisfies both conditions. Table 3 (column 2) indicates that even among examples where most annotators agree that there is an error, the majority agree on a specific label of 1, 2, or -1. 7334 Category Dataset 0 1 2 -1 Insertion Wikilarge 91.1 6.3 0.3 2.3 Newsela 68.2 20.2 11.1 0.5 Deletion Wikilarge 76.2 18.0 3.5 2.3 Newsela 15.8 40.8 42.9 0.5 Substitution Wikilarge 90.1 6.7 0.9 2.3 Newsela 94.9 3.8 0.8 0.5 Table 4: Insertion, deletion, and substitution error distributions (%) in Wikilarge and Newsela test datasets. We also measured Krippendorff's alpha (Krip-pendorff, 1970) with an ordinal level of measurement (assigning the -1 label a value of 3 to indicate maximum severity). Dataset annotations for insertion enjoy moderate agreement (  = 0 . 425 ), those for deletion imply substantial agreement (  = 0 . 639 ), and those for substitution exhibit fair agreement (  = 0 . 200 ) (Artstein and Poesio, 2008). The latter is possibly due to the clear majority label of 0 among substitution labels. The % majority agreement scores indicate that although the annotation scheme involves a degree of subjectivity in distinguishing between minor and major errors, with proper screening crowdsource workers can label text pairs with our annotation scheme consistently enough so that a well-defined label can be assigned to the vast majority of examples. Quantitative Analysis Table 4 reports distributions of acquired labels for information insertion, deletion, and substitution errors over the annotated reference examples. Deletion errors are far more common than insertion errors in both datasets, though Wikilarge has fewer of both than Newsela. This is unsurprising, as one of the motivations for introducing the Newsela dataset was that it contains shorter and less syntactically-complex simplifications. Reassuringly, there were very few substitution errors found in either dataset. Table 5 shows a clear positive correlation between length reduction and the severity of deletion errors present. As expected, sentences are shortened more substantially in Newsela than in Wikilarge. One the other hand, while Table 5 indicates that the examples with nonzero insertion labels collectively see a greater increase in length than those with no insertion errors, the mean length increase for level 2 examples is smaller than that for level 1. Simplifications in Newsela are more abstractive (Xu et al., 2015), i.e., simplified sentences copy fewer phrases verbatim from inputs. This can be quantified via normalized edit distance (Lev-enshtein, 1965), which yielded a median of 0.46 for Newsela examples compared to the 0.38 for Wikilarge (after noise filtering described in Appendix B). Table 5 indicates that on average the more erroneous the insertion or deletion, the greater the normalized edit distance between the original and simplified sentences. These results suggest that while reducing sentence length and rewording can be beneficial (Klare, 1963), too much can negatively impact factuality. Qualitative Analysis We also manually inspected insertion and deletion errors in both datasets, revealing clear patterns of deletion errors. Label 1 deletions by definition involve omissions of nonsalient details that do not much affect the meaning of the sentence, e.g.: Original: Mayfield wrote and sang on a string of message-oriented records, including Keep on Pushing" and People Get Ready." Simplified: Mayfield wrote and sang on records that had a message. ( Newsela, deletion-1 ) Label 2 deletions have two common manifestations across the datasets. The first involves deletion of the main clause and subsequent promotion of a secondary clause: Original: Until you know how the sausage is made, you don't know how expensive it is to make that sausage, said Josh Updike, creative director of Rethink Leisure & Entertainment, which is working on several projects in China and elsewhere in Asia. Simplified: The company is working on several projects in China and Asia. ( Newsela, deletion-2 ) Another common type of label 2 deletion involves removing a key (though often small) phrase that effectively reframes the entire sentence, e.g.: Original: You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version . Simplified: You may add a passage of up to five words as a Front-Cover Text and a passage of up to 25 words as a Back-Cover Text to the end of the list of Cover Texts. ( Wikilarge, deletion-2 ) By deleting in the Modified Version (emphasis ours), the simplified sentence erroneously states that one may add frontand back-cover passages to the list of cover texts to the unmodified version, which is implicitly forbidden in the original. Because of the small number of insertion errors on Wikilarge, we were unable to identify any meaningful trends. However, we observed patterns in 7335 % length change Normalized edit distance Level 0 Level 1 Level 2 Level 0 Level 1 Level 2 Insertion Wikilarge -5.0 (17.0) 22.4 (36.9) 7.1 (0.0) 0.20 (0.20) 0.55 (0.40) 0.58 (0.0) Newsela -39.4 (23.8) -19.0 (36.9) -38.3 (29.0) 0.41 (0.17) 0.51 (0.21) 0.54 (0.04) Deletion Wikilarge 2.8 (15.8) -22.3 (18.9) -35.9 (15.9) 0.19 (0.23) 0.35 (0.18) 0.39 (0.14) Newsela 1.5 (27.6) -34.8 (23.1) -49.6 (22.8) 0.34 (0.31) 0.46 (0.13) 0.53 (0.10) Table 5: % length change (left) and normalized edit distances (right) in simplified sentences in each insertion and deletion error category (mean  standard deviation). Newsela for both levels 1 and 2 of insertions, pertaining to quotative phrases (e.g., inserting experts said to the beginning of a sentence even though the original sentence did not mention an expert), and temporal phrases, e.g.: Original: They could not afford to pay their son's roughly $10,000 cost for classes at the University of Texas at Austin. Simplified: When he grew up, they could not afford to pay $10,000 for him to go to the University of Texas at Austin. ( Newsela, insertion-1 ) We observed more contextually related errors for Newsela due to its style and its simplification process. Newsela documents were edited by professionals who rewrote the entire original document, and so information inserted or deleted could move from or to adjacent sentences. This preserves information for the whole document but causes problems at the sentence level. Also, compared to Wikilarge, Newsela's news articles naturally involve more complex discourse (Van Dijk, 2013). These factors lead to relatively underspeci-fied sentences (Li et al., 2016) in the simplified text when they are taken out-of-context during training and evaluation. This observation calls for the inclusion of document context during simplification (Sun et al., 2020), or performing decontextual-ization (Choi et al., 2021) before simplifying. Table 6 shows the distributions of insertion, deletion, and substitution errors annotated in system outputs. 2 It also shows the standard simplification evaluation metricSARI scores (Xu et al., 2016) for the annotated set. For the three models that 2 DRESS only released their Wikilarge outputs; ControlTS had different data splits for Newsela. reported both Wikilarge and Newsela outputs, the relative frequency of deletion errors between the two datasets appears to be preserved in model outputs, though for the RNN models errors are milder on Newsela and amplified on Wikilarge. A clear relationship between dataset and system output distributions does not exist for insertion and substitution errors. For Dress and EditNTS , this is due to the fact that the minor differences in insertion errors are dwarfed by the larger number of -1 (gibberish) labels assigned to Newsela outputs. Interestingly, outputs from the T5 model were rarely labeled as -1 errors, so the difference in insertion errors is more apparent. In the case of substitution, the Newsela outputs for Dress and T5 models show much higher rates of substitution errors than the Wikilarge outputs, despite the opposite being true for the datasets themselves. EditNTS does not show the same pattern, but again, the high rate of -1 errors subsumes every other trend. One possible reason for this phenomenon could be that the higher abstractiveness of Newsela encourages models to rewrite the input sentence to a greater extent and destroy the original meaning in the process. In general the models produce substitution errors more frequently than are found in the dataset, meaning that they are introduced by the models themselves and not merely learned from the data. Model comparisons There are a few differences in error distributions between the RNN-based and Transformer-based models, and between pretrained vs. non-pretrained Transformer models. All three Transformer models have less severe deletion errors than the RNN models on Wikilarge, and in addition T5 has lower deletion error rates on Newsela. Perhaps the most striking trend is that the Transformer models have far lower -1 gibberish errors than RNN-based models, even Access , which is not pre-trained on the language modeling task. T5 which has been pre-trained on large amounts of dataproduced more insertion errors, while Access produced more substitution errors. Quantitative Analysis We explore the relationships between the factuality annotations of system outputs and both length reduction and normalized edit distance. We briefly describe our findings here and defer numerical details to Appendix C. For every model except Access , there is a clear positive correlation between the severity of deletion errors and the degree of length reduction between the complex input and generated simplification. This is consistent with the trend observed for the datasets. No consistent relationships between length change and levels of insertion and substitution errors are exhibited by the system outputs. As in the case of length reduction, mean edit distances increase with the severity of deletion error with no consistent trends found for insertion and substitution labels. Qualitative analysis We also manually inspect model outputs, detailed in Appendix D, and summarize main observations here. As in the data, models also produce deletions ranging from single words and short phrases to clauses. For the two RNN models, DRESS and EditNTS , level 1 errors primarily consist of shorter deletion errors, which include pronoun errors and modifiers. Level 2 errors are almost always longer deletions, yet we did not observe the promotion of a subordinate clause to a main one as in the references, suggesting that models tend to follow syntactic rules more strictly. For T5 , we additionally observe level 2 errors in which the model deletes a semantically critical word. We observed more error variability in the other two transformer models, Access and ControlTS . Models introduced varying numbers of insertion and substitution errors, but in inspection we did not observe any clear properties of these as a function of model type. Relationship to SARI. SARI is the most popular metric used to evaluate text simplification models (Xu et al., 2016). For each model, we report Spearman's rank correlation coefficient (Spearman, 1904) between SARI and each error category. As Table 7 reports, there is only a weak correlation between SARI and the prevalence of information errors, and both the direction and magnitude of the correlation are highly dependent on model and dataset. This lack of correlation is unsurprising since SARI uses lexical overlap between the generated text with the reference text pair to judge simplification quality. This parallels the case with ROUGE in summarization (Falke et al., 2019a; Maynez et al., 2020; Wallace et al., 2021). Measures of Semantic Similarity. Many existing text simplification systems attempt to address the problem of meaning preservation by using a semantic similarity score either directly in their loss/reward function or in a candidate ranking step (Zhang and Lapata, 2017; Kriz et al., 2019; Zhao et al., 2020; Maddela et al., 2021). Additionally, some of these metrics have been included in recent factuality evaluation platforms in summarization (Pagnoni et al., 2021). We explore the extent to which existing similarity methods detect 7337 Similarity Measure I D S Jaccard Similarity  0 . information errors as outlined in our annotation scheme. We consider: (1) Jaccard similarity; (2) cosine similarity between averaged GloVe (Pen-nington et al., 2014) or ELMo (Peters et al., 2018) embeddings of the original and simplified sentences; (3) cosine similarity between Sentence-BERT (Reimers and Gurevych, 2019) embeddings; and (4) BERTScore (Zhang et al., 2019). As Table 8 indicates, the semantic similarity measures explored capture deletion errors quite well, while being a moderate indicator of insertion errors and a very weak one for substitution errors. Since deletion and substitution errors are common in most of the models we evaluated, the results indicate that better methods are needed to detect unacceptable deletions and intrinsic hallucinations in simplification outputs. Measures of Factuality. As in text simplification, the most common evaluation metrics used in text summarization like ROUGE do not adequately account for the factuality of model generations with respect to the input texts (Kryscinski et al., 2019). For this reason, recent works have proposed model-based metrics to automatically assess factuality (Falke et al., 2019b; Durmus et al., 2020; Wang et al., 2020b; Kryscinski et al., 2020b; Goyal and Durrett, 2020). We consider the following systems: (1) FACT-CC, which is a BERT-based model trained on a synthetic dataset to classify text pairs as being factually inconsistent or not (Kryscin-ski et al., 2020b), and (2) DAE, which is another BERT-based model that classifies each dependency arc in the model output as entailing the source text or not (Goyal and Durrett, 2020). More specifically, for FACT-CC we use the model's probability that each simplification example is inconsistent. For DAE we use the average of the lowest k probabilities that a dependency arc in the target sentence does not entail the source for k = 1 , 3 , 5 . semantic similarity like Jaccard similarity, though DAE scores correlate better with substitution errors than do FACT-CC and all evaluated measures of semantic similarity. Since manual annotation is costly and time-consuming, as a first step towards large-scale evaluation, we present an initial attempt at automating factuality assessment by training a model on human annotations. To supplement training, we explore methods of generating synthetic data to improve model performance. We framed automatic factuality assessment as a classification task in which a separate classifier is trained for each category (Insertion, Deletion, and Substitution), for each of the levels 0, 1, and 2. We treat the annotations used in our previous analyses as the test set and have additional data annotated to function as the training set for this task. We therefore collected a total of 1004 additional examples annotated across Wikilarge, Newsela, Access outputs on Wikilarge, and T5 outputs on Newsela and Wikilarge. We fine-tuned RoBERTa (Liu et al., 2019) with a classification head. Synthetic Data Generation As Table 10 indicates, the validation dataset is both small and highly imbalanced, with very few level 2 insertion and substitution errors. To alleviate this issue, we experimented with a few methods of generating synthetic insertion and substitution errors on which to pretrain the model. We accomplished this by modifying each of the complex sentences in the validation set. To generate insertion errors, we replace names with pronouns and remove phrases from the source text to create target texts (informa-tion deletions) and then swap the source and target to produce information insertions. To generate substitutions, we change numbers in the source text, negate statements, and used BERT masking to perturb information in the sentence. We generated 10K examples in total; Appendix E.1 describes these 7338 Level 0 Level 1 Level 2 Category # F1 # F1 # F1 Insertion 823 87.9 104 36.6 40 30.4 Deletion 413 84.2 356 57.1 204 52.1 Substitution 810 82.7 110 19.8 33 9.5 Table 10: Annotated label counts in the training set, and F1 on the test set. Training and Evaluation The model is evaluated using the F1-scores with respect to each class (0,1,2), and when selecting checkpoints during training, the average of the label 1 and 2 F1 scores is used. The deletion model was trained directly on its training data, whereas the insertion and substitution models were initially pretrained on the synthetic datasets. Training details are provided in Appendix E.3. Results Table 10 shows the test F1 scores achieved by the three classifiers. As expected, the deletion classifier achieved the best 1 and 2 F1 scores, likely due to the fact that the training dataset had plenty of level 1 and 2 deletion errors. Although the insertion and substitution datasets are similarly skewed, the insertion classifier significantly outperforms the substitution one. We found that using synthetic data is useful: without it, F1s for levels 1 and/or 2 are near 0 for insertion and substitution. Even with data augmentation, however, detecting errors is a challenging task. We have presented an evaluation of the factuality of automated simplification corpora and model outputs, using an error typology with varied degrees of severity. We found that errors appear frequently in both references and generated outputs. In the datasets, deletion errors are quite frequent, with Newsela containing more than Wikilarge. The system outputs indicate that the models also tend to delete information, which is likely a behavior learned from the training data. Model outputs contain more substitution errors than the datasets, so that behavior is probably a model bias rather than something picked up from the data. Although we examined the two commonly used sentence-level datasets, factuality errors do extend to other domains and larger units of text. Our initial analysis of factuality in medical text simplification (Devaraj et al., 2021) found errors of all three types, an indication that factual simplification is an open problem in such high-stake areas. The details of our analysis are in Appendix F. We also found that factuality errors are not well captured by existing metrics used in simplification such as SARI (Xu et al., 2016). While semantic similarity metrics correlate with deletion errors, they poorly correlate with insertion or substitution. We further present an initial model for automatic factuality assessment, which we demonstrate is a challenging task. This work was partially supported by NSF grants IIS-1850153, IIS-2107524, IIS-1901117, as well as the National Institutes of Health (NIH), grant R01-LM012086. We also acknowledge the Texas Advanced Computing Center (TACC) at UT Austin for providing the computational resources for many of the results within this paper. We are grateful to the anonymous reviewers for their comments and feedback.
abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain objective objective objective objective objective objective objective objective objective objective objective abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain result method abstain result abstain method objective other other other abstain other other other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain abstain method method method abstain abstain method method method method method abstain abstain abstain abstain abstain abstain method abstain method result abstain abstain abstain objective result result abstain objective other other other