In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced. Modern Electronic Health Records (EHR) systems require clinicians to keep a thorough record of every patient interaction and management decision. While this creates valuable data that may lead to better health decisions, it also significantly increases the burden on the clinicians, with studies showing this is a major contributor to burnout (Arndt et al., 2017). In most primary healthcare practices, the universal record of a clinician-patient interaction is the SOAP (Subjective, Objective, Assessment, Plan) note, which captures the patient's history, and the clinician's observations, diagnosis, and management plan (Pearce et al., 2016). At the end of a consultation, the clinician is required to write up a SOAP note of the encounter. With the exception of the clinician's internal observations on how the patient looks and feels, most of the SOAP note is verbalised and could be automatically constructed from the transcript of the consultation. A number of recent studies (Enarvi et al., 2020; Joshi et al., 2020; Zhang et al., 2021a) propose using summarisation systems to automatically generate consultation notes from the verbatim transcript of the consultationa task henceforth referred to as Note Generation. Yet, there is very limited work on how to evaluate a Note Generation system so that it may be safely used in the clinical setting. Where evaluations are present, they are most often carried out with automatic metrics; while quick and cheap, these metrics were devised for general purpose summarisation or machine translation, and it is unclear whether they work just as well on this new task. In the field of automatic summarisation and Natural Language Generation (NLG) in general, human evaluation is the gold standard protocol. Even in cases where the cost of using human evaluation is prohibitive, it is essential to establish the ground truth scores which automatic metrics should aim for. Our contributions are: (i) a large-scale human evaluation performed by 5 clinicians on a set of 285 consultation notes, (ii) a thorough analysis of the clinician annotations, and (iii) a correlation study with 18 automatic metrics, discussing limitations and identifying the most suitable metrics to this task. We release all annotations, human judgements, and metric scores. 1 2 Related Work Note Generation has been in the focus of the aca-demic community with both extractive methods (Moen et al., 2016b; Alsentzer and Kim, 2018), and with abstractive neural methods (Zhang et al., 2018; Liu et al., 2019; MacAvaney et al., 2019; Zhang et al., 2020; Enarvi et al., 2020; Joshi et al., 2020; Krishna et al., 2021; Chintagunta et al., 2021; Yim and Yetisgen-Yildiz, 2021; Moramarco et al., 2021; Zhang et al., 2021a). Whether these studies 1 https://github.com/babylonhealth/primock57 5739 Transcript Note Clinician Hello. discuss the generation of radiology reports, patient-nurse summaries, discharge summaries, or SOAP notes, they all deal with long passages of text in the medical domain. This is a critical distinction from other application contexts (e.g. news summarisa-tion): here, commonly used and well-studied evaluation criteria such as fluency', relevance', and adequacy' are superseded by other criteria, such as omissions of important negatives', mislead-ing information', contradictions', etc. In addition, common summarisation metrics such as ROUGE (Lin, 2004) or BertScore (Zhang et al., 2019) measure the standalone quality of outputs and are not typically evaluated against more extrinsic criteria, such as post-editing times. Of the 18 studies on the subject that we could identify, 13 present an automatic evaluation (typically based on ROUGE and sometimes on medical entity linking) and 12 carry out a small-scale intrinsic human evaluation. In particular, Moen et al. (2016a) employ three domain experts to review 40 generated notes with Likert scales along 30 criteria (including Long-term di-agnosis', Reason for admission', assessment'), but report that the subjects found the 30 item scale too difficult and detailed to assess. MacAvaney et al. (2019) use one domain expert to review 100 notes and report Likert scale values for Readabil-ity', Accuracy', and Completeness'. Moramarco et al. (2021) employ three clinicians and compare the times to post-edit generated notes with those of writing them from scratch, reporting that, while faster, post-editing may be more cognitively intensive than writing. Outside of the medical domain, our work is comparable to Fabbri et al. (2021), who run an automatic metrics correlation study for news article summaries for the CNN/DailyMail dataset (Nallap-ati et al., 2016). They also release code 2 for evaluating text with a suite of common metrics, some of which we include in our own list of metrics to evaluate. Our evaluation study is based on a dataset of 57 pairs of mock consultation transcripts and summary notes (Papadopoulos Korfiatis et al., 2022). 3 The data was produced by enacting consultations using clinical case cards. The clinicians that conducted the mock consultations also wrote the corresponding SOAP note. The consultations span common topics within primary healthcare and are about 10 minutes long. To mimic a live clinical environment, the audio 2 https://github.com/Yale-LILY/SummEval 3 The dataset is available at: https://github.com/ babylonhealth/primock57 5740 Figure 1: Diagram of the dataset creation and the four tasks involved in the human evaluation. of the consultations was transcribed with Google Speech-to-text engine 4 . These transcripts form the input to the Note Generation models. The aim is to generate the Subjective part of a SOAP note. Table 1 shows an example transcript and respective note. Figure 1 describes the creation of the dataset and how the data feeds into the human evaluation tasks described below. In a fashion similar to Chintagunta et al. (2021); Moramarco et al. (2021); Zhang et al. (2021a), we fine-tune 10 neural summarisation models based on BART (Lewis et al., 2020) on a proprietary dataset of 130,000 real consultation notes and transcripts. In accordance with our evaluation dataset, the training set consists of automatic Google Speech-to-text transcripts as inputs and the Subjective part of the corresponding notes as outputs. The base models are large BART architectures pretrained on the CNN/Dailymail dataset 5 . Since our focus is on evaluation, the aim was to obtain models which would produce different outputs to cover a wider range of errors. The differences between the models included: fine-tuning on different sized datasets; using pre-processing techniques such as filtering the transcripts for relevant sentences; and using post-processing techniques such as filtering the generated notes for irrelevant sentences. 4 https://cloud.google.com/speech-to-text 5 https://huggingface.co/facebook/bart-large-cnn 4 Human Evaluation Setup Under the supervision of one of the authors (a clinician expert in AI development henceforth referred to as the Lead Clinician) we design the following evaluation tasks: 1. Listen to the mock consultation audio and take notes ( eval_notes ). These eval_notes appear on the evaluator screen throughout to help reduce the cognitive load of remembering what was discussed in the consultation. 2. Relying on the eval_notes and the consultation audio, read 5 different notes and post-edit each one of them. Post-editing consists of correcting an imperfect note to produce a factually accurate and relevant note (Sripada et al., 2005). It mimics how a synthetic note could be used in clinical practice while also bootstrapping the error identification (Mora-marco et al., 2021). For this purpose, the evaluation platform includes a track-changes interface, which highlights insertions and deletions (Figure 2), and records the time taken to post-edit. 3. For each note, classify the errors into two categories: incorrect statements' and omissions', by copying the spans of text from the post-editing interface and pasting them in the appropriate table (as in Figure 5741 Figure 2: Screenshot of the post-editing task where the evaluator is correcting a note with the track-changes interface. 3). We define incorrect statements' as sentences in the generated notes which contain one or more factual errors (compared to the consultation audio). Conversely, omissions' are medical facts which should be recorded in a consultation note and were omitted by the model. Examples and edge cases (which were given to the evaluators for training) can be found in the Appendix, Figure A.4. Each error is also tagged as critical' if the information contained has essential clinical importance. Specifically, if the error would lead to medico-legal liability. 4. Report any qualitative feedback (e.g. regarding order of statements, repetition) in the Other issues' box. Figure 1 (bottom half) shows a diagram of the human evaluation workflow. The subjects of the study were 5 regularly practising clinicians (GPs) with a minimum of 3 years experience. As part of our ethical consideration, all clinicians were paid the UK standard GP working rate and were free to cease participation at any time if they wished. For diversity and inclusion, 2 male clinicians and 3 female clinicians were enlisted from a range of ethnic backgrounds. Following the tasks described above, each clinician evaluated the entire dataset of 57 mock consultations. Each consultation included 5 notes to evaluate, 4 of which were sampled from our 10 models and 1 was written by the consulting doctor ( human_note ). We shuffled these for every consultation andto avoid biasesdid not specify that one of the notes was not synthetic. The evaluation study took circa 30 working hours per evaluator to complete over a period of 8 weeks. Before commencing, each evaluator went through a training and practice process conducted by the Lead Clinician, who explained the evaluation interface and guided them through the annotation of a practice note. A copy of the evaluator instructions can be found in Appendix A. Throughout the study, the authors and the Lead Clinician held weekly sessions with each evaluator where we shadowed the evaluation tasks through screen sharing. This helped us understand the difficulties in performing the tasks while ensuring the evaluators followed the guidelines set out for them. The result of the human evaluation consists of 285 evaluator notes (57 consultations x 5 evaluators), 1,425 post-edited notes (285 x 5 notes per con-sultation), post-editing times, count and spans of incorrect statements, count and spans of omissions, whether they are critical, and qualitative comments. When compared with more common evaluation approaches such as Likert scales and ranking methods, we believe our set-up provides a more granular and more interpretable set of judgements, albeit at the price of lowering the inter-annotator agreement. To compensate for this, the 5 evaluators annotate the same 57 tasks (Sheng et al., 2008) and the scores Criterion Human Generated Post-edit times 96.5s 136.4s Number of Incorrect 1.3 3.9 Number of Omissions 3.9 6.6 Note length 16.9 21.5 Table 3: Aggregated judgements for human notes and generated notes. 6). As shown in Table 2, we compute inter-annotator agreement on the post-editing times, incorrect statements, and omissions. The absolute post-editing times are converted to a list of rankings for each evaluator, and agreement is computed with Krippendorff's alpha (Krippendorff, 2018) with ordi-nal' level of measurement. This ensures only the ranking of each note is captured in the agreement and not the editing speed of each evaluator. For example, where evaluator 1 takes 60 seconds and 120 seconds to post-edit two given notes and evaluator 2 takes 180 seconds and 240 seconds respectively, their agreement would be perfect because they both agreed that note 1 is quicker to edit than note 2. Conversely, for incorrect statements and omissions we calculate interval' Krippendorff's Alpha on the counts of errors identified by the evaluators. As the counts don't ensure that two evaluators have selected the same statements, we also compute word overlap F1 score as suggested by Popovic and Belz (2021). As shown in Table 2, the agreements for times and incorrect statements are not very strong (Krippendorff (2018) indicate that   0 . 667 is the lowest conceivable limit). We investigate the source of disagreement and attribute it to two main factors: (i) human error due to the difficulty inherent in the task, and (ii) stylistic differences in note writing. Examples of human error can be found in subsection 5.2. As for stylistic differences, these are especially evident in the Omissions category, where some clinicians are thorough in their note taking and others only document the most important facts. See Appendix B for more details on pairwise agreement. To compare the accuracy of the models against human-written notes, we average all the judgements for our criteria (post-edit times, incorrect statements, and omissions), aggregate by the gener-5743 ated notes and the human_notes respectively, and report the results in Table 3. As expected, the hu-man_notes performed better for all criteria; in particular, they contain fewer omissions while being on average 4.6 sentences shorter. However, the evaluators found imperfections in human notes too: it takes over 1.5 minutes on average to read and post-edit a human_note , and it contains over 1 incorrect statement and almost 4 omissions on average. While the omissions can be reconciled as stylistic differences among evaluators, the incorrect statements are potentially more impactful. To investigate, we select two human notes and ask the Lead Clinician to post-edit them, comparing the results with those of the evaluators. In the first case, the Lead Clinician agrees with the evaluators in that the human note contains the following two incorrect statements: Inc. statement Correction Also vomiting  mainly bilous Also vomiting Wife and children also unwell with vomiting, but no diarrhea. Upon inspecting the consultation recording, the Lead Clinician found that the word bilious' was not stated by the patient. However, the consulting clinician may have used this term due to a personal habitual documentation style (as clinically, vomit with no red flags can conventionally be referred to as bilious). The words Wife and children also unwell with vomiting, but no diarrhea' were not stated by the patient. Instead, the patient made a tangential statement summarised here: One child had some vomiting but no other symptoms in wife and other child.' Therefore, it is inferred that this clinician likely made a normal human error due to excessive patient detail (non-critical). In the second case, the Lead Clinician found no issues with the human note. Upon inspecting the corrections from the evaluators, he concluded that what they selected as incorrect statements were medical conditions inferred by the consulting clinician yet not specifically stated by the patient. We highlight this to show that it is unclear whether the task has a single ground truth, as even human experts don't completely agree; well thought-out evaluation tasks can mitigate this and produce one or more good ground truth approximations. Detailed examples can be found in Appendix C. Criterion 1 Criterion 2 Pears. To understand the interdependence between our criteria, we compute Pearson's correlation (Freedman et al., 2007) and Spearman's rank correlation (Zar, 2005) coefficients between each pair. Table 4 shows a moderately strong correlation between the time it takes to post-edit a note and the number of incorrect statements it contains. The correlation between post-edit times and omissions is stronger, which could be explained by the fact that it takes longer to type an omitted statement than to delete or edit an incorrect one. Finally, the correlation between post-edit times and incor-rect+omissions is strong, which suggests that post-edit times is a function of the number of edits and that one of these criteria could be a proxy for the other. We also compute the correlation between each criterion and the length of the generated note. These numbers can be used as a benchmark for automatic metric correlation; for example, the 0.413 Spearman's correlation between post-edit times and note length indicates that any automatic metric needs to surpass this value in order to be more useful than simply counting the number of sentences in the note. As introduced in Section 4, the evaluators provide qualitative feedback about the generated notes in the Other Issues' field. When analysing these comments, a number of repeated patterns emerged, highlighting common pitfalls in the generated notes. Based on these we defined a small taxonomy (Table 5; issues in the human_notes are excluded), providing examples and occurrences of each issue type. Aside from incorrect statements and omissions, the most significant issues revolve around repetition, 5744 Issue Examples Occ. disjointed notes, and contradiction. Upon investigating, we believe that all three are related to the tendencies of the models to generate the consultation note following the chronological order of the transcript. While that is an intuitive behaviour, consultations are seldom carried out in the order of SOAP note sections (Subjective, Objective, Assessment, Plan), with the patient providing relevant information whenever they can, sometimes after the clinician has discussed assessment and plan. Borrowing from the field of Automatic Summarisation, most studies on Note Generation rely on ROUGE and fact-extraction based metrics to evaluate the generated notes (Section 2 for more details). While some studies carry out a small human evaluation, there is little effort to investigate whether the scores from ROUGE or the other metrics employed correlate well with the human judgements, especially extrinsic criteria such as post-edit times. However, scores from these metrics are featured on leaderboards 6 for summarisation tasks, driving future research. To address this, we carry out a correlation study of automatic metrics for the task of Note Generation. A total of 18 automatic metrics are tested against statistics produced by the human judgements of our criteria: post-edit times, number of incorrect statements, and number of omissions. Following the taxonomies reported by Celikyilmaz et al. (2020) and Sai et al. (2020), the metrics considered can be loosely grouped in:  Text overlap metrics. These are based on string matching, whether character based, word based, or n-gram based. Some use stemming, synonyms, or paraphrases. They include: ROUGE (Lin, 2004), CHRF (Popovic, 2015), METEOR (Lavie and Agarwal, 2007), 6 https://nlpprogress.com/english/summarization.html 5745 Criterion: Post-edit times Inc+Omi Incorrect Omissions Reference: human edited eval avg max avg max avg max avg max ROUGE-1-F1 * 0.334 0.627 0.160 0.443 0.550 0.580 0.704 0.378 0.505 0.561 0.651 ROUGE-2-F1 * 0.384 0.653 0.166 0.551 0.570 0.694 0.731 0.501 0.557 0.641 0.654 ROUGE-3-F1 * 0.366 0.645 0.117 0.576 0.565 0.734 0.731 0.555 0.568 0.663 0.646 ROUGE-4-F1 * 0.342 0.632 0.076 0.575 0.557 0.745 0.726 0.581 0.573 0.661 0.636 ROUGE-L-Pr * 0.348 0.471 0.169 0.366 0.427 0.500 0.613 0.607 0.745 0.306 0.375 ROUGE-L-Re * 0.409 0.614 0.300 0.520 0.551 0.640 0.680 0.374 0.416 0.660 0.688 ROUGE-L-F1 * 0.384 0.646 0.285 0.538 0.564 0.661 0.719 0.479 0.534 0.610 0.655 CHRF * 0.341 0.460 -0.075 0.463 0.438 0.581 0.560 0.504 0.484 0.483 0.462 METEOR * 0.415 0.667 0.203 0.529 0.581 0.674 0.713 0.429 0.463 0.668 0.699 BLEU * 0.382 0.642 0.098 0.557 0.565 0.698 0.702 0.447 0.453 0.685 0.686 Levenshtein dist. 0.547 0.780 0.453 0.600 0.654 0.650 0.760 0.566 0.555 0.531 0.697 WER 0.239 0.629 0.059 0.326 0.550 0.412 0.704 0.499 0.535 0.252 0.631 MER 0.392 0.635 0.156 0.565 0.557 0.703 0.706 0.500 0.513 0.659 0.651 WIL 0.394 0.649 0.117 0.590 0.566 0.747 0.723 0.578 0.566 0.668 0.638 ROUGE-WE * 0.402 0.624 0.165 0.496 0.549 0.621 0.712 0.415 0.524 0.595 0.650 SkipThoughts * 0.298 0.403 -0.067 0.229 0.375 0.366 0.504 0.338 0.407 0.288 0.439 Embedding Avg * 0.266 0.375 -0.209 0.064 0.412 0.223 0.572 0.147 0.392 0.211 0.543 VectorExtrema * 0.409 0.553 0.127 0.424 0.500 0.550 0.648 0.367 0.468 0.531 0.600 GreedyMatching * 0.308 0.577 -0.041 0.295 0.520 0.436 0.670 0.281 0.479 0.428 0.624 USE * 0.339 0.522 0.201 0.366 0.476 0.474 0.637 0.327 0.452 0.454 0.598 WMD 0.354 0.594 0.154 0.421 0.529 0.561 0.670 0.319 0.414 0.577 0.670 BertScore * 0.497 0.688 0.340 0.571 0.590 0.710 0.744 0.530 0.552 0.645 0.676 MoverScore * 0.360 0.640 0.246 0.570 0.559 0.687 0.688 0.448 0.467 0.669 0.657 Stanza+Snomed * 0.334 0.508 0.118 0.354 0.460 0.528 0.643 0.447 0.533 0.449 0.553 Table 6: Spearman's correlation coefficients for each metric and each criterion. and BLEU (Papineni et al., 2002). Edit distance metrics. These count the number of character or word level transformations required to convert the system output into the reference text. They include: Levenshtein edit distance (Levenshtein et al., 1966), WER (Su et al., 1992), MER and WIL (Morris et al., 2004). Embedding metrics, including word-level, byte-level, and sentence-level embeddings. These metrics encode units of text with pretrained models and compute cosine similarity between them. They include: ROUGE-WE (Morris et al., 2004), SkipThoughts, Embed-dingAverage, VectorExtrema (Forgues et al., 2014), GreedyMatching (Sharma et al., 2017), USE 7 (Cer et al., 2018), WMD (Kusner et al., 7 Cosine similarity between the reference and the hypothesis embeddings. Embeddings computed with Universal Sentence Encoder. 2015), BertScore (Zhang et al., 2019), and MoverScore (Zhao et al., 2019). Fact-extraction. The Stanza+Snomed metric extracts medical concept spans with Stanza (Zhang et al., 2021b), then uses similarity measures to map them to entities in the SNOMED CT clinical ontology (Spackman et al., 1997). The metric computes F1 score between reference and hypothesis over the set of extracted entities. For more details on each metric please refer to their respective papers. All these metrics attempt to measure the accuracy of the generated text by comparing it against a reference text. Our human evaluation study produces three distinct human-curated notes which can be used as reference: the human_note is the original note, written by the consulting clinician (and also one of the hypotheses), the eval_note is the note written by the evaluators after listening to the consultation audio, and 5746 the edited_note is the generated note after being post-edited by the evaluators. Table 6 reports the correlation coefficients. When correlating against post-edit times, we consider each reference text ( human_note , edited_note , eval_note ) separately, then take the average and the maximum of the metric scores for each reference. For count of incorrect statements, omissions, and incorrect+omissions we only report the average and the maximum scores, taking all three references into account as commonly done by the metrics that support multiple references (e.g. BLEU, ROUGE, METEOR, BertScore). We compute Pearson's and Spearman's coefficients and, upon finding similar patterns, only report Spearman's coefficients in Table 6. The Pearson's coefficients can be found in Table A.8 in the Appendix. As shown in Table 6, all metrics display a strong bias towards the choice of reference. In particular, the correlation scores with the edited_note as reference are much higher than those of either hu-man_note or eval_note . As the edited_note is a transformation of the generated note (refer to Figure 2), these high correlations show how reliant all the metrics are on the surface form of the text. The significant difference between taking human_note and eval_note as reference can be traced to two main factors: (i) the human_note is unique per consultation so the human judgements are averaged across evaluators (reducing noise and collapsing disagreement), and (ii) the eval_note was not written to replace a SOAP note but is rather a list of the most salient points in the consultation, and sometimes contains more information than would typically be detailed in a SOAP note. The top three metrics in most scenarios are Levenshtein distance, BertScore, and METEOR. While METEOR and BertScore are established metrics in NLG evaluation, Levenshtein distance is not typically used as a metric in long-form text evaluation. From a semantic point of view, edit distance has the least amount of knowledge and should be very brittle when comparing text that is meaningfully similar but lexically very different. Yet Levenshtein distance has the highest correlation even when the reference is the eval_note , which is syntactically very different from the generated note; whereas even contextual metrics like BertScore perform more poorly. A possible explanation for this behaviour may be that our post-editing times and count of incorrect statements/omissionsunlike Likert scales scoresmeasure the amount of work required to convert a synthetic note into a factually-correct and relevant note, just as Levenshtein distance measures the character-level distance between the synthetic note and the reference. We notice that all the metrics correlate better with counts of incorrect+omissions than with post-edit times, despite the two criteria being strongly correlated with each other (0.829 Spearman's correlation, see Table 4). We believe this is due to post-editing times containing more noise and capturing more of the stylistic differences between evaluators than the number of errors does. We conducted a human evaluation study for the task of consultation Note Generation, computed agreement between evaluators, and quantified the extent to which human error impacts the judgements. We then carried out a correlation study with 18 automatic metrics, discussing their limitations and identifying the most successful ones. We found that the choice of human reference has a significant effect on all automatic metrics and that simple character-based metrics like Levenshtein distance can be more effective than complex model-based metrics for the task of Note Generation. Based on our findings, character-based Levenshtein distance, BertScore, and METEOR are the most suitable metrics to evaluate this task. We release all the data and annotations and welcome researchers to assess further metrics. The authors would like to thank Rachel Young and Tom Knoll for supporting the team and hiring the evaluators, Vitalii Zhelezniak for his advice on revising the paper, and Kristian Boda for helping to set up the Stanza+Snomed fact-extraction system.
abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain other other abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method result method abstain other