Following each patient visit, physicians draft long semi-structured clinical summaries called SOAP notes. While invaluable to clinicians and researchers, creating digital SOAP notes is burdensome, contributing to physician burnout. In this paper, we introduce the first complete pipelines to leverage deep summarization models to generate these notes based on transcripts of conversations between physicians and patients. After exploring a spectrum of methods across the extractive-abstractive spectrum, we propose CLUSTER 2S ENT , an algorithm that (i) extracts important utterances relevant to each summary section; (ii) clusters together related utterances; and then (iii) generates one summary sentence per cluster. CLUSTER 2S ENT outperforms its purely abstractive counterpart by 8 ROUGE-1 points, and produces significantly more factual and coherent sentences as assessed by expert human evaluators. For reproducibility, we demonstrate similar benefits on the publicly available AMI dataset. Our results speak to the benefits of structuring summaries into sections and annotating supporting evidence when constructing summarization corpora. Electronic health records (EHR) play a crucial role in patient care. However, populating them can take as much time as attending to patients (Sinsky et al., 2016) and constitutes a major cause of physician burnout (Kumar and Mezoff, 2020). In particular, doctors document patient encounters with SOAP notes, semi-structured written accounts containing four sections: (S)ubjective information reported by the patient; (O)bjective observations, e.g., lab results; (A)ssessments made by the doctor (typically, the diagnosis); and a (P)lan for future care, including diagnostic tests, medications, and treatments. Sections can be subdivided into 15 subsections. In a parallel development, patients increasingly record their doctor's visits, either in lieu of taking notes or to share with a family member. A budding line of research has sought to leverage transcripts of these clinical conversations both to provide insights to patients and to extract structured data to be entered into EHRs (Liu et al., 2019b; Schloss and Konam, 2020; Krishna et al., 2021). In this paper, we introduce the first end-to-end methods for generating whole SOAP notes based on clinical conversations. Our work builds on a unique corpus, developed in collaboration with Abridge AI, Inc. 1 ), that consists of thousands of transcripts of recorded clinical conversations together with associated SOAP notes drafted by a work force trained in the official style of SOAP note documentation. On one hand, this task is much harder than traditional summarization benchmarks, in part, because SOAP notes are longer (320 words on average) than summaries in popular datasets like CNN/Dailymail (Nallapati et al., 2016), Newsroom (Grusky et al., 2018), and SamSum (Gliwa et al., 2019) (55, 27, and 24 words on average). On the other hand, our dataset offers useful structure: (i) segmentation of each SOAP note into subsections; and (ii) a set of supporting utterances that provide evidence for each sentence in the SOAP note. Exploiting this structure, our methods outperform appropriate baselines. Our first methodological contribution is to propose a spectrum of methods, for decomposing sum-marizaton tasks into extractive and abstractive subtasks. Starting from a straightforward sequence-to-sequence model, our methods shift progressively more work from the abstractive to the extractive component: (i) CONV 2N OTE : the extractive module does nothing, placing the full burden of summarization on an end-to-end abstractive module. (ii) 1 http://abridge.com ...DR: So are you taking the Monteluekast EXT 2N OTE : the extractive module selects all utterances that are noteworthy (i.e., likely to be marked as supporting utterances for at least one SOAP note sentence), and the decoder is conditioned only on these utterances; (iii) EXT 2S EC : the extractive module extracts per-subsection noteworthy utterances and the decoder generates each subsection, conditioned only on the corresponding utterances; (iv) CLUSTER 2S ENT : the extractive module not only extracts per-subsection noteworthy utterances but clusters together those likely to support the same SOAP sentencehere, the decoder produces a single sentence at a time, each conditioned upon a single cluster of utterances and a token indicating the SOAP subsection. We see consistent benefits as we move from approach (i) through (iv). Both to demonstrate the generality of our methods and to provide a reproducible benchmark, we conduct parallel experiments on the (publicly available) AMI corpus (Carletta, 2007) 2 Like our medical conversations dataset, the AMI corpus exhibits section-structured summaries and contains annotations that link summary sentences to corresponding supporting utterances. Our experiments with AMI data show the same trends, favoring pipelines that demand more from the extractive component. These results speak to the wider usefulness of our proposed approaches, EXT 2S EC and CLUSTER 2S ENT , whenever section-structured summaries and annotated evidence utterances are available. Our best performing model, CLUSTER 2S ENT (Figure 1), demands the most of the extractive module, requiring that it both select and group each subsection's noteworthy utterances. Interestingly, we observe that given oracle (per-subsection) noteworthy utterances, a simple proximity-based 2 Our code and trained models for the AMI dataset: https://github.com/acmi-lab/ modular-summarization clustering heuristic leads to similar performance on SOAP note generation as we obtain when using ground-truth clusterseven though the ground truth noteworthy utterances are not always localized. Applied with predicted noteworthy utterances and clusters, this approach achieves the highest ROUGE scores and produces the most useful (fac-tual, coherent, and non-repetitive) sentences as rated by human experts. As an additional benefit of this approach, due to the smaller lengths of the input and output sequences involved, we can feasibly train large transformer-based abstractive summarization models (e.g., T5), whose memory requirements grow quadratically with sequence length. Additionally, our approach localizes the precise utterances upon which each SOAP note sentence depends, enabling physicians to verify the correctness of each sentence and potentially to improve the draft by highlighting the correct noteworthy utterances (versus revising the text directly). In summary, we contribute the following:  The first pipeline for drafting entire SOAP notes from doctor-patient conversations. A detailed human study to evaluate the factuality and quality of generated SOAP notes, and qualitative error analysis. A new collection of extractive-abstractive approaches for generating long section-segmented summaries of conversations, including new methods that leverage annotations attributing summary sentences to conversation utterances. A rigorous quantitative evaluation of our proposed models and appropriate baselines for both the extractive and abstractive components, including sensitivity of the pipeline to simulated ASR errors. Summarization is a well-studied problem in NLP (Nenkova et al., 2011). While early works focused on simply extracting important content from a document (Erkan and Radev, 2004; Wong et al., 2008), later approaches attempted to paraphrase the content into new sentences (abstractive summarization) (Filippova, 2010; Berg-Kirkpatrick et al., 2011; Wang and Cardie, 2013). Following the development of neural sequence models (Sutskever et al., 2014), more research focuses on neural generation of abstractive summaries (Nallapati et al., 2016; See et al., 2017; Celikyilmaz et al., 2018). While many papers summarize news articles, others summarize conversations, in business meetings (Wang and Cardie, 2013; Zhu et al., 2020), customer service (Liu et al., 2019a), and tourist information center (Yuan and Yu, 2019) contexts. In the space of two-step extractive-abstractive summarization approaches, Subramanian et al. (2019) summarize scientific papers by first extracting sentences from it and then abstractively summarizing them. Chen and Bansal (2018) extract important sentences from the input and then paraphrase each of them to generate the abstractive summary. While they assume that each summary sentence is supported by exactly one source sentence, in our medical conversations, many summary sentences synthesize content spread across multiple dialogue turns (e.g., a series of questions and answers). Past work on abstractive summarization of medical conversations has focused on summarizing patient-nurse conversations with goals including capturing symptoms of interest (Liu et al., 2019c) and past medical history (Joshi et al., 2020). These tasks are respectively similar to generating the review of systems and past medical history subsections of a SOAP note. In contrast, we aim to generate a full-length SOAP note containing up to 15 subsections, and propose methods to address this challenge by extracting supporting context for smaller parts and generating them independently. We use two different datasets in this work. The primary medical dataset, developed through a collaboration with Abridge AI, consists of doctor-patient conversations with annotated SOAP notes. Additionally, we evaluate our summarization methods on the AMI dataset (Carletta, 2007), comprised of business meeting transcripts and their summaries. Our work builds on a unique resource: a corpus consisting of thousands of recorded English-language clinical conversations, with associated SOAP notes created by a work force trained in SOAP note documentation standards. Our dataset consists of transcripts from real-life patient-physician visits from which sensitive information such as names have been de-identified. The full medical dataset consists of 6862 visits consisting of 2732 cardiologist visits, 2731 visits for family medicine, 989 interventional cardiologist visits, and 410 internist visits. Owing to the sensitive nature of the data, we cannot share it publicly (an occupational hazard of research on machine learning for healthcare). For each visit, our dataset contains a human-generated transcript of the conversation. The transcript is segmented into utterances, each annotated with a timestamp and speaker ID. The average conversation lasts 9 . 43 minutes and consists of around 1 . 5 k words (Appendix Figure A1). Associated with each conversation, we have a human-drafted SOAP note created by trained, professional annotators. The annotators who created the SOAP notes worked in either clinical transcription, billing, or related documentation-related departments, but were not necessarily professional medical scribes. The dataset is divided into train, validation and test splits of size 5770 , 500 and 592 , respectively. Our annotated SOAP notes contain (up to) 15 subsections, each of which may contain multiple sentences. The subsections vary in length. The Allergies subsections is most often empty, while the Assessment subsection contains 5 . 16 sentences on average (Table 1). The average SOAP note contains 27 . 47 sentences. The different subsections also differ in the style of writing. The Medications subsection usually consists of bulleted names of medicines and their dosages, while the Assessment subsection typically contains full sentences. On average, the fraction of novel (i.e., not present in the conversation) unigrams, bigrams, and trigrams, in each SOAP note are 24 . 09% , 67 . 79% and 85 . 22% , respectively. Each SOAP note sentence is also annotated with utterances from the conversation which provide evidence for that sentence. A SOAP note sentence can have one or more supporting utterances. On average, each SOAP sentence has 3 . 84 supporting utterances, but the mode is 1 (Appendix Figure A1). We refer to these utterances as noteworthy utterances Subsection Mean length Family Medical History 0.23 Past Surgical History 0.58 Review of Systems 3.65 Chief Complaint 2.17 Miscellaneous 2.81 Allergies 0.06 Past Medical History 2.93 Social History 0.27 Medications 3.74 Immunizations 0.11 Laboratory and Imaging Results 2.27 Assessment 5.16 Diagnostics and Appointments 1.65 Prescriptions and Therapeutics 1.75 Healthcare Complaints 0.09 Table 1: Average number of sentences in different SOAP note subsections grouped by parent sections ( Subjective , Objective , Assessment , Plan , Others resp.) throughout this paper. Throughout this work, we deal with the 15 more granular subsections rather than the 4 coarse sections of SOAP notes, and thus for convenience, all further mentions of section technically denote a SOAP subsection . The AMI dataset is a collection of 138 business meetings, each with 4 participants with various roles (e.g., marketing expert, product manager, etc.). Each meeting transcript comes with an associated abstractive summary that is divided into four sections abstract , decisions , actions , and problems . Each conversation also has an associated extractive summary, and there are additional annotations linking the utterances in the extractive summary to sentences in the abstractive summary. For any given sentence in the abstractive summary, we refer to the linked set of utterances in the extractive summary as its noteworthy utterances . We note that 7.9% of the abstractive summary sentences have no annotated noteworthy utterances. To simplify the analysis, we remove these sentences from summaries in the training, validation, and test splits. We investigate the following four decompositions of the summarization problem into extractive and abstractive phases, ordered from abstraction-heavy to extraction-heavy: CONV 2N OTE takes an end-to-end approach, generating the entire SOAP note from the entire conversation in one shot. EXT 2N OTE first predicts all of the noteworthy utterances in the conversation (without regard to the associated section) and then generates the entire SOAP note in one shot from only those utterances. EXT 2S EC extracts noteworthy utterances, while also predicting the section(s) for which they are relevant, and then generates each SOAP section separately using only that section's predicted noteworthy utterances. CLUSTER 2S ENT attempts to group together the set of noteworthy utterances associated with each summary sentence. Here, we cluster separately among each set of section-specific noteworthy utterances and then generate each section one sentence at a time, conditioning each on the associated cluster of utterances. Each of these pipelines leaves open many choices for specific models to employ for each subtask. For the abstractive modules of CONV 2N OTE and EXT 2N OTE , we use a pointer-generator network. The abstractive modules of EXT 2S EC and CLUSTER 2S ENT , which require conditioning on section are modeled using conditioned pointer-generator networks (described in Section 5), and fine-tuned T5 models which condition on the section being generated by means of prepending it to the input. T5 models could not be used in the CONV 2N OTE and EXT 2N OTE settings because their high memory requirement for long inputs could not be accommodated even with 48GB of GPU memory. For noteworthy utterance extraction, we primarily use a hierarchical LSTM model and a BERT-LSTM model as described in the next section. All models are configured to have a scalar output for binary classification in EXT 2N OTE , whereas for EXT 2S EC and CLUSTER 2S ENT , they have multilabel output separately predicting noteworthiness for each section. Note that the same utterance can be noteworthy with respect to multiple sections. We use the same trained utterance extraction models for both EXT 2S EC and CLUSTER 2S ENT . For the clustering module in CLUSTER 2S ENT , we propose a heuristic that groups together any two supporting utterances that are close , meaning they have less than or equal to  utterances separating them, where  is a hyperparameter. This process is iterated, with the clusters growing in size by merging with other singletons or clusters, until every pair of close utterances have the same cluster membership. The value of  is tuned on the validation set. Since each cluster necessarily produces one sentence in the SOAP note, having too many or too few clusters can make the SOAP note too long or too short, respectively. Therefore, for any given value of the hyper-parameter  and any given section, the prediction thresholds of the extractor are tuned on the validation set to produce approximately the same number of clusters over the entire validation set as present in the ground truth for that section. Among ground truth clusters containing multiple noteworthy utterances, 82% are contiguous. In an experiment where the heuristic is used to cluster the oracle noteworthy utterances for each section, and summaries are subsequently generated via the abstractive modules from CLUSTER 2S ENT , ROUGE-1 and ROUGE-2 metrics deteriorate by less than 1 point as compared to oracle clusterings (Appendix Table A3), demonstrating our heuristic's effectiveness. Pointer-Generator Network We use the pointer-generator network introduced by See et al. (2017) for CONV 2N OTE and EXT 2N OTE . The model is a bidirectional LSTM-based encoder-decoder model with attention. It employs a pointer mechanism to copy tokens directly from the input in addition to generating them by predicting generation probabilities for the entire vocabulary. The model also computes the weights that govern copying versus generating at each decoding timestep. Section-conditioned Pointer-Generator Network We modify the pointer-generator network for algorithms EXT 2S EC and CLUSTER 2S ENT , to condition on the (sub)section of the summary to be generated. The network uses a new lookup table to embed the section z into an embedding e z . The section embedding is concatenated to each input word embedding fed into the encoder. The section embedding is also appended to the inputs of the decoder LSTM in the same fashion. T5 We use the recently released T5 model (Raf-fel et al., 2020) as an abstractive module. It is an encoder-decoder model, where both encoder and decoder consist of a stack of transformer layers. The T5 model is pre-trained on 5 tasks, including summarization, translation etc. We use the pre-trained T5 model parameters and fine-tune it on our task dataset. For introducing the section-conditioning in EXT 2S EC and CLUSTER 2S ENT , we simply add the name of the section being generated to the beginning of the input. Hierarchical LSTM classifier(H-LSTM) In this model, we first encode each utterance u i independently by passing its tokens through a bidirectional LSTM and mean-pooling their encoded representations to get the utterance representation h i . We pass the sequence of utterance representations { h 1 , h 2 , ..., h n } through another bidirectional LSTM to get new utterance representations which incorporate neighboring contexts. These are then passed through a sigmoid activated linear layer to predict each utterance's probability of noteworthiness with respect to each section. BERT-LSTM classifier(B-LSTM) In this model, tokens in the utterance u i are passed through a BERT encoder to obtain their contextual-ized representations, which are mean-pooled to get the utterance representation h i . The subsequent architecture exactly mirrors hierarchical LSTM, and involves passing utterance representations through a bidirectional LSTM and linear layer to get output probabilities. BERT-LSTM is fine-tuned in an end-to-end manner. We first establish two baselines. RANDOMNOTE randomly and uniformly samples a SOAP note from the training set and outputs it as the summary for any input conversation. ORACLEEXT presents all the ground truth noteworthy utterances (evidence) from the conversation as the SOAP note without any abstractive summarization. Thus, the ORACLEEXT baseline has the advantage of containing all the desired information (e.g., names of medicines) from the conversation, but the disadvantage of not being expressed in the linguistic style of a SOAP note which leads to lower n-gram overlap. The opposite is true for the RANDOMNOTE baseline. Both baselines give similar performance and are outperformed by the simple CONV 2N OTE approach (Table 2). We train the abstractive modules for the 4 approaches described in Section 4 with the ground truth noteworthy utterances as inputs. To estimate an upper bound on the performance we can reasonably hope to achieve by improving our noteworthy utterance extractors, we test our models with oracle noteworthy utterances in the test set. All algorithms relying on oracle noteworthy utterances outperform CONV 2N OTE , and exhibit a monotonic and signifi-cant rise in ROUGE scores as we move towards the extraction-heavy end of the spectrum (Table 3) 3 . For predicting noteworthy utterances, we use two baselines: (i) logistic regression on TF-IDF utterance representations; and (ii) a model with a bidirectional LSTM to compute token-averaged utterance representations, followed by a linear classification layer. These two models make the predictions for each utterance independent of others. In contrast, we also use models which incorporate context from neighboring utterances: (a) a hierarchical LSTM; and (b) a BERT-LSTM model as described in Section 5. The latter two methods perform much better (Table 5), demonstrating the benefit of incorporating neighboring context, with BERT-LSTM performing the best (see Appendix Table A6 for section-wise performance). Using predicted noteworthy utterances and clusters instead of oracle ones leads to a drop in ROUGE scores, but the performance of EXT 2S EC and CLUSTER 2S ENT is still better than CONV 2N OTE (Table 2). For the medical dataset, using a BERT-LSTM extractor leads to the best performance, with CLUSTER 2S ENT outperforming CONV 2N OTE by about 8 points in ROUGE-1 (see Appendix Table A5 for section-wise performance). Interestingly, the T5-Small variant achieves similar performance to T5-Base, despite being only about a quarter of the latter's size. Performance on AMI dataset We see a similar trend in the ROUGE scores when applying these methods on the AMI dataset. One exception is the poor performance of pointer-generator based EXT 2N OTE , which excessively repeated sentences despite using a high coverage loss coef-ficient. There is a larger gap between the performance of the T5-Small and T5-Base abstractive models on this dataset. As an extractor, the performance of BERT-LSTM is again better than HLSTM (Table 5), but when used in tandem with the abstractive module, ROUGE scores achieved by the overall pipeline do not always follow the same order. We also observe that the clustering heuristic does not work as well on this dataset. Specifically, tuning the thresholds of the extractive model, while fixing the clustering threshold  gave worse results on this dataset. Tuning the thresholds independent 3 The character -' represents GPU memory overflow of the clusters performed better. Performance with ASR errors In the absence of human-generated transcripts of conversations, Automatic Speech Recognition (ASR) techniques can be used to transcribe the conversations for use by our models. To account for ASR errors, we ar-tificially added errors in transcripts of the medical dataset by randomly selecting some percentage of the words and replacing them with phonetically similar words using RefinedSoundEx (Commons) (details in the Appendix). Models trained on clean dataset perform worse on a 10% corrupted test dataset (Table 4). Since ASR errors lead to replacement of a correct word by only a small set of phonetically similar words, there is still some information indicating the original word that can be used by the models. When we train our models on data corrupted at the 10% ASR error rate, our models recover much of the performance drop (Table 4). Notably when simulated ASR errors are dialed up to a 30% error rate, (both at train and test time) we see a smaller performance drop for CLUSTER 2S ENT as compared to CONV 2N OTE . The conditioned pointer-generator and T5 models used in CLUSTER 2S ENT learn to place information regarding different topics in appropriate sections. Hence, given a cluster of supporting utterances, the models can generate different summaries for multiple sections (Figure 2). For example, given the same supporting utterances discussing the pa-tient's usage of lisinopril for low blood pressure, a model generates low blood pressure in the review of systems section, and lisinopril in medications section. We direct the reader to the appendix for examples of full-length generated SOAP notes. Interestingly, when the abstractive model is given a cluster of utterances that are not relevant to the section being generated, the model sometimes outputs fabricated information relevant to that section such as saying the patient is a non-smoker in social history , or that the patient has taken a flu shot in immunizations . Hence, the quality of produced summaries heavily depends on the ability of the extractive step to classify the extracted utterances to the correct section. Another cause of false information is the usage of pronouns in clusters without a mention of the referred entity. In such Medical dataset AMI corpus Method R-1 R-2 R-L R-1 R-2 R-L RANDOMNOTE 34.99 12.69 21.37 42.47 11.55 21.47 ORACLEEXT 33.07 12.22 17.42 39.97 11.17 20.91 CONV 2N OTE (PG) 49.56 25.68 32.87 39.62 13.16 23.95 EXT 2N OTE (PG + HLSTM) 49.58 24.91 31.68 21.28 7.06 15.96 EXT 2N OTE (PG + BLSTM) 50.50 25.4 31.93 21.71 6.83 15.69 EXT 2N OTE (T5-Small + HLSTM) --40.48 13.82 24.64 EXT 2N OTE (T5-Small + BLSTM) --40.36 13.73 24.13 EXT 2S EC (PG + HLSTM) 55.23 27.14 35.15 43.75 15.25 23.46 EXT 2S EC (PG + BLSTM) 55.74 27.54 36.09 40.48 15.61 23.31 EXT 2S EC (T5-Small + HLSTM) 55.77 28.64 37.50 42.45 15.20 23.92 EXT 2S EC (T5-Small + BLSTM) 56.00 29.16 38.38 45.44 16.59 26.14 CLUSTER 2S ENT (PG + HLSTM) 55.46 27.41 35.81 46.19 16.64 24.29 CLUSTER 2S ENT (PG + BLSTM) 55.60 27.68 36.29 42.31 15.92 23.51 CLUSTER 2S ENT (T5-Small + HLSTM) 56.88 28.63 36.78 45.10 15.06 23.52 CLUSTER 2S ENT (T5-Small + BLSTM) 57.14 29.11 37.43 42.38 15.36 23.9 CLUSTER 2S ENT (T5-Base + HLSTM) 57.27 29.10 37.38 50.52 17.56 24.89 CLUSTER 2S ENT (T5-Base + BLSTM) 57.51 29.56 38.06 45.91 17.70 25.24 Table 2: ROUGE scores achieved by different methods on the two datasets Cluster of utterances Subsection Summary-PG Summary-T5 DR That one thing that we can do to reduce risk with that cholesterol is 100 mg metoprolol. Occasionally, the abstractive module produces new inferred information that is not mentioned explicitly in the conversation. In one instance, the model generated that the patient has a history of heart disease conditioned on a cluster that mentioned he/she takes digoxin , a popular medicine for heart disease. Similarly, the model can infer past medical history of high cholesterol upon seeing pravastatin usage. Such inferences can also lead to incorrect summaries, e.g., when a doctor explained that a patient has leaky heart valves, a model added a sentence to the diagnostics and appointments section saying check valves. CLUSTER 2S ENT summarizes localized regions Medical dataset AMI corpus Method PG T5-Small PG T5-Small EXT 2N OTE 52.95 -20.44 41.10 EXT 2S EC 61.00 62.37 43.32 46.85 CLUSTER 2S ENT 63.63 66.50 51.86 54.23 Table 3: ROUGE-1 achieved on test set when using the abstractive models with oracle noteworthy utterances and clusters (more results with oracle in the Appendix) Method R-1 R-2 R-L Train on clean data + Test on data with 10% error rate CONV 2N OTE (PG) 46.52 22.60 30.45 CLUSTER 2S ENT (PG + BLS) 51.84 23.74 32.94 CLUSTER 2S ENT (T5-Base+ BLS) 54.88 26.65 35.88 Train and test on data with 10% error rate CONV 2N OTE (PG) 48.85 24.85 31.27 CLUSTER 2S ENT (PG + BLS) 54.68 26.59 35.70 CLUSTER 2S ENT (T5-Base+ BLS) 56.35 28.50 37.04 Train and test on data with 30% error rate CONV 2N OTE (PG) 45.16 22.26 30.14 CLUSTER 2S ENT (PG + BLS) 53.69 25.88 35.12 CLUSTER 2S ENT (T5-Base+ BLS) 55.90 27.73 36.06 Table 4: Performance of models trained and tested on data with different simulated ASR error rates. In one visit, the patient was asked about chest pain twiceonce in the beginning to get to know his/her current state, and once as a question about how he/she felt just before experiencing a fall in the past. This led to the model generating both that the patient denied chest pain as well as confirmed chest pain, without clarifying that one statement was for the present and another for the past. We asked trained human annotators to evaluate generated SOAP notes for 45 conversations. Every sentence in each SOAP note was labeled according to various quality dimensions such whether it was factually correct, incoherent, irrelevant, redundant, or placed under an inappropriate section. The detailed statistics of annotations received for each quality dimension are provided in the Appendix. We also collected aggregate annotations for the comprehensiveness of each SOAP note and the extent to which it verbatim copied the transcript on a 5-point Likert scale. Human raters were presented with a web in-Medical conversations AMI corpus Metric LR LS HLS BLS HLS BLS Accuracy 96.0 96.1 96.5 96.5 93.77 94.16 Ma-AUC 78.1 79.3 90.0 90.5 83.81 90.76 Ma-F1 29.5 31.0 38.6 40.9 19.95 33.08 Mi-AUC 87.3 87.6 92.7 93.3 93.21 94.90 Mi-F1 31.2 32.9 39.6 41.1 43.76 49.93 Table 5: Performance on multilabel classification of noteworthy utterances with logistic regres-sion(LR), LSTM(LS), Hierarchical-LSTM(HLS) and BERT-LSTM(BLS). The summaries generated by three methods (CONV 2N OTE (pointer-generator), CLUSTER 2S ENT (pointer-generator) and CLUSTER 2S ENT (T5-base)) were presented in random order to hide their identities. For each sentence, we asked for (i) Factual correctness of the sentence; (ii) If the statement is simply repeating what has already been mentioned before; (iii) If the statement is clinically irrelevant; (iv) If the statement is incoherent (not understandable due to grammatical or semantic errors); and (v) If the state-ment's topic does not match the section in which it is placed. In addition, we asked two separate questions for rating the overall summary on a scale of 1-5 for its (i) comprehensiveness and (ii) extent of verbatim copying from conversation. The human evaluation of the SOAP notes was done by workers who had also participated in the creation of the dataset of SOAP notes. Hence, they had already been extensively trained in the task of SOAP note creation, which gave them appropriate knowledge to judge the SOAP notes. To quantify the performance among different methods, we consider a scenario where each generated SOAP note has to be post-edited by discarding undesirable sentences. For a generated SOAP note, we define its yield as the fraction of its total sentences that are not discarded. The sentences that are retained are those that are both factually correct and were not labeled as either repetitive or incoherent. The human annotations show that both CLUSTER 2S ENT -based methods tested produced a higher yield than the CONV 2N OTE baseline (p < 0 . 02 ). T5-base performs better than conditioned pointer-generator as the abstractive module in CLUSTER 2S ENT setting, producing significantly more yield (Table 6). T5 also produces fewer inco-Medical conversations AMI corpus Metric C2N C2S-P C2S-T C2N C2S-P C2S-T Length 21.2 28.2 28.4 20.7 17.9 19.05 %Yield 62.0 69.0 74.7 27.22 30.22 59.45 Comp 2.44 2.42 2.76 2.30 2.55 3.75 Copy 2.18 2.64 2.76 1.80 1.80 1.90 Table 6: Averages of different metrics for CONV 2N OTE (C2N), CLUSTER 2S ENT with pointer-generator (C2S-P) and T5-base (C2S-T). herent sentences (Appendix Table A4) likely due to its exposure to a large number of well-formed coherent sentences during pretraining. We conducted an analogous human evaluation of summaries generated for all 20 conversations in the test set of the AMI corpus, and saw a similar trend in the expected yield for different methods. Notably, for the AMI corpus, CONV 2N OTE produced a very high proportion of redundant sentences ( > 0 . 5 ) despite using the coverage loss, while the pointer-generator based CLUSTER 2S ENT produced a high proportion of incoherent sentences (Appendix Table A4). This paper represents the first attempt at generating full-length SOAP notes by summarizing transcripts of doctor-patient conversations. We proposed a spectrum of extractive-abstractive summarization methods that leverage: (i) section-structured form of the SOAP notes and (ii) linked conversation utterances associated with every SOAP note sentence. The proposed methods perform better than a fully abstractive approach and standard extractive-abstractive approaches that do not take advantage of these annotations. We demonstrate the wider applicability of proposed approaches by showing similar results on the public AMI corpus which has similar annotations and structure. Our work demonstrates the benefits of creating section-structured summaries (when feasible) and collecting evidence for each summary sentence when creating any new summarization dataset. The methods proposed in this work to generate SOAP notes involve neural models that sometimes generate factually incorrect text (Maynez et al., 2020). The detection and correction of such factual errors in automatically generated summaries is an active area of research (Cao et al., 2018; Zhang et al., 2020; Dong et al., 2020). We emphasize that the methods are intended to be used with supervision from a medical practitioner who can check for factual errors and edit the the generated SOAP note if needed. We have estimated the frequency of such factual errors (Appendix Table A4) and characterized multiple types of errors seen in generated SOAP notes in Section 7, for which the medical practitioners should remain vigilant. For example, there is a bias to incorrectly generate information that occur frequently in specific sections (e.g. pa-tient took flu shot), and to replace pronouns with frequently seen entities (such as lisinopril for references to medicine). All data used in this study was manually de-identified before we accessed it. Deploying the proposed methods does not require long-term storage of conversations. After the corresponding SOAP notes are generated, conversations can be discarded. Hence, we do not anticipate any additional privacy risks from using the proposed methods. This work was funded by the Center for Machine Learning and Health in a joint venture between UPMC and Carnegie Mellon University. We gratefully acknowledge support from Abridge AI, Inc. for creating the dataset of SOAP notes and providing human resources for evaluation.
abstain abstain objective objective abstain abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain result objective abstain abstain other abstain abstain abstain result abstain abstain abstain result objective method other abstain method result objective objective objective objective other other other other other other method other other objective method abstain method method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain other abstain abstain method abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other abstain abstain abstain objective objective abstain abstain abstain objective objective abstain abstain abstain abstain abstain method abstain abstain abstain other other