## Agentic LLM Workflows for Generating Patient-Friendly Medical Reports

Malavikha Sudarshan , Sophie Shih , Estella Yee 1 2 2 , Alina Yang 2 , John Zou , 3 Cathy Chen , Quan Zhou , Leon Chen , Chinmay Singhal 4 5 5 5 and George Shih 6

1 Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA

2 Stuyvesant High School, NY, USA

3 Department of Computer Science, Brown University, RI, USA

4 Stern School of Business, New York University, NY, USA 5 MD.ai, NY, USA

6

Department of Radiology, Weill Cornell Medicine, NY, USA malavikhasudarshan@berkeley.edu, {sshih60, eyee60, ayang6}@stuy.edu, john\_zou@brown.edu, hc2845@nyu.edu, {quan, leon, chinmay}@md.ai, george@cornellradiology.org

## Abstract

The application of Large Language Models (LLMs) in healthcare is expanding rapidly, with one potential use case being the translation of formal medical reports into patient-legible equivalents. Currently, LLM outputs often need to be edited and evaluated by a human to ensure both factual accuracy and comprehensibility, and this is true for the above use case. We aim to minimize this step by proposing an agentic workflow with the Reflexion framework, which uses iterative self-reflection to correct outputs from an LLM. This pipeline was tested and compared to zero-shot prompting on 16 randomized radiology reports. In our multi-agent approach, reports had an accuracy rate of 94.94% when looking at verification of ICD-10 codes, compared to zero-shot prompted reports, which had an accuracy rate of 68.23%. Additionally, 81.25% of the final reflected reports required no corrections for accuracy or readability, while only 25% of zero-shot prompted reports met these criteria without needing modifications. These results indicate that our approach presents a feasible method for communicating clinical findings to patients in a quick, efficient and coherent manner whilst also retaining medical accuracy. The codebase is available for viewing at http://github.com/ malavikhasudarshan/Multi-Agent-Patient-Letter-Generation .

Keywords : Large Language Models, Patient-Friendly Letters, Patient Literacy, Radiology, Report Generation, GPT.

## 1 Background

The 21st Century Cures Act grants patients the right to access their electronic health record data, and since its implementation, the number of patients accessing their Electronic Health Records (EHRs) before the ordering provider has increased significantly [1]. While intended to improve transparency and promote a shared flow of information, this increased level of accessibility can often lead to patient anxiety, misinterpretation and confusion when reading jargon-filled medical reports that they are not the primary audience for [2]. Radiology reports are a prime example of these; mostly intended for referring physicians, when abnormal or ambiguous results are received by patients before discussion with their physician, the impact can often be more harmful than beneficial [3]. To address this, the

creation of patient-friendly letters that simplify complex medical information has been explored [4]. These letters aim to explain medical terms clearly, ensure factual accuracy, and also maintain a compassionate and reassuring tone.

In recent years, Large Language Models (LLMs) have been increasingly leveraged in healthcare applications, from producing discharge summaries [5] to structured radiology reports [6]. In applying generative artificial intelligence to the creation of patient-friendly reports, the pipeline can be made more efficient and patients can have access to more meaningful and legible letters [7, 8]. Most current developments invoke zero-shot prompting to create a patient-friendly version of a medical report included in the input prompt, where the LLM's internal representations are relied upon to produce a suitable letter, and no template or example output is provided in the prompt for guiding the structure, style or comprehensiveness of the generated letters [9, 10]. Through this method, LLMs often generate outputs that need to be manually reviewed or go through alternative mechanisms to be critiqued and improved before being delivered to the patient. One research study concluded that 80.4% ( n = 41 ) of tested patient-friendly LLM-generated summaries of medical reports required editing before being released to patients [11].

Our goal was to develop an agentic pipeline where verification would be minimized, and where patient letters would be evaluated for both accuracy and readability before being released.

## 2 Methods

Agentic workflows are iterative and consist of several intermediary steps performed in addition to LLM prompting, as opposed to non-agentic or zero-shot/few-shot prompts which consist of a single input and a single output [12]. The former approach means that multiple agents can be leveraged, and they are often structured similar to professional businesses, where each agent plays a specific role in the organization. Addition-by-Subtraction collaboration is one example of a multi-agent method, where one agent provides information and the other removes unnecessary details and provides feedback [13].

Agentic workflows allow for reinforcement learning through reflection [12], and can utilize chainof-thought prompting by appending reflected feedback at the end of the next prompt. We leveraged an existing framework, Reflexion [14], which incorporates verbal reinforcement into its iterative refinement process. Typically, agents receive feedback from their environment in a simple form, like a binary signal (e.g., success/failure) or a scalar value (a numerical score). Reflexion agents take this basic feedback and translate it into a more detailed, verbal form-a textual summary that explains the feedback in natural language. This verbal feedback is then added to the context of the following prompt, and acts as a 'semantic' gradient signal, meaning that it provides the agent with specific, meaningful directions on how to improve.

Our implementation prompts an LLM to generate a specific number of patient-friendly letters based on a formal medical report. The accuracy and readability of each generated letter is calculated and weighted appropriately, and the Reflexion model is then used to run a certain number of self-reflection trials and output the letter that it considers to be optimal at the end of this. Reflexion has three separate legs - AlfWorld (for decision-making problems), HotPotQA (for reasoning on single-step iterations) and Programming (for programming tasks using interpreters and compilers). We used AlfWorld, as decision-making made the most sense when prompting for multiple letters and asking for the most optimal output.

The original medical report can either be provided as an argument, or, as we presented at the Society for Imaging Informatics in Medicine (SIIM) 2024 Hackathon, be pulled from an EHR server. Our integration involved extracting one of the five medical reports available on the SIIM Fast Healthcare Interoperability Resources (FHIR) server and pushing our results back onto the server. Manually including the medical report in the input was also later tested on 15 other test radiology reports of various modalities: Computed Tomography (CT), Magnetic Resonance Imaging (MR), and Ultrasound (US) (see Figs. 1, 2 and 3 in the Appendix). These reports differed in length, ranging from 84 to 264 words, and covered a range of medical findings and body parts, including the abdomen, pelvis, chest, head, and lumbar spine.

Our pipeline (Fig. 1) operates as follows: we first make one LLM call to extract the International Classification of Diseases, Tenth Revision (ICD-10) codes from the original report. The temperature

is kept at 0 to minimize variance, and these codes are stored to be compared later. A second LLM call is then used to generate a number of patient-friendly reports ( n=5 , for example) based on the original, and this time we ask the agent to produce ICD-10 codes based on the content of each patient-friendly letter. These ICD-10 codes are verified against the master ICD-10 code database (using the simple-icd-10 package [15]) and the description for each code is also retrieved and compared against the LLM's output to see if they match. The accuracy of each letter is calculated as the number of validated and identical ICD-10 codes between the patient-friendly version and the original medical report, divided by the total number of ICD-10 codes on the original report. This value should be maximized. Readability is quantified using the Flesch-Kincaid Grade Level. This value is calculated using a predefined formula incorporating the average sentence length, number of syllables and number of words per sentence [16] and can be accessed by importing the readability module [17]. The average American's reading ability is equivalent to a US 8th-grade level [18]. A previous study examining 97,052 radiology reports revealed that only 4% were at a reading level equal to or below the reading ability of a US 8th-grader [19], suggesting that much of this information may be unintelligible to a significant portion of the population. Our 16 test reports had an average Flesch-Kincaid Grade Level of 11.03, corresponding to an 11th-grader's expected level of vocabulary.

A Flesch-Kincaid Grade Level of 6.0 (corresponding to a US 6th-grader's reading ability) is the recommended level of readability advised by the American Medical Association [20] and the National Institute of Health [21] for patient-facing medical materials, to allow for greater comprehensibility and accessibility [22]. Each generated patient letter's overall score is calculated by weighting the readability and accuracy - we wanted to prioritize medical accuracy so opted to compute the score as follows:

overall\_score = (readability * 0.3) + (accuracy * 0.7)

The readability value is standardized to be as close to 6.0 as possible, therefore, we can aim for an overall\_score that has a maximum value of 1.0. Reflexion's Alfworld module is then used to reflect on the overall\_score, looking to improve both the accuracy and readability of each letter on each iteration. The algorithm outputs the best version of the letter, which is then directly pushed to the linked EHR server for patient access, demonstrating end-to-end integration. The LLM used in our tests was OpenAI's GPT-4o (gpt-4o-2024-05-13).

Figure 1: Flowchart of the Multi-Agent Algorithm

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-2458641c-577e-4db0-bcf9-dc56c0bdba80.jpg)

## 3 Results

Our reflection agent increased the medical accuracy of reports, ensuring that ICD-10 codes were retained in the final patient letter which the zero-shot output sometimes missed. When given an identical medical report, system prompt and user prompt, the reflected output consistently scored higher in terms of accuracy and readability, as well as in the overall\_score measure. Zero-shot prompts were sometimes not professional enough, and even when specified in the prompt that the reading level should match that of a US 6th grader's, the language used was too juvenile. However, when the reflected agent was used, the final outputs seemed to be consistently more concise, structured and formal.

In the example below (Figs. 2 and 3), the same medical report was used to compare a zero-shot (Fig. 2) and reflection agent output (Fig. 3). With zero-shot prompting, only half of the desired ICD-10 codes were registered, whereas the reflection agent successfully generated all 4 ICD-10 codes. Additionally, the ICD-10 codes generated by the reflection agent precisely matched those from the original medical report, while the codes from the zero-shot report did not.

From 16 test radiology reports, zero-shot prompting (using the same original prompt as given in our multi-agent workflow) led to 11/16 patient-friendly versions needing to be edited, whilst our agentic workflow resulted in only 3/16 reports that required modification. We considered 'modification' to be any changes in medical factuality (including ICD-10 codes), grammar, punctuation, tonality and readability. On average, accuracy was 26.71% better, and readability scored 3.29% higher in the reflected patient letters, compared to the zero-shot letters (Fig. 4). This resulted in a 17.51% increase in overall\_score in reflected letters vs zero-shot generated letters.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-5f026b50-695f-4cc9-a39a-693f4084eeeb.jpg)

Figure 2: Zero-Shot Generated Patient-Friendly Letter

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-5f70f47b-4b06-47f5-ba06-359840f965a6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-cc70721d-0a9b-42d2-bb82-c6269f76b717.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-4ec37f53-ac6b-4a04-b906-132611e4ad65.jpg)

Figure 3: Reflection/Multi-Agent Generated Patient-Friendly Letter

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-d96585ad-ff1c-4ad6-8692-a147e8d4dcd8.jpg)

Figure 4: Summary Table of Results

## 4 Discussion

The use of ChatGPT and similar LLMs for the generation of patient friendly letters is something that many others in the healthcare space have been experimenting with [5][10][23][24][25]. However, LLMs are known to hallucinate and are extremely sensitive to input, which can often lead to errors in the outputted patient-friendly letter. Additionally, the more complex the medical report, the higher the tendency for LLMs to hallucinate [26]. The inclusion of multiple agents and programmatic prompts 1 aim to manage the complexity of medical reports, whilst simultaneously minimizing hallucinations. This workflow reduces the need for proofreading, as the patient letters are evaluated for both accuracy and readability before being outputted.

As part of our accuracy metric, we make use of the get\_description(icd10\_code) [15] function to verify whether the ICD-10 code definitions match industry-standard for the original and patientfriendly reports. However, as this function uses string matching, it is possible that we may miss out on synonymous definitions or phrases with a few variances in words. A better alternative may be to use fuzzy matching algorithms such as calculating the Levenshtein distance [27], or looking at the K-Nearest Neighbors [28] of the two description strings to categorize them, instead of comparing two strings for an exact match.

1 LLM prompts have been programmed and cannot be altered by users-handles issue of sensitivity to input.

One assumption we make is the accuracy of the ICD-10 codes generated by the LLM model (GPT-4o). In separate human validation tests, we have seen high accuracy and consistency when generating these codes from test radiology reports, so in this study we assume that these generated ICD-10 codes can be trusted.

This is still a very early prototype and can be improved upon in several ways. In the future, we hope to be more inclusive of different reading levels, languages, and medical fields. Currently, we have standardized the level of readability to a 6th grade level; however, it would be beneficial to have a variety of literacy levels available depending on the patient. Additionally, adding the functionality for accurate translation in various languages would significantly enhance communication abilities as well as global applicability and reach. Finally, we are aiming to be applicable to various medical fields outside radiology.

As of now, our weighting system is based upon readability and accuracy. However, we understand the importance of maintaining a certain level of compassion within these letters. One possible approach is utilizing the PERMA model [29] as a metric for factoring in compassion into our weighting system. The PERMA scale can help our model determine whether a patient letter has the appropriate tone and level of sensitivity. Other additional metrics we are looking into to further enhance patient letters include CDE codes [30], which can help to accurately convey a patient's treatment process and future action required.

## 5 Conclusion

Our objective was to find a method of generating patient-friendly radiology reports that would really reduce the need for a medical professional to review and verify them. Although our approach does significantly improve the quality of patient-legible letters, it does not have an 100% success rate, and therefore cannot eradicate the need for verification completely. However, by significantly reducing the percentage of LLM-generated reports requiring edits-from 68.75% to 18.75%-through the incorporation of a multi-agent workflow, we can show that the time spent making changes in medical accuracy and readability will also be substantially decreased. Our method not only enhances the efficiency of report generation but also contributes to the overall goal of making healthcare information more accessible and understandable to patients. This development has strong potential to streamline clinical workflows, lessen the burden on medical professionals and administrators, and improve the patient experience by swiftly providing clearer and more accurate medical information.

## 6 Acknowledgements

The authors would like to thank the SIIM community, most notably Teri M. Sippel Schmidt, Alex Barrington, Tom O'Sullivan, Mohannad Hussain, and those who took the time to provide feedback, for supporting our work.

## References

| [1]   | Pollock JR, Petty SA, Schmitz JJ, Varner J, Metcalfe AM, and Tan N. Patient access of their radiology reports before and after implementation of 21st century cures act information- blocking provisions at a large multicampus health system. Am J Roentgenol , 222(6), 2024. doi: 10.2214/ajr.23.30343.   |
|-------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [2]   | Gerber DE. 21st century cures act: Implementation without understanding implication? JCO Oncol Pract , 18(2):85-87, 2022. doi: 10.1200/OP.21.00436.                                                                                                                                                         |
| [3]   | Winget M, Haji-Sheikhi F, Brown-Johnson C, et al. Electronic release of pathology and radiology results to patients: Opinions and experiences of oncologists. J Oncol Pract , 12(8), 2016. doi: 10.1200/JOP.2016.011098.                                                                                    |
| [4]   | Smolle C, Schwarz CM, Hoffmann M, et al. Design and preliminary evaluation of a newly designed patient-friendly discharge letter: A randomized, controlled participant-blind trial. BMC Health Serv Res , 21:450, 2021. doi: 10.1186/s12913-021-06468-3.                                                    |
| [5]   | Zaretsky J et al. Generative artificial intelligence to transform inpatient discharge sum- maries to patient-friendly language and format. JAMA Netw Open , 7(3), 2024. doi: 10.1001/jamanetworkopen.2024.0357.                                                                                             |
| [6]   | Liu J, Wang C, and Liu S. Utility of chatgpt in clinical practice. J Med Internet Res , 25:e48568, 2023. doi: 10.2196/48568.                                                                                                                                                                                |
| [7]   | Doo FX, Cook TS, Siegel EL, et al. Exploring the clinical translation of generative models like chatgpt: Promise and pitfalls in radiology, from patients to population health. J Am Coll Radiol , 20(9):877-885, 2023. doi: 10.1016/j.jacr.2023.07.007.                                                    |
| [8]   | Park J, Oh K, Han K, et al. Patient-centered radiology reports with generative artificial intelligence: Adding value to radiology reporting. Sci Rep , 14:13218, 2024. doi: 10.1038/ s41598-024-63824-z.                                                                                                    |
| [9]   | Cork SC and Hopcroft K. Evaluating the utility of chatgpt to convert clinic letters into patient- friendly language, 2024. Published online July 9, 2024.                                                                                                                                                   |
| [10]  | Roberts RHR, Ali SR, Dobbs TD, and Whitaker IS. Can large language models generate outpatient clinic letters at first consultation that incorporate complication profiles from uk and usa aesthetic plastic surgery associations? Aesthet Surg J Open Forum , 6, 2023. doi: 10.1093/asjof/ojad109.          |
| [11]  | Berigan K, Short R, Reisman D, et al. The impact of large language model-generated radiology report summaries on patient comprehension: A randomized controlled trial. J Am Coll Radiol , 2024. doi: 10.1016/j.jacr.2024.06.018. Published online July 1, 2024.                                             |
| [12]  | Ng A. Issue 242. one agent for many worlds, cross-species cell embeddings, and more, 2024. URL https://www.deeplearning.ai/the-batch/issue-242/ . April 2, 2024. Accessed July 22, 2024.                                                                                                                    |
| [13]  | Wu M, Yuan Y, Haffari G, and Wang L. (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts, 2024. URL https://arxiv. org/abs/2405.11804 . Published online May 20, 2024. Accessed July 24, 2024.                                               |
| [14]  | Shinn N, Cassano F, Berman E, Gopinath A, Narasimhan K, and Yao S. Reflexion: Language agents with verbal reinforcement learning, 2023. Published online 2023.                                                                                                                                              |
| [15]  | Travasci S. Simple-icd-10, 2024. URL https://pypi.org/project/simple-icd-10/ . PyPI. Accessed July 24, 2024.                                                                                                                                                                                                |
| [16]  | Kincaid JP, Fishburne RP Jr, Rogers RL, and Chissom BS. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Institute for Simulation and Training, 1975.                                         |
| [17]  | Readability. Readability, 2024. URL https://pypi.org/project/readability/ . PyPI. Accessed July 30, 2024.                                                                                                                                                                                                   |

| [18]   | Amin K, Khosla P, Doshi R, Chheang S, and Forman HP. Artificial intelligence to improve patient understanding of radiology reports, 2023. Published September 29, 2023.                                                                                                                                       |
|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [19]   | Martin-Carreras T, Cook TS, and Kahn CE Jr. Readability of radiology reports: implications for patient-centered care. Clin Imaging , 54:116-120, 2019. doi: 10.1016/j.clinimag.2018.12.006.                                                                                                                   |
| [20]   | Weiss BD. Health Literacy: A Manual for Clinicians . American Medical Association, American Medical Foundation, Chicago, IL, 2003.                                                                                                                                                                            |
| [21]   | Badarudeen S and Sabharwal S. Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res , 468(10):2572-2580, 2010. doi: 10.1007/s11999-010-1380-y.                                                                                                            |
| [22]   | Davis TC, Crouch MA, Wills G, Miller S, and Abdehou DM. The gap between patient reading comprehension and the readability of patient education materials. J Fam Pract , 31(5):533-538, 1990.                                                                                                                  |
| [23]   | Ali S, Dobbs TD, Hutchings HA, and Whitaker IS. Using chatgpt to write patient clinic letters, 2023. URL https://www.researchgate.net/publication/369076647_Using_ ChatGPT_to_write_patient_clinic_letters . Published online 2023.                                                                           |
| [24]   | Guo R, Farnan G, McLaughlin N, and Devereux B. Qub-cirdan at 'discharge me!': Zero shot discharge letter generation by open-source llm, 2024. URL https://arxiv.org/abs/2406. 00041 . Published online June 27, 2024. Accessed July 26, 2024.                                                                 |
| [25]   | Doshi R, Amin KS, Khosla P, Bajaj S, Chheang S, and Forman HP. Quantitative evaluation of large language models to streamline radiology report impressions: A multimodal retrospective analysis. Radiology , 310(3), 2024. doi: 10.1148/radiol.231593.                                                        |
| [26]   | Xu Z, Jain S, and Kankanhalli M. Hallucination is inevitable: An innate limitation of large language models, 2024. URL https://arxiv.org/abs/2401.11817 . arXiv.org. January 22, 2024. Accessed July 29, 2024.                                                                                                |
| [27]   | Lee C, Kim Y, Kim YS, and Jang J. Automatic disease annotation from radiology reports using artificial intelligence implemented by a recurrent neural network. Am J Roentgenol , 212(4): 734-740, 2019. doi: 10.2214/AJR.18.19869.                                                                            |
| [28]   | Lüschow Aand Wartena C. Classifying medical literature using k-nearest-neighbours algorithm, 1970. URL https://serwiss.bib.hs-hannover.de/frontdoor/index/index/docId/ 1146 . Accessed July 29, 2024.                                                                                                         |
| [29]   | Butler J and Kern ML. The perma-profiler: A brief multidimensional measure of flourishing, 2016. URL https://www.internationaljournalofwellbeing.org/index.php/ijow/ article/view/526 .                                                                                                                       |
| [30]   | National Institutes of Health. Common data elements: Standardizing data collection, 2024. URL https://www.nlm.nih.gov/oet/ed/cde/tutorial/03-100.html#: ~:text=A%20common%20data%20element%20(CDE),to%20ensure%20consistent% 20data%20collection . U.S. National Library of Medicine. Accessed July 24, 2024. |

## A Appendix

The following examples display LLM outputs for MR, CT, and Ultrasound test reports, offering straightforward visual comparisons between zero-shot and multi-agent generated patient-friendly reports.

## ZERO-SHOT GENERATED PATIENT LETTER

Dear [Patient s Name],

Ihope doing well share the results of your recent MRI scan your head This tes was done headaches and feeling dizzy.

Hene tound

Smnall Spot the Brain" Wc lound pca (1.2 cm) part brain called hemangioblastoma-Itis type usually not cancer

Extra Fluid' There news This means you do not have condition called hydrocephalus. brin Qood

Pressure ontne Brain" The spot we found putting any pressure other parts your brain-Everything else looks normal.

Wewill talk more about what this means andthenext steps when we see each other vournextappoiniment You have any questions belore then, please call our office.

Take care

[Your Doctor's Name]

Codes

Ison

## REFLECTION AGENT PATIENT LETTER

Ihope you are doing your headWe did this test because you have been having headaches and dizziness

Heres what we found:

tiny spot, about the size pea (12 cm)in a part your brain called the cerebellum. This spot is likely something called usually not cancer

"Brain Fluid Spaces The spaces in your brain that hold fluid (called good.

"No Pressure on the shitt your brain This mcans the smal not pushing other parts of your brain

In summary wefound that might be needed ,

Please feel free ask any questions when we meet next

Best regards ,

[Your Doctor's Name]

D33.1 : "Benign neoplasm of brain, infratentorial" ,

"R5I' : Headache"

Figure 5: Patient-Friendly Letters generated from an MR Head Report

## ORIGINAL RADIOLOGY REPORT

chronic hcadaches and dizziness.

MRI

Technique: comprehensive examination was performed utilizing a variety Imaging planes and naging paramelers visualization suspecieo pathology Images vere obtained with and without intravenous gadolinium,

## Findings:

consistent with hemangioblastoma;

Ventricles: No evidence hydrocephalus.

Other: Negative;

Enhancing lesion in theleft cerebellar hemisphere, consistent with a hydrocephalus \_ significant mass etfect.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-8697c230-25db-44ec-99b1-7ce8a84546a9.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-b666b1c0-73dd-4162-b4ad-4e69f730790a.jpg)

## ORIGINAL RADIOLOGY REPORT

CT Chest with Contrast

Technique Contiquous axlal CTimages the chest were obtained after the

Findings:

noma calibur; Doune configuration ncic Linear bandkke areas scarringlatelectasis presentin the nght

There osseousles on evidence pricr trauma delormity ofthe right Sthnb

Impression:

cavitary tavored

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-ec417a17-e374-4a75-a7ce-08a75b682e47.jpg)

## ZERO-SHOT GENERATED PATIENT LETTER

your chest CT scanis a special type ot X-ray that gives detailed

- extrafud around yourchest calledthe codtaalso Ioksnormal changes this blcod vessel, but nothing our
- Welound covityin the lower part your nghtlung whch about mmn size These cavties mgnt be due Infection
- upper part ot
- 'Lymph Nodes and Thyroid"
- yournbs
- Intection Wo ncedto keepaneyeon these-

The scar-like

We wil discuss these {indings have any

[Your Doctors Name]

son

Other disoiders

## REFLECTION AGENT PATIENT LETTER

CT scan look inside your body- louna

- but nothing worrY about right now biood
- Wefound of yourlett lung that looks about mmvin slze These cavities
- There are some scar-Meareas the upper part ol your
- your immune syslemi
- FOther Areas The upper evidence old Injury to
- The scar-like right lung not changed and are likely from an old Intection

We will d scuss these findings

[Your Dociors Name]

son aorta ,

fracture

Figure 6: Patient-Friendly Letters generated from a CT Chest Report

## ORIGINAL RADIOLOGY REPORT

History: [History thyroid nodules change voice]

Technique: The thyroid structures were scanned with high

Comparison: [None]

Findings

Thyroid

Right lobe measures; [3 5 with irregular margins m crocalcifications, concerning Ior malignancy ] and

Left lobe measures: (Multiple benign-appearing nodules ]

Isthmus: (The Isthmnus normal in size and configuratlon ]

Lymoh Nodes: [Sweep images of the neck b laterally adjacent the thyroid Imph nodes]

Addîtional Findings: [None]

[Rght thyroid lobe with 3,5 * 40 cm solid nodule with irregular margins and biopsy ]

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-6e0c7d21-b719-4b14-8dcc-ece7733b020c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01112v2-figure-2545355c-eecc-4f0a-a3ff-bb6f78ae5b3d.jpg)

## ZERO-SHOT GENERATED PATIENT LETTER

Dear [Patients Namel

Ihope thyroid scan and explain what they mean simple terms, nds in your neck that helps control your bodys energy- You mentioned that your voice has changed recently check thyroid might be the

Heres what we found:

"Right Side Thyroid" small plum (3.5 bit worriedit could

- There are fovsmail they look harmlcss.
- "'Middle Part of Thyroid (Isthmus)* This looks normal
- 'Lymph Nodes" glands that help fight Intections We checked the ones near your thyroid, and look normal too ney

## Whats Next?

Bccause couple morc tests

- test where doctor looks at your voice box and throat with

'Biopsy: This tiny piece the lump look at it more closcly under

Moe the lump and decide the best way take care of it

[Your Doctor s Namc]

Codes

E04.1 : "Nontoxic single thyroid nodule

## REFLECTION AGENT PATIENT LETTER

Ihopc you arc doing wcll. vantcd share the results of your recent thyroid scanand erplain what they mean;

"Right Sidc (nodulc) that mcasurcs about 40 cm This nasuneven edges and tiny calcium spots, Because check more carelully make sureitis nol somcthing scrious, like cancor Jump unp

- 2 *Lelt Side
- This part looks normal
- 'Lymph Nodes` We checked the lymph nodes ncar your thyroidand they

spcclalist They might want tiny camera (called laryngoscopy) and lake sample Irom the biopsy)

Please let knowif youhave any questions , there[ anything else we can

[Your Doctors Namc]

ison

'E041: Nontoxic single thyroid nodule" C73 : "Malignant neoplasm thyroid gland , "R49o : Dysphonta"

Figure 7: Patient-Friendly Letters generated from a US Thyroid Report
Malavikha Sudarshan Sophie Shih Estella Yee Alina Yang John Zou Cathy Chen Quan Zhou Leon Chen Chinmay Singhal George Shih
Agentic LLM Workflows for Generating Patient-Friendly Medical Reports
The application of Large Language Models (LLMs) in healthcare is expanding
rapidly, with one potential use case being the translation of formal medical
reports into patient-legible equivalents. Currently, LLM outputs often need to
be edited and evaluated by a human to ensure both factual accuracy and
comprehensibility, and this is true for the above use case. We aim to minimize
this step by proposing an agentic workflow with the Reflexion framework, which
uses iterative self-reflection to correct outputs from an LLM. This pipeline
was tested and compared to zero-shot prompting on 16 randomized radiology
reports. In our multi-agent approach, reports had an accuracy rate of 94.94%
when looking at verification of ICD-10 codes, compared to zero-shot prompted
reports, which had an accuracy rate of 68.23%. Additionally, 81.25% of the
final reflected reports required no corrections for accuracy or readability,
while only 25% of zero-shot prompted reports met these criteria without needing
modifications. These results indicate that our approach presents a feasible
method for communicating clinical findings to patients in a quick, efficient
and coherent manner whilst also retaining medical accuracy. The codebase is
available for viewing at
http://github.com/malavikhasudarshan/Multi-Agent-Patient-Letter-Generation.