Suneil A Raju, Freya J Bowker-Howell, Imran Aziz, Mo Thoufeeq, Alan J Lobo, Dermot C Gleeson, Amer Al-Joudeh, Mark E McAlindon, Andrew D Hopper, Sampath Kumar, Reena Sidhu, David S Sanders
What is the role of out of programme clinical fellowships in the era of Shape of Training? A single-centre cohort study
BMJ open gastroenterology
background latest large language models llms solve unseen problems via userdefined text prompts without need retraining offering potentially efficient information extraction freetext medical records manual annotation purpose compare performance llms chatgpt gpt4 data mining labeling oncologic phenotypes freetext ct reports lung cancer using userdefined prompts materials methods retrospective study included patients underwent lung cancer followup ct september 2021 march 2023 subset 25 reports reserved prompt engineering instruct llms extracting lesion diameters labeling metastatic disease assessing oncologic progression output fed rulebased natural language processing pipeline match ground truth annotations four radiologists derive performance metrics oncologic reasoning llms rated fivepoint likert scale factual correctness accuracy occurrence confabulations recorded statistical analyses included wilcoxon signed rank mcnemar tests results 424 ct reports 424 patients mean age 65 years 11 sd 265 male gpt4 outperformed chatgpt extracting lesion parameters 986 vs 840 p 001 resulting 96 correctly mined reports vs 67 chatgpt p 001 gpt4 achieved higher accuracy identification metastatic disease 981 95 ci 977 985 vs 903 95 ci 894 910 higher performance generating correct labels oncologic progression f1 score 096 95 ci 094 098 vs 091 95 ci 089 094 p 001 oncologic reasoning gpt4 higher likert scale scores factual correctness 43 vs 39 accuracy 44 vs 33 lower rate confabulation 17 vs 137 chatgpt p 001 conclusion using userdefined prompts gpt4 outperformed chatgpt extracting oncologic phenotypes freetext ct reports lung cancer demonstrated better oncologic reasoning fewer confabulations rsna 2023 supplemental material available article see also editorial hafezinejad trivedi issue