Alfredo de Micheli
[William Harvey and the beginnings of modern medical science]
Gaceta medica de Mexico
large language models llms chatgpt developed use research medical education clinical decision systems however usage increases llms face ongoing regulatory concerns study aims analyse chatgpts performance postgraduate examination identify areas strength weakness may provide insight role healthcare evaluated performance chatgpt 4 24 may 2023 version official mrcp membership royal college physicians parts 1 2 written examination practice questions statistical analysis performed using python spearman rank correlation assessed relationship probability correctly answering question two variables question difficulty question length incorrectly answered questions analysed using clinical reasoning framework assess errors made online using chatgpt web interface primary outcome score percentage questions correct mrcp postgraduate written examinations secondary outcomes qualitative categorisation errors using clinical decisionmaking framework chatgpt achieved accuracy rates 863 part 1 703 part 2 weak significant correlations found chatgpts accuracy justpassing rates part 2 r034 p00001 question length part 1 r019 p0008 eight types error identified frequent factual errors context errors omission errors chatgpt performance greatly exceeded passing mark exams multiple choice examinations provide benchmark llm performance comparable human demonstrations knowledge also highlighting errors llms make understanding reasons behind chatgpts errors allows us develop strategies prevent medical devices incorporate llm technology