Olabimpe Asupoto, Shamsuddin Anwar, Alysse G Wurcel
A health literacy analysis of online patient-directed educational materials about mycobacterium avium complex
Journal of clinical tuberculosis and other mycobacterial diseases
artificial intelligence ai applications specifically generative pretrained transformers shown potential medical education boardstyle examinations assess capability conducted study comparing performance gpt35 gpt4 american urological association aua 2022 selfassessment study program sasp exams 20122023 used standardized prompt administer questions aua sasp exams spanning 20122023 totalling 1679 questions performance two ai models gpt35 gpt4 evaluated based number questions answered correctly statistical analysis performed using fishers exact test independent sample ttests compare performance gpt4 gpt35 among test years urology topic areas percentile scores calculable however score 50 required acquire cme credits aua sasp exams analysis showed significantly superior performance gpt4 scored 50 across exam years except 2018 scores ranging 4864 contrast gpt35 consistently scored threshold scores ranging 2638 total combined score gpt4 55 significantly higher 33 achieved gpt35 odds ratio 25 95 confidence interval ci 2229 p0001 gpt4 significantly outperformed gpt35 among aua sasp test years 20122023 mean difference 23 t22 14 95 ci 1926 p0001 well among urology topic areas mean difference 21 t5255 95 ci 1329 p0001 gpt4 scored significantly higher gpt35 aua sasp exams overall performance across test years various urology topic areas suggests improvement evolving ai language models answering clinical urology questions however certain aspects medical knowledge clinical reasoning remain challenging ai language models