Erika Olsson, Helle Wallach-Kildemoes, Ban Ahmed, Pontus Ingman, Susanne Kaae, Sofia Kälvemark Sporrong
The influence of generic substitution on the content of patient-pharmacist communication in Swedish community pharmacies
The International journal of pharmacy practice
quantify chatgpts concordance expert otolaryngologists posed highlevel questions require blending rote memorization critical thinking crosssectional survey openais chatgpt35 platform two boardcertified otolaryngologists hz rs input 2 sets 30 textbased questions openended singleanswer multiplechoice chatgpt35 model responses rated scale correct partially correct incorrect otolaryngologist working simultaneously ai model interrater agreement percentage based binomial distribution calculating 95 confidence intervals performing significance tests statistical significance defined p 05 2sided tests testing openended questions chatgpt model 567 initially answering questions complete accuracy 867 chance answer accuracy corrected agreement 801 p 001 repeat questions chatgpt improved 733 complete accuracy 967 accuracy corrected agreement 888 p 001 multiplechoice questions chatgpt model performed substantially worse 433 correct chatgpt currently provide reliably accurate responses sophisticated questions otolaryngology professional societies must aware potential tool prevent unscrupulous use testtaking situations consider guidelines clinical scenarios expert clinical oversight still necessary myriad use cases eg hallucination