Jia Xue, Micheal L Shier, Junxiang Chen, Yirun Wang, Chengda Zheng, Chen Chen
A Typology of Social Media Use by Human Service Nonprofits: Mixed Methods Study
Journal of medical Internet research
background generative artificial intelligence ai systems chatgpt35 claude2 may assist explaining complex medical science topics studies shown ai solve complicated physiology problems require critical thinking analysis however studies required validate effectiveness ai answering conceptual multiplechoice questions mcqs human physiology objective study aimed evaluate compare proficiency chatgpt35 claude2 answering explaining curated set mcqs medical physiology methods crosssectional study set 55 mcqs 10 competencies medical physiology purposefully constructed required comprehension problemsolving analytical skills solve mcqs structured prompt response generation presented chatgpt35 claude2 explanations provided ai systems documented excel spreadsheet three authors subjected explanations rating process using scale 0 3 rating 0 assigned incorrect 1 partially correct 2 correct explanation aspects missing 3 perfectly correct explanation ai models evaluated ability choose correct answer option provide clear comprehensive explanations mcqs mannwhitney u test used compare ai responses fleiss multirater kappa used determine score agreement among three raters statistical significance level decided p 005 results claude2 answered 40 mcqs correctly significantly higher 26 correct responses chatgpt35 rating distribution explanations generated claude2 significantly higher chatgpt35 values 0804 0818 claude2 chatgpt35 respectively conclusion terms answering elucidating conceptual mcqs medical physiology claude2 surpassed chatgpt35 however accessing claude2 india requires use virtual private network may raise security concerns