Sylviane Gentil, Jacky Montmain, Christophe Combastel
Combining FDI and AI approaches within causal-model-based diagnosis
IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society
large language models gpt4 generative pretrained transformer 4 increasingly used medicine medical education however models prone hallucinations ie outputs seem convincing factually incorrect currently unknown errors large language models relate different cognitive levels defined blooms taxonomy study aims explore gpt4 performs terms blooms taxonomy using psychosomatic medicine exam questions used large data set psychosomatic medicine multiplechoice questions n307 realworld results derived medical school exams gpt4 answered multiplechoice questions using 2 distinct prompt versions detailed short answers analyzed using quantitative approach qualitative approach focusing incorrectly answered questions categorized reasoning errors according hierarchical framework blooms taxonomy gpt4s performance answering exam questions yielded high success rate 93 284307 detailed prompt 91 278307 short prompt questions answered correctly gpt4 statistically significant higher difficulty questions answered incorrectly p002 detailed prompt p001 short prompt independent prompt gpt4s lowest exam performance 789 1519 thereby always surpassing pass threshold qualitative analysis incorrect answers based blooms taxonomy showed errors primarily remember 2968 understand 2368 cognitive levels specific issues arose recalling details understanding conceptual relationships adhering standardized guidelines gpt4 demonstrated remarkable success rate confronted psychosomatic medicine multiplechoice exam questions aligning previous findings evaluated blooms taxonomy data revealed gpt4 occasionally ignored specific facts remember provided illogical reasoning understand failed apply concepts new situation apply errors confidently presented could attributed inherent model biases tendency generate outputs maximize likelihood