Malik Sallam|Dhia Mousa
Evaluating ChatGPT performance in Arabic dialects: A comparative study showing defects in responding to Jordanian and Tunisian general health prompts
University of Jordan|University of Jordan
background role artificial intelligence ai increasingly recognized enhance digital health literacy particular importance widespread availability popularity ai chatbots chatgpt possible impact health literacy involves need understand ai models performance across different languages dialects cultural contexts study aimed evaluate chatgpt performance response prompting two different arabic dialects namely tunisian jordanianx0d methods descriptive study followed metrics checklist design reporting ai based studies healthcare ten general health queries translated tunisian jordanian dialects arabic bilingual native speakers performance two ai models chatgpt35 chatgpt4 response tunisian jordanian english evaluated using clear tool tailored assessment health information generated ai modelsx0d results chatgpt35 performance categorized average tunisian arabic overall clear score 283 compared average score 340 jordanian arabic chatgpt4 showed similar pattern marginally better outcomes clear score 320 tunisian rated average average performance jordanian clear score 353 clear components consistently showed superior performance jordanian dialect models despite lack statistical significance using english content reference responses tunisian jordanian dialects significantly inferior plt001x0d conclusion findings highlight critical dialectical performance gap chatgpt underlining need enhance linguistic cultural diversity ai models development particularly healthrelated content collaborative efforts among ai developers linguists healthcare professionals needed improve performance ai models across different languages dialects cultural contexts future studies recommended broaden scope across extensive range languages dialects would help achieving equitable access health information across various communities