The performance of OpenAI ChatGPT-4 and Google Gemini in virology multiple-choice questions: a comparative analysis of English and Arabic responses
Arabic is the native language for over 400 million people with a major role for effective communication in the majority of Middle East and North African countries . Nevertheless, English is the official language of teaching and learning in healthcare education across a majority of Arab countries . Therefore, university students are challenged by a linguistic shift at the start of healthcare education . The emergence of generative artificial intelligence (AI) models can provide support to bridge the linguistic challenge in healthcare for non-native English speakers . However, the utility of AI models should be preceded by a critical evaluation of the reliability of its generated content especially in non-English languages, given the dominant training of large language models (LLMs) in English . If generative AI models are integrated in education, the bias toward English in LLM training could undermine efforts to achieve global educational equity . The integration of generative AI models into various aspects of daily life has been marked by a growing interest . This trend was particularly notable in healthcare, where AI models can increase the operational efficiency and improve the quality of delivery of patient care and healthcare education . The potential benefits of AI in education are widely recognized; nevertheless, valid ethical concerns are recognized besides the concern regarding inaccuracies reported for the AI-generated content . The use of multiple-choice questions (MCQs) in healthcare education is recognized as a reliable method to evaluate the students’ achievement of learning outcomes . A relevant approach of classifying MCQs is the revised Bloom’s taxonomy which is based on cognitive functions ranging from basic knowledge recall to the application of knowledge in problem-solving and systematic analysis of various concepts . Since the utility and integration of generative AI in various aspects of healthcare education appears inevitable, it is important to consider the strengths and limitations of such innovative technology . This involves a thorough evaluation of the performance of the widely used AI chatbots, such as ChatGPT and Gemini, in different educational contexts . Recent studies investigated the capability of different AI models to pass exams in different domains, with a wide variability in performance as reviewed recently by Newton and Xiromeriti . This variability can be attributed to different factors, such as the AI model used, the prompting approach, and importantly the language(s) used in prompting . Such findings highlight the necessity for continued research to elucidate the determinants of AI models’ performance, thereby informing the refinement of AI algorithms for improved performance and subsequent improved utility in various disciplines such as healthcare education . Therefore, the current study aimed to compare the performance of two prominent AI models (ChatGPT-4 versus Gemini) in English and Arabic languages within the specialized field of Virology. The original hypothesis postulated that generative AI models’ performance in English is superior to that in Arabic, inferred based on the presumed higher quality of training data available in English and based on the few reports describing this disparity in language performance . Highlighting the critical discrepancies in generative AI performance across languages can lead to identification of possible areas for improvement by AI developers. 

Study design The study utilized the METRICS checklist for the design and reporting of generative AI studies in healthcare . The basis of the study was randomly selected 40 Virology MCQs, used for testing of medical students during the period 2017–2022. The MCQs were fully designed by the first author (M.S.), with a PhD degree in Clinical Virology. The MCQs were original, without any copyright issues. The MCQs were classified based on the revised Bloom’s taxonomy into two cognitive levels: higher involving 20 “Remember” and “Understand” MCQs; and lower involving 20 “Apply” and “Analyze” MCQs classified based on a consensus between the first and senior authors. The MCQs were translated into Arabic by the first author and back translated into English by the senior author, both bilingual in English and Arabic. The study was approved by the institutional review board (IRB) at the Faculty of Pharmacy – Applied Science Private University (reference number: 2024-PHA-5). Models of generative AI tested, settings, and testing time Two generative AI models were selected for testing based on their relevance, popularity, and advanced capabilities. The two models were ChatGPT-4 (OpenAI, San Francisco, CA) , and Gemini (Google, Mountain View, CA) . We did not use the “regenerate response” or “modify response” features and refrained from providing any feedback for the two models to avoid feedback bias. Testing was conducted during 17 February–2 March 2024. Prompt and language specificity The following exact prompt was used: “For the following virology MCQ, please select the single most appropriate answer with an explanation for the rationale behind selecting this choice and excluding the other choices”. All MCQs were presented independently one-by-one in English and one-by-one in Arabic. AI content evaluation approach and individual involvement in evaluation First, we assessed the correctness of responses based on the key answers of the MCQs. Then, subjective evaluation of the AI generated content was based on a modified version of the CLEAR tool . This involved assessing the content on three dimensions: (1) Completeness of the generated response; (2) Accuracy reflected by lack of false knowledge and the content being evidence-based; and (3) Appropriateness and relevance of content being easy to understand, well organized, and free from irrelevant content (Additional file 1) . Each dimension was evaluated using a 5-point Likert scale, with 1 indicating “poor” and 5 representing “excellent” . To enhance the objectivity of the assessment, a predefined list of criteria pertinent to each MCQ was established through discussions between the first and senior authors. Subsequently, the content produced by the two models underwent independent evaluation by the two raters. The CLEAR score for each piece of content was calculated by averaging the scores across the three assessed dimensions. The overall average CLEAR scores were then derived by averaging the scores assigned by the two raters. Statistical and data analyses The statistical analysis was conducted using IBM SPSS Statistics Version 26.0 (Armonk, NY: IBM Corp). To explore the associations between categorical variables, we employed two-sided Fisher’s exact test (FET), while the associations between the scale variable (CLEAR score) and categorical variables was assessed using the non-parametric the Mann–Whitney U test (M-W). The Kolmogorov-Smirnov test was employed to confirm the non-normality of the scale variable. 

The study utilized the METRICS checklist for the design and reporting of generative AI studies in healthcare . The basis of the study was randomly selected 40 Virology MCQs, used for testing of medical students during the period 2017–2022. The MCQs were fully designed by the first author (M.S.), with a PhD degree in Clinical Virology. The MCQs were original, without any copyright issues. The MCQs were classified based on the revised Bloom’s taxonomy into two cognitive levels: higher involving 20 “Remember” and “Understand” MCQs; and lower involving 20 “Apply” and “Analyze” MCQs classified based on a consensus between the first and senior authors. The MCQs were translated into Arabic by the first author and back translated into English by the senior author, both bilingual in English and Arabic. The study was approved by the institutional review board (IRB) at the Faculty of Pharmacy – Applied Science Private University (reference number: 2024-PHA-5). 

Two generative AI models were selected for testing based on their relevance, popularity, and advanced capabilities. The two models were ChatGPT-4 (OpenAI, San Francisco, CA) , and Gemini (Google, Mountain View, CA) . We did not use the “regenerate response” or “modify response” features and refrained from providing any feedback for the two models to avoid feedback bias. Testing was conducted during 17 February–2 March 2024. 

The following exact prompt was used: “For the following virology MCQ, please select the single most appropriate answer with an explanation for the rationale behind selecting this choice and excluding the other choices”. All MCQs were presented independently one-by-one in English and one-by-one in Arabic. 

First, we assessed the correctness of responses based on the key answers of the MCQs. Then, subjective evaluation of the AI generated content was based on a modified version of the CLEAR tool . This involved assessing the content on three dimensions: (1) Completeness of the generated response; (2) Accuracy reflected by lack of false knowledge and the content being evidence-based; and (3) Appropriateness and relevance of content being easy to understand, well organized, and free from irrelevant content (Additional file 1) . Each dimension was evaluated using a 5-point Likert scale, with 1 indicating “poor” and 5 representing “excellent” . To enhance the objectivity of the assessment, a predefined list of criteria pertinent to each MCQ was established through discussions between the first and senior authors. Subsequently, the content produced by the two models underwent independent evaluation by the two raters. The CLEAR score for each piece of content was calculated by averaging the scores across the three assessed dimensions. The overall average CLEAR scores were then derived by averaging the scores assigned by the two raters. 

The statistical analysis was conducted using IBM SPSS Statistics Version 26.0 (Armonk, NY: IBM Corp). To explore the associations between categorical variables, we employed two-sided Fisher’s exact test (FET), while the associations between the scale variable (CLEAR score) and categorical variables was assessed using the non-parametric the Mann–Whitney U test (M-W). The Kolmogorov-Smirnov test was employed to confirm the non-normality of the scale variable. 

General performance of ChatGPT-4 versus Gemini in English and Arabic A higher percentage of correct responses for both generative AI models was observed in English compared to Arabic despite the lack of statistical significance. For Gemini in Arabic, the total number of correct responses was 22/40 (55.0%) while the correct responses for the same MCQs in English was 25/40 (62.5%, P = .650, FET). A similar trend was observed for ChatGPT-4 with correct responses in Arabic at 26/40 (65.0%) compared to 32/40 (80.0%) in English ( P = .210, FET). In Arabic, higher number of correct responses was seen in ChatGPT-4 compared to Gemini (65.0% vs. 55.0%, P = .494, FET) and similarly higher number of correct responses was observed in English for ChatGPT-4 compared to Gemini (80.0% vs. 62.5%, P = .137, FET). Performance of ChatGPT-4 and Gemini based on revised Bloom’s categories In the evaluation of Gemini and ChatGPT-4 performance across the two revised Bloom’s cognitive domains, the overall performance was consistently better in lower cognitive domain (Table ). Within each revised Bloom’s domain, ChatGPT-4 was marginally better in performance outscoring Gemini as follows: in lower cognitive domain, ChatGPT-4 had 35 correct responses (87.5%) vs. 28 Gemini correct responses (70.0%, P = .099, FET). In higher cognitive domain, ChatGPT-4 had 23 correct responses (57.5%) vs. 19 Gemini correct responses (47.5%, P = .502, FET). Performance of ChatGPT-4 versus Gemini based on the average CLEAR scores The performance of both generative AI models was superior in English as opposed to Arabic based on the average CLEAR scores as follows. For Gemini, the average CLEAR score was 3.48 ± 1.16 in Arabic compared to 4.00 ± 1.06 in English ( P = .022, M-W, Fig. A). For ChatGPT-4, the average CLEAR score was 4.20 ± 0.74 in Arabic compared to 4.68 ± 0.57 in English ( P = .001, M-W, Fig. B). In Arabic, Gemini received lower average CLEAR scores compared to ChatGPT-4 (3.48 ± 1.16 vs. 4.20 ± 0.74, P = .005, M-W, Fig. ), and the same pattern was noticed for English language with Gemini having lower average CLEAR scores compared to ChatGPT-4 (4.00 ± 1.06 vs. 4.68 ± 0.57, P = .002, M-W, Fig. ). Table illustrates the performance of the two AI models per CLEAR score component 

A higher percentage of correct responses for both generative AI models was observed in English compared to Arabic despite the lack of statistical significance. For Gemini in Arabic, the total number of correct responses was 22/40 (55.0%) while the correct responses for the same MCQs in English was 25/40 (62.5%, P = .650, FET). A similar trend was observed for ChatGPT-4 with correct responses in Arabic at 26/40 (65.0%) compared to 32/40 (80.0%) in English ( P = .210, FET). In Arabic, higher number of correct responses was seen in ChatGPT-4 compared to Gemini (65.0% vs. 55.0%, P = .494, FET) and similarly higher number of correct responses was observed in English for ChatGPT-4 compared to Gemini (80.0% vs. 62.5%, P = .137, FET). 

In the evaluation of Gemini and ChatGPT-4 performance across the two revised Bloom’s cognitive domains, the overall performance was consistently better in lower cognitive domain (Table ). Within each revised Bloom’s domain, ChatGPT-4 was marginally better in performance outscoring Gemini as follows: in lower cognitive domain, ChatGPT-4 had 35 correct responses (87.5%) vs. 28 Gemini correct responses (70.0%, P = .099, FET). In higher cognitive domain, ChatGPT-4 had 23 correct responses (57.5%) vs. 19 Gemini correct responses (47.5%, P = .502, FET). 

The performance of both generative AI models was superior in English as opposed to Arabic based on the average CLEAR scores as follows. For Gemini, the average CLEAR score was 3.48 ± 1.16 in Arabic compared to 4.00 ± 1.06 in English ( P = .022, M-W, Fig. A). For ChatGPT-4, the average CLEAR score was 4.20 ± 0.74 in Arabic compared to 4.68 ± 0.57 in English ( P = .001, M-W, Fig. B). In Arabic, Gemini received lower average CLEAR scores compared to ChatGPT-4 (3.48 ± 1.16 vs. 4.20 ± 0.74, P = .005, M-W, Fig. ), and the same pattern was noticed for English language with Gemini having lower average CLEAR scores compared to ChatGPT-4 (4.00 ± 1.06 vs. 4.68 ± 0.57, P = .002, M-W, Fig. ). Table illustrates the performance of the two AI models per CLEAR score component 

The current study focused on comparative analysis of the generative AI models Gemini and ChatGPT-4 abilities to answer Virology MCQs across English and Arabic languages. The findings revealed potential limitations inherent in the current versions of AI technologies which should be addressed prior to its incorporation in healthcare education especially for non-English speakers. The results highlighted a discernible performance disparity between English and Arabic, with both AI models showing a lower accuracy in Arabic. This finding can be attributed to challenges encountered within LLMs’ processing capabilities, particularly for languages that possess complex grammatical structures or languages with limited digital resources. The reduced accuracy in Arabic emphasizes the necessity for enriched training datasets that more comprehensively cover the linguistic diversity inherent in global languages. This comes in light of growing evidence of lower performance of different generative AI models in non-English languages. In line with our observations, Samaan et al. reported ChatGPT lower accuracy in Arabic compared to English for cirrhosis-related queries . Similarly, Banimelhem and Amayreh reported suboptimal English to Arabic translation capabilities for ChatGPT . Additionally, a recent study showed the superior performance of four generative AI models in English compared to Arabic in infectious disease queries , while an earlier study showed the inferior performance of ChatGPT in general health queries in Arabic dialects . Additionally, the inferior performance of AI chatbots was reported in other non-English languages including Chinese , Polish , and Spanish . Interestingly, the study findings showed that both Gemini and ChatGPT-4 struggled with higher cognitive MCQs, which need advanced critical thinking and problem-solving skills. This limitation of generative AI performance is particularly relevant in healthcare education, where the ability to apply knowledge creatively and critically is essential . The observed limitation raises concerns about the current reliability of AI as an educational tool, which was reported in the context of various AI chatbots . Collectively, these results highlight the critical areas for future development and improvement in AI training approaches. Of note, this study highlighted ChatGPT-4 superior performance compared to Gemini in processing both Arabic and English languages across various cognitive levels, particularly emphasizing a pronounced advantage in addressing higher cognitive MCQs in Arabic. These findings might hint to OpenAI leading position in the development of LLMs, while also acknowledging the continued need for enhancements to improve performance in educational contexts . Finally, this study showed the need for substantial improvements in generative AI training to enhance performance in non-English languages and in processing of higher-order cognitive queries. Addressing these challenges can improve the quality of AI-generated content and ensure its reliability, rendering AI chatbots as effective educational tools across diverse linguistic and cultural contexts. In conclusion, the study findings showed the capabilities and limitations of ChatGPT-4 and Gemini in the future of AI-assisted education. The variations in performance observed between languages and cognitive categories highlight the need for continued research, development, and optimization of generative AI models. A special attention should be paid into enhancing the linguistic diversity and cognitive understanding capabilities of generative AI models to achieve global educational equity. Limitations The study limitations included the limited number of MCQs, which can restrict the scope of performance evaluation in this study. The subjective assessment of AI-generated content based on the CLEAR scores is another limitation highlighting the need for caution in interpretation. Additionally, this study focused solely on Virology MCQs, which may limit the generalizability of the findings to other healthcare disciplines. Moreover, the rapid evolution of LLMs highlights that the results may not fully reflect the evolving capabilities of the same generative AI models over time. 

The study limitations included the limited number of MCQs, which can restrict the scope of performance evaluation in this study. The subjective assessment of AI-generated content based on the CLEAR scores is another limitation highlighting the need for caution in interpretation. Additionally, this study focused solely on Virology MCQs, which may limit the generalizability of the findings to other healthcare disciplines. Moreover, the rapid evolution of LLMs highlights that the results may not fully reflect the evolving capabilities of the same generative AI models over time. 

Below is the link to the electronic supplementary material. Supplementary Material 1 

