Assessing ChatGPT’s capacity for clinical decision support in pediatrics: A comparative study with pediatricians using KIDMAP of Rasch analysis
ChatGPT, which stands for Chat Generative Pretrained Transformer, is a cutting-edge language model created by OpenAI.  Its purpose is to generate text that resembles human language when given a prompt or context.  ChatGPT has been trained on a vast amount of Internet text data, which allows it to comprehend and generate a wide range of language styles and topics.  ChatGPT is a versatile language model developed by OpenAI with numerous applications.  One major use of ChatGPT is for text generation, which has the potential to transform how we create content, including academic publications.  With the increasing sophistication of language models such as ChatGPT, distinguishing between text generated by humans and that produced by artificial intelligence (AI) will become increasingly challenging.  It is able to respond to user prompts by answering questions and writing essays, poems, love letters, computer code, or business plans. ChatGPT can solve problems, including math or physics.  1.1. Assessing ChatGPT’s capacity for clinical decision support According to Gilat and Cole,  there are significant concerns regarding ChatGPT’s reliability and fairness, echoing reports from the popular press that have highlighted the chatbot’s issues with misinformation. The authors warn that ChatGPT may not always provide accurate information,  and there are worries that ChatGPT could be manipulated to disseminate false information  or used to generate “deepfakes.”  Prior research in medical question answering has often focused on more specific tasks to evaluate the model performance of ChatGPT.  For example, Jin et al  achieved 68.1% accuracy with their model for answering yes-or-no questions that can be found in PubMed-available abstracts; ChatGPT was performed with accuracy on 4 datasets on the United States Medical Licensing Examination: 64.4% (56/87) on NBME-Free-Step1, and 57.8% (59/102) on NBME-Free-Step2  ; a ChatGPT score of 1.83 (out of 2), a select all that apply (SATA) average percentage correct rate of 88.9% for breast cancer screening prompts, a SATA average percentage correct rate of 58.3% for breast pain prompts, and an average Open-ended (OE) score of 1.125 (out of 2) was achieved by ChatGPT  ; answers to prompts about Kawasaki disease  in ChatGPT can be graded by A−  ; Ha and Yaneva  reported only a 29% accuracy on 454 USMLE Step 1 and Step 2 questions on automatic question answering for medical multiple choice questions (MCQs). 1.2. ChatGPT for patients in pediatrics Pediatrics is a specialized medical field that deals with the healthcare needs of infants, children, and adolescents. Clinical decision-making in pediatrics requires a thorough understanding of age-specific developmental patterns, growth milestones, and unique clinical presentations.  Assessing ChatGPT’s capacity for clinical decision support (CDS) in pediatrics is particularly relevant as it addresses the specific challenges and considerations in this field.  Effective clinical decision-making in pediatrics can significantly impact patient outcomes, as early detection and appropriate management of pediatric conditions are crucial. By evaluating ChatGPT’s performance in pediatric CDS, this study aims to enhance the quality of care provided to pediatric patients and potentially improve health outcomes in this vulnerable population.  While the use of large language models has gained attention in various domains, there is a scarcity of research specifically evaluating their application in pediatrics. It is required to fill a gap in the existing literature by focusing on pediatrics and assessing the capabilities of ChatGPT in this specialized medical field.  Furthermore, pediatric medicine involves unique considerations such as growth and development, age-specific clinical presentations, and the need for age-appropriate communication. By evaluating ChatGPT’s performance in pediatrics, we can understand how well the ChatGPT can adapt to the specific needs of pediatric patients and provide accurate and contextually appropriate CDS. In this study, the first research question is what is ChatGPT’s grade (e.g., A, B, C, D, or E) about questions in the field of pediatrics. 1.3. Subjective judgments of ChatGPT in an OE format In ChatGPT, there are 2 types of prompts: MCQs  and OEs.  The OE format of ChatGPT is more subjective than the MCQs. Examining the correlation coefficient and differences in measures of ChatGPT scores between 2 pediatricians assessing ChatGPT answers in an OE format would be useful and necessary. In the second research question, we examine whether Rasch analysis  based on the rating scale model (RSM)  can be used to examine differences in the assessment of ChatGPT performance evaluated by pediatricians. A study using Rasch analysis to examine the performance of ChatGPT has not yet been published in the literature. It is thus necessary to compare the assessment differences in ChatGPT responses between judges. 1.4. Features of Rasch analysis Rasch analysis  is a statistical technique used to analyze the performance of individuals on tests or assessments. When applied to examine the performance of ChatGPT,  Rasch analysis can help researchers evaluate the quality of its responses and identify any areas where improvements may be needed.  Some features of Rasch analysis applied to examine ChatGPT performance are listed below: Item difficulty : Rasch analysis can help to determine the level of difficulty of each question or prompt that ChatGPT is presented with. This information can be used to identify areas where ChatGPT may be struggling (e.g., performance is lower when the questions are harder) or excelling (e.g., performance is higher when the questions are easier). Person ability : Rasch analysis can also help to measure the ability level of ChatGPT based on its responses to the questions or prompts. This can provide insight into ChatGPT’s overall performance and help to identify areas where improvements may be needed. Item fit statistics : Rasch analysis produces item fit statistics, which measure the degree to which each question or prompt fits the overall model. This information can be used to identify items that may need to be revised or removed from the assessment.  Differential item functioning (DIF )  : Rasch analysis can detect DIF, which occurs when different groups of people (such as males and females or people from different cultural backgrounds) respond differently to the same item. This information can be used to identify potentially biased items and improve the fairness of the assessment.  Three visualizations are frequently applied to present item features and person measures, including Wright Map  with groups, DIF  using forest plots,  and KIDMAP.  1.5. Study aims This study aimed to determine the ChatGPT grade (e.g., A, B, C, or D) compared to a normal sample and the difference in the assessment of ChatGPT between pediatricians.   According to Gilat and Cole,  there are significant concerns regarding ChatGPT’s reliability and fairness, echoing reports from the popular press that have highlighted the chatbot’s issues with misinformation. The authors warn that ChatGPT may not always provide accurate information,  and there are worries that ChatGPT could be manipulated to disseminate false information  or used to generate “deepfakes.”  Prior research in medical question answering has often focused on more specific tasks to evaluate the model performance of ChatGPT.  For example, Jin et al  achieved 68.1% accuracy with their model for answering yes-or-no questions that can be found in PubMed-available abstracts; ChatGPT was performed with accuracy on 4 datasets on the United States Medical Licensing Examination: 64.4% (56/87) on NBME-Free-Step1, and 57.8% (59/102) on NBME-Free-Step2  ; a ChatGPT score of 1.83 (out of 2), a select all that apply (SATA) average percentage correct rate of 88.9% for breast cancer screening prompts, a SATA average percentage correct rate of 58.3% for breast pain prompts, and an average Open-ended (OE) score of 1.125 (out of 2) was achieved by ChatGPT  ; answers to prompts about Kawasaki disease  in ChatGPT can be graded by A−  ; Ha and Yaneva  reported only a 29% accuracy on 454 USMLE Step 1 and Step 2 questions on automatic question answering for medical multiple choice questions (MCQs).   Pediatrics is a specialized medical field that deals with the healthcare needs of infants, children, and adolescents. Clinical decision-making in pediatrics requires a thorough understanding of age-specific developmental patterns, growth milestones, and unique clinical presentations.  Assessing ChatGPT’s capacity for clinical decision support (CDS) in pediatrics is particularly relevant as it addresses the specific challenges and considerations in this field.  Effective clinical decision-making in pediatrics can significantly impact patient outcomes, as early detection and appropriate management of pediatric conditions are crucial. By evaluating ChatGPT’s performance in pediatric CDS, this study aims to enhance the quality of care provided to pediatric patients and potentially improve health outcomes in this vulnerable population.  While the use of large language models has gained attention in various domains, there is a scarcity of research specifically evaluating their application in pediatrics. It is required to fill a gap in the existing literature by focusing on pediatrics and assessing the capabilities of ChatGPT in this specialized medical field.  Furthermore, pediatric medicine involves unique considerations such as growth and development, age-specific clinical presentations, and the need for age-appropriate communication. By evaluating ChatGPT’s performance in pediatrics, we can understand how well the ChatGPT can adapt to the specific needs of pediatric patients and provide accurate and contextually appropriate CDS. In this study, the first research question is what is ChatGPT’s grade (e.g., A, B, C, D, or E) about questions in the field of pediatrics.   In ChatGPT, there are 2 types of prompts: MCQs  and OEs.  The OE format of ChatGPT is more subjective than the MCQs. Examining the correlation coefficient and differences in measures of ChatGPT scores between 2 pediatricians assessing ChatGPT answers in an OE format would be useful and necessary. In the second research question, we examine whether Rasch analysis  based on the rating scale model (RSM)  can be used to examine differences in the assessment of ChatGPT performance evaluated by pediatricians. A study using Rasch analysis to examine the performance of ChatGPT has not yet been published in the literature. It is thus necessary to compare the assessment differences in ChatGPT responses between judges.   Rasch analysis  is a statistical technique used to analyze the performance of individuals on tests or assessments. When applied to examine the performance of ChatGPT,  Rasch analysis can help researchers evaluate the quality of its responses and identify any areas where improvements may be needed.  Some features of Rasch analysis applied to examine ChatGPT performance are listed below: Item difficulty : Rasch analysis can help to determine the level of difficulty of each question or prompt that ChatGPT is presented with. This information can be used to identify areas where ChatGPT may be struggling (e.g., performance is lower when the questions are harder) or excelling (e.g., performance is higher when the questions are easier). Person ability : Rasch analysis can also help to measure the ability level of ChatGPT based on its responses to the questions or prompts. This can provide insight into ChatGPT’s overall performance and help to identify areas where improvements may be needed. Item fit statistics : Rasch analysis produces item fit statistics, which measure the degree to which each question or prompt fits the overall model. This information can be used to identify items that may need to be revised or removed from the assessment.  Differential item functioning (DIF )  : Rasch analysis can detect DIF, which occurs when different groups of people (such as males and females or people from different cultural backgrounds) respond differently to the same item. This information can be used to identify potentially biased items and improve the fairness of the assessment.  Three visualizations are frequently applied to present item features and person measures, including Wright Map  with groups, DIF  using forest plots,  and KIDMAP.    This study aimed to determine the ChatGPT grade (e.g., A, B, C, or D) compared to a normal sample and the difference in the assessment of ChatGPT between pediatricians.   2.1. Data source In the study, 300 virtual participants responded to 8 symptom-related items (Table ) with 5 ordinal categories  and were analyzed according to item difficulty (with logit unit, i.e., log odds: [−2, −1.33, −0.66, 0.00,0.66, 1.33, 1.99, and 2.5]) in the Rasch model in accordance with normal distribution of person measures. Each item in Table was prompted. Answers from ChatGPT were gathered and scored on a scale of 0 to 100 by 2 pediatricians; see Supplemental Digital Content 1, http://links.lww.com/MD/J151 (responses by CHatGPT), Table (comments by pediatricians), and the MP4 video (demonstrations of prompts and answers from ChatGPT).  The 302 virtual participants were randomly divided into 2 groups based on gender. There were 5 grades (from A to E) assigned based on the person measures (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits). By using the formula round (Score − minimum) ÷ (maximum − minimum) × 5, 0), the scores from the 2 sets of assessments by the 2 pediatricians were transformed into 5-category scores from 0 to 4. As a final step, the 302 individuals (including the 2 assessed by the 2 pediatricians) were analyzed using Raschonline software.  This study used simulated data shown in Supplemental Digital Content 2, http://links.lww.com/MD/J152 . According does not require ethical approval due to its design. 2.2. R person responses using RaschOnline RaschOnline  based on Rasch RSM  was used to analyze the data. The multitomous data can therefore be analyzed using RaschOnline. In the Rasch model, the probability and standardized error (SE) of the person estimate can be expressed as Equations 1 and 2: f ( θ n − δ i ) = P r o ( X n i | θ ) = e ( θ n − δ i ) 1 + e ( θ n − δ i ) (1) S E ( θ ) ^ = 1 ∑ i = 1 L ( p ′ i ( θ ) ) 2 P i ( θ ) Q i ( θ ) (2) where θ and δ are defined as person ability and item difficulty, respectively. L is the item length. p ′ i ( θ ) is the first-order derivative for person n with ability ′ on item i in Equation 1; P i ( θ ) is identical to Equation 1; Q i ( θ ) refers to Equation 3, as shown below: Q i ( θ ) = 1 − P i ( θ ) = 1 − e ( θ n − δ i ) 1 + e ( θ n − δ i ) = 1 + e ( θ n − δ i ) − e ( θ n − δ i ) 1 + e ( θ n − δ i ) = 1 1 + e ( θ n − δ i ) (3) The processes of the first-order derivative for S E ( θ ) ^ in Equation 2 are described below: P ′ i ( θ ) = e ( θ n − δ i ) ( 1 + e ( θ n − δ i ) ) − e ( θ n − δ i ) ( 0 + e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 = e ( θ n − δ i ) ( 1 + e ( θ n − δ i ) − e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 = e ( θ n − δ i ) × 1 ( 1 + e ( θ n − δ i ) ) 2 = P i ( θ ) × Q i ( θ ) (4) Equation 2 can then be extended to Equation 5, indicating that person SE is associated with the inverse of its total variances across all items. S E ( θ ) ^ = 1 ∑ i = 1 L ( P i ′ ( θ ) ) 2 P i ( θ ) Q i ( θ ) = 1 ∑ i = 1 L ( P i ( θ ) × Q i ( θ ) ) 2 P i ( θ ) Q i ( θ ) = 1 ∑ i = 1 L P i ( θ ) × Q i ( θ ) (5) The processes of the first-order derivative for variance (denoted by V a r n i ) on ( θ n − δ i ) can also be described based on Equation 4 and are shown below: V a r n i = f ′ ( θ n − δ i ) = ( e ( θ n − δ i ) 1 + e ( θ n − δ i ) ) ′ = ( e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 (6) If ( e ( θ n − δ i ) ) is replaced with e x p ∑ x k = 0 ( ​ ​ θ n − ( δ i + τ k ) ) , the variance for person n on item i adaptive to the RSM equals the result in Equation 6.  Through the Newton–Raphson iteration method  and the person estimate and S E ( θ ) ^ in Equations 1 and 5, RaschOnline  was programed and developed. To visualize item features and individual measures, several visualizations are commonly used, including Wright Map  with groups, DIF  using forest plots,  and KIDMAP.  The method of drawing these visualizations refers to the manual of RaschOnline  and Supplemental Digital Content 3, http://links.lww.com/MD/J153 (how to conduct this study). 2.3. Two tasks required to achieve the study goals 2.3.1. Determining the grade of ChatGPT performance (Task 1). Rasch analysis was employed to observe item features and person responses (e.g., the determination of grade in ChatGPT performance), and some significant terms in Rasch analysis are defined below: Wright Map  with groups was used to display sample distributions of groups compared to the overall sample of item difficulties and person performance abilities with a log-odds(=logit) unit on a common equal-interval continuum. Analysis of variance (ANOVA) was performed to examine differences in measures between groups (e.g., Female and Male). DIF  analysis was performed to examine whether there are items in favor of a specific group (e.g., Female or Male). For example, a specific item might be in favor of female (or male) to be easy in response. Details about DIF are in Supplemental Digital Content 3, http://links.lww.com/MD/J153 . In the KIDMAP,  individual person performance is assessed using the Z score (observed × expected ÷ SD) across items. The Z scores of items outside the upper limit (> 2.0) indicate that the observed responses are significantly higher than those expected or Z scores (<-2.0) with unexpected responses based on the individual’s ability. In Task 1, the first study goal of the determination of ChatGPT’s grade (e.g., A, B, C, D, or E) compared to a normal sample would be achieved, based on the criteria of grades from A to E (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits). and (2) the difference in the assessment of ChatGPT between pediatricians by Rasch measures and their corresponding individual standard errors (SE). 2.3.2. The difference in assessment of ChatGPT between pediatricians (Task 2). Using Rasch analysis, the 2 participants assessed by the 2 pediatricians had their own individual measures and standard errors (SEs). If the 95% confidence intervals (CIs) of the 2 measures are overlaid, no difference is apparent between them. In Task 2, the second study goal of the difference in assessment of ChatGPT between pediatricians would be determined based on their 95% CIs. 2.4. Statistical tools and data analysis To perform descriptive statistics and frequency distributions, as well as to calculate the correlation coefficient of scores assessed by the 2 pediatricians and the difference in person measures between groups, IBM SPSS Statistics 22.0 for Windows (SPSS Inc., Chicago, IL) and MedCalc 9.5.0.0 for Windows (MedCalc Software, Ostend, Belgium) were used. Type I errors were set at a significance level of 0.05. In addition, we calculated the intraclass correlation coefficient (ICC) to evaluate interrater reliability.  ICC estimates and 95% CIs were reported based on a 2-way mixed-effects model (mean of k raters type and consistency definition). For ICC estimates, below 0.5 represents low reliability; 0.5 to 0.74 represents moderate reliability; 0.75 to 0.90 represents good reliability; and greater than 0.9 represents excellent reliability  ; see Supplemental Digital Content 4, http://links.lww.com/MD/J154 . The 2 tables contain 8 items and comments by pediatricians, as shown in Section 2.1. Six visualizations include distribution of item difficulties, DIF or DIF-free in items between groups, scatter plot for assessments by the 2 pediatricians, Wright map to observe item fit statistics (e.g., Infit mean square error, MNSQ < 1.5),  KIDMAP for ChatGPT A, and KIDMAP for ChatGPT B. Details about how to conduct this study can be found in the link (MP4) provided in reference  and in the Supplementary Digital Contents.   In the study, 300 virtual participants responded to 8 symptom-related items (Table ) with 5 ordinal categories  and were analyzed according to item difficulty (with logit unit, i.e., log odds: [−2, −1.33, −0.66, 0.00,0.66, 1.33, 1.99, and 2.5]) in the Rasch model in accordance with normal distribution of person measures. Each item in Table was prompted. Answers from ChatGPT were gathered and scored on a scale of 0 to 100 by 2 pediatricians; see Supplemental Digital Content 1, http://links.lww.com/MD/J151 (responses by CHatGPT), Table (comments by pediatricians), and the MP4 video (demonstrations of prompts and answers from ChatGPT).  The 302 virtual participants were randomly divided into 2 groups based on gender. There were 5 grades (from A to E) assigned based on the person measures (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits). By using the formula round (Score − minimum) ÷ (maximum − minimum) × 5, 0), the scores from the 2 sets of assessments by the 2 pediatricians were transformed into 5-category scores from 0 to 4. As a final step, the 302 individuals (including the 2 assessed by the 2 pediatricians) were analyzed using Raschonline software.  This study used simulated data shown in Supplemental Digital Content 2, http://links.lww.com/MD/J152 . According does not require ethical approval due to its design.   RaschOnline  based on Rasch RSM  was used to analyze the data. The multitomous data can therefore be analyzed using RaschOnline. In the Rasch model, the probability and standardized error (SE) of the person estimate can be expressed as Equations 1 and 2: f ( θ n − δ i ) = P r o ( X n i | θ ) = e ( θ n − δ i ) 1 + e ( θ n − δ i ) (1) S E ( θ ) ^ = 1 ∑ i = 1 L ( p ′ i ( θ ) ) 2 P i ( θ ) Q i ( θ ) (2) where θ and δ are defined as person ability and item difficulty, respectively. L is the item length. p ′ i ( θ ) is the first-order derivative for person n with ability ′ on item i in Equation 1; P i ( θ ) is identical to Equation 1; Q i ( θ ) refers to Equation 3, as shown below: Q i ( θ ) = 1 − P i ( θ ) = 1 − e ( θ n − δ i ) 1 + e ( θ n − δ i ) = 1 + e ( θ n − δ i ) − e ( θ n − δ i ) 1 + e ( θ n − δ i ) = 1 1 + e ( θ n − δ i ) (3) The processes of the first-order derivative for S E ( θ ) ^ in Equation 2 are described below: P ′ i ( θ ) = e ( θ n − δ i ) ( 1 + e ( θ n − δ i ) ) − e ( θ n − δ i ) ( 0 + e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 = e ( θ n − δ i ) ( 1 + e ( θ n − δ i ) − e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 = e ( θ n − δ i ) × 1 ( 1 + e ( θ n − δ i ) ) 2 = P i ( θ ) × Q i ( θ ) (4) Equation 2 can then be extended to Equation 5, indicating that person SE is associated with the inverse of its total variances across all items. S E ( θ ) ^ = 1 ∑ i = 1 L ( P i ′ ( θ ) ) 2 P i ( θ ) Q i ( θ ) = 1 ∑ i = 1 L ( P i ( θ ) × Q i ( θ ) ) 2 P i ( θ ) Q i ( θ ) = 1 ∑ i = 1 L P i ( θ ) × Q i ( θ ) (5) The processes of the first-order derivative for variance (denoted by V a r n i ) on ( θ n − δ i ) can also be described based on Equation 4 and are shown below: V a r n i = f ′ ( θ n − δ i ) = ( e ( θ n − δ i ) 1 + e ( θ n − δ i ) ) ′ = ( e ( θ n − δ i ) ) ( 1 + e ( θ n − δ i ) ) 2 (6) If ( e ( θ n − δ i ) ) is replaced with e x p ∑ x k = 0 ( ​ ​ θ n − ( δ i + τ k ) ) , the variance for person n on item i adaptive to the RSM equals the result in Equation 6.  Through the Newton–Raphson iteration method  and the person estimate and S E ( θ ) ^ in Equations 1 and 5, RaschOnline  was programed and developed. To visualize item features and individual measures, several visualizations are commonly used, including Wright Map  with groups, DIF  using forest plots,  and KIDMAP.  The method of drawing these visualizations refers to the manual of RaschOnline  and Supplemental Digital Content 3, http://links.lww.com/MD/J153 (how to conduct this study).   2.3.1. Determining the grade of ChatGPT performance (Task 1). Rasch analysis was employed to observe item features and person responses (e.g., the determination of grade in ChatGPT performance), and some significant terms in Rasch analysis are defined below: Wright Map  with groups was used to display sample distributions of groups compared to the overall sample of item difficulties and person performance abilities with a log-odds(=logit) unit on a common equal-interval continuum. Analysis of variance (ANOVA) was performed to examine differences in measures between groups (e.g., Female and Male). DIF  analysis was performed to examine whether there are items in favor of a specific group (e.g., Female or Male). For example, a specific item might be in favor of female (or male) to be easy in response. Details about DIF are in Supplemental Digital Content 3, http://links.lww.com/MD/J153 . In the KIDMAP,  individual person performance is assessed using the Z score (observed × expected ÷ SD) across items. The Z scores of items outside the upper limit (> 2.0) indicate that the observed responses are significantly higher than those expected or Z scores (<-2.0) with unexpected responses based on the individual’s ability. In Task 1, the first study goal of the determination of ChatGPT’s grade (e.g., A, B, C, D, or E) compared to a normal sample would be achieved, based on the criteria of grades from A to E (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits). and (2) the difference in the assessment of ChatGPT between pediatricians by Rasch measures and their corresponding individual standard errors (SE). 2.3.2. The difference in assessment of ChatGPT between pediatricians (Task 2). Using Rasch analysis, the 2 participants assessed by the 2 pediatricians had their own individual measures and standard errors (SEs). If the 95% confidence intervals (CIs) of the 2 measures are overlaid, no difference is apparent between them. In Task 2, the second study goal of the difference in assessment of ChatGPT between pediatricians would be determined based on their 95% CIs.   Rasch analysis was employed to observe item features and person responses (e.g., the determination of grade in ChatGPT performance), and some significant terms in Rasch analysis are defined below: Wright Map  with groups was used to display sample distributions of groups compared to the overall sample of item difficulties and person performance abilities with a log-odds(=logit) unit on a common equal-interval continuum. Analysis of variance (ANOVA) was performed to examine differences in measures between groups (e.g., Female and Male). DIF  analysis was performed to examine whether there are items in favor of a specific group (e.g., Female or Male). For example, a specific item might be in favor of female (or male) to be easy in response. Details about DIF are in Supplemental Digital Content 3, http://links.lww.com/MD/J153 . In the KIDMAP,  individual person performance is assessed using the Z score (observed × expected ÷ SD) across items. The Z scores of items outside the upper limit (> 2.0) indicate that the observed responses are significantly higher than those expected or Z scores (<-2.0) with unexpected responses based on the individual’s ability. In Task 1, the first study goal of the determination of ChatGPT’s grade (e.g., A, B, C, D, or E) compared to a normal sample would be achieved, based on the criteria of grades from A to E (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits). and (2) the difference in the assessment of ChatGPT between pediatricians by Rasch measures and their corresponding individual standard errors (SE).   Using Rasch analysis, the 2 participants assessed by the 2 pediatricians had their own individual measures and standard errors (SEs). If the 95% confidence intervals (CIs) of the 2 measures are overlaid, no difference is apparent between them. In Task 2, the second study goal of the difference in assessment of ChatGPT between pediatricians would be determined based on their 95% CIs.   To perform descriptive statistics and frequency distributions, as well as to calculate the correlation coefficient of scores assessed by the 2 pediatricians and the difference in person measures between groups, IBM SPSS Statistics 22.0 for Windows (SPSS Inc., Chicago, IL) and MedCalc 9.5.0.0 for Windows (MedCalc Software, Ostend, Belgium) were used. Type I errors were set at a significance level of 0.05. In addition, we calculated the intraclass correlation coefficient (ICC) to evaluate interrater reliability.  ICC estimates and 95% CIs were reported based on a 2-way mixed-effects model (mean of k raters type and consistency definition). For ICC estimates, below 0.5 represents low reliability; 0.5 to 0.74 represents moderate reliability; 0.75 to 0.90 represents good reliability; and greater than 0.9 represents excellent reliability  ; see Supplemental Digital Content 4, http://links.lww.com/MD/J154 . The 2 tables contain 8 items and comments by pediatricians, as shown in Section 2.1. Six visualizations include distribution of item difficulties, DIF or DIF-free in items between groups, scatter plot for assessments by the 2 pediatricians, Wright map to observe item fit statistics (e.g., Infit mean square error, MNSQ < 1.5),  KIDMAP for ChatGPT A, and KIDMAP for ChatGPT B. Details about how to conduct this study can be found in the link (MP4) provided in reference  and in the Supplementary Digital Contents.   3.1. Determining the grade of ChatGPT performance The distribution of item difficulties is shown in Figure . All 8 items fit to Rasch rather well (i.e., MNSQs of Infit and Outfit less than 1.5). The reason is that all data were simulated under the Rasch RSM model. Two easier items are #1 (What is about that pediatric patients are considered to have a fever above body temperature?) and # 8 (o you think it is possible for ChatGPT AI to replace medical doctors at this moment or in the future?). Several items are more difficult than these two; see the total summation scores in the last column of Fig. ). A higher score corresponds to an easier item. The items in Figure are all DIF-free. The reason for this is that all responses are generated using a Rasch RSM model, and the gender groups are randomly assigned to each virtual participant. Figure illustrates the scatter plot of the 2 pediatricians’ scores. The original scores evaluated by pediatricians A and B are depicted on axes y and x, respectively. There is a correlation coefficient of 0.66 with t = 2.16 and df = 6 ( P < .01). The ICC (=0.61) is identical to the 2-way ANOVA shown in Supplemental Digital Content 2, http://links.lww.com/MD/J152 , implying that the moderate reliability (within 0.5–0.74)  was observed between the 2 pediatricians in terms of assessment of consistency. The Wright map in Figure illustrates several findings. First, item difficulties are arranged from harder to easier on the right panel. Second, the middle panel displays person measures distributed from high to low abilities. Third, the left panel shows the display of person measures in groups. Fourth, the bottom panel indicates that there is no significant difference in measures between the 2 groups of males and females ( P = .853). Finally, based on the grade criteria of person measures (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits), GPT301 and GPG302, with measures of −0.83 and 0.90 logits, are classified into grade C, indicating their medium performance in answering questions about symptoms in pediatrics when compared to the normal sample generated by responses under the Rasch rating scale model. The first study goal of ChatGPT’s ability as medium performance (grade C) has been determined. 3.2. The difference in the assessment of ChatGPT between pediatricians According to Figures and , the responses of GPT301 and GPT302 are expected within the upper and lower limits (i.e., Zscore in item i = (observed − expected)/(standard deviation) of item i < 2.0). The Outfit MNSQs are smaller than 2.0, which indicates that no aberrant responses exist in items  (i.e., person responses are consistent with Rasch’s expectations). In Figures and , we observed that the abilities of GPT301 and GPT 302 are located in the middle of the person-measure distribution, with grade C. According to 2 pediatricians’ assessments, ChatGPT had an average score of −0.89 logits and 0.90 logits, with SEs of 0.37 and 0.41, respectively; see the bottom panel of Figure . A significant difference in performance assessments between pediatricians was found ( P < .05). Accordingly, this study confirms the second goal of finding differences in pediatrician bias of ChatGPT between the 2 pediatricians. 3.3. Online dashboards shown on Google Maps For readers who wish to manipulate dashboards independently, those QR codes are provided in Figures (or at links  ).   The distribution of item difficulties is shown in Figure . All 8 items fit to Rasch rather well (i.e., MNSQs of Infit and Outfit less than 1.5). The reason is that all data were simulated under the Rasch RSM model. Two easier items are #1 (What is about that pediatric patients are considered to have a fever above body temperature?) and # 8 (o you think it is possible for ChatGPT AI to replace medical doctors at this moment or in the future?). Several items are more difficult than these two; see the total summation scores in the last column of Fig. ). A higher score corresponds to an easier item. The items in Figure are all DIF-free. The reason for this is that all responses are generated using a Rasch RSM model, and the gender groups are randomly assigned to each virtual participant. Figure illustrates the scatter plot of the 2 pediatricians’ scores. The original scores evaluated by pediatricians A and B are depicted on axes y and x, respectively. There is a correlation coefficient of 0.66 with t = 2.16 and df = 6 ( P < .01). The ICC (=0.61) is identical to the 2-way ANOVA shown in Supplemental Digital Content 2, http://links.lww.com/MD/J152 , implying that the moderate reliability (within 0.5–0.74)  was observed between the 2 pediatricians in terms of assessment of consistency. The Wright map in Figure illustrates several findings. First, item difficulties are arranged from harder to easier on the right panel. Second, the middle panel displays person measures distributed from high to low abilities. Third, the left panel shows the display of person measures in groups. Fourth, the bottom panel indicates that there is no significant difference in measures between the 2 groups of males and females ( P = .853). Finally, based on the grade criteria of person measures (e.g., >3.0, >1.5, >−1.5, >−3.0, and <=−3.0 logits), GPT301 and GPG302, with measures of −0.83 and 0.90 logits, are classified into grade C, indicating their medium performance in answering questions about symptoms in pediatrics when compared to the normal sample generated by responses under the Rasch rating scale model. The first study goal of ChatGPT’s ability as medium performance (grade C) has been determined.   According to Figures and , the responses of GPT301 and GPT302 are expected within the upper and lower limits (i.e., Zscore in item i = (observed − expected)/(standard deviation) of item i < 2.0). The Outfit MNSQs are smaller than 2.0, which indicates that no aberrant responses exist in items  (i.e., person responses are consistent with Rasch’s expectations). In Figures and , we observed that the abilities of GPT301 and GPT 302 are located in the middle of the person-measure distribution, with grade C. According to 2 pediatricians’ assessments, ChatGPT had an average score of −0.89 logits and 0.90 logits, with SEs of 0.37 and 0.41, respectively; see the bottom panel of Figure . A significant difference in performance assessments between pediatricians was found ( P < .05). Accordingly, this study confirms the second goal of finding differences in pediatrician bias of ChatGPT between the 2 pediatricians.   For readers who wish to manipulate dashboards independently, those QR codes are provided in Figures (or at links  ).   4.1. Principal findings According to 2 pediatricians’ assessments, ChatGPT with a grade of C had an average score of −0.89 logits and 0.90 logits. There was a significant difference in in pediatrician bias of ChatGPT between pediatricians ( P < .05), with scores of −0.89 (SE = 0.37) and 0.90 (SE = 0.41) in a unit of log odds (logits in Rasch analysis), respectively. However, the ICC (=0.61) shows that the moderate reliability (within 0.5–0.74)  was observed between the 2 pediatricians in terms of assessment of consistency. Accordingly, 2 objectives have been achieved: ChatGPT’s ability as medium performance (grade C) and differences in the assessment of ChatGPT between pediatricians. 4.2. What this knowledge adds to what we already knew The performance of a large language model called ChatGPT was evaluated on the United States Medical Licensing Exam (USMLE), which includes 3 exams: Step 1, Step 2CK, and Step 3.  The results showed that ChatGPT performed at or near the passing threshold for all 3 exams, without any specialized training or reinforcement. In a study,  a total of 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts were analyzed by 5 clinicians. Among the top 20 suggestions based on the survey scores, 9 were generated by ChatGPT. The AI-generated suggestions were noted for providing unique perspectives and were evaluated as highly understandable and relevant. However, their usefulness was deemed moderate, with low acceptance due to concerns regarding bias, inversion, and redundancy, similar to the finding in our study. Prior research on medical question answering has focused on evaluating model performance on specific tasks.  ChatGPT has achieved accuracy on various datasets, including answering yes-or-no questions from PubMed-available abstracts and questions on the US Medical Licensing Examination.  ChatGPT has also demonstrated high accuracy rates for breast cancer screening prompts and SATA prompts.  Additionally, ChatGPT has been graded as a minor for answering prompts about Kawasaki disease.  In comparison, a study reported only 29% accuracy on medical Multiple Choice Questions (MCQs).  This study seeks to determine ChatGPT grade C for questions in the field of pediatrics, similar results to previous studies.  The potential for ChatGPT and future virtual assistants become important tools for patients and healthcare providers based on assessments ranging from basic fact-based questions to complex clinical questions in a study.  Although ChatGPT was able to provide interpretable responses that reduced the likelihood of causing alarm compared to Google’s feature snippet, there is an urgent need for regulators and healthcare professionals to be involved in developing standards for minimum quality and to educate patients about the limitations of emerging AI assistants.  To raise awareness about the paradigm shift in medical education and research, the future of medical education and research is a blessing or a blight in disguise  and needs to be given careful consideration. Gilat and Cole  have raised concerns about ChatGPT’s reliability and fairness, in line with reports in the popular press regarding misinformation issues. The authors caution that ChatGPT may not always provide accurate information, and there are fears that it could be manipulated to spread false information  or produce “deepfakes.  ” Based on the results of this study, ChatGPT is capable of providing answers to pediatric-related questions with an average level (with grade C) of accuracy and consistency across the 8 prompts provided. According to the study, ChatGPT can serve as a useful tool for healthcare professionals in diagnosing and treating pediatric patients, but caution should be taken into consideration. As prompts to ChatGPT with item 8 (Do you think it is possible for ChatGPT AI to replace medical doctors at this moment or in the future?), ChatGPT does not have such personal beliefs or opinions about the question: while ChatGPT and other AI technologies have shown promise in aiding medical decision-making, they are not capable of replacing medical doctors at this moment or in the foreseeable future. This is because the field of medicine requires a human touch that includes empathy, ethical judgment, and communication skills, which are not yet replicable by AI or ChatGPT. Additionally, healthcare providers have ethical and legal responsibilities to provide the best possible care to their patients, which cannot be replaced by a machine or, said, ChatGPT. It should be noted that the study objective was not to prove that ChatGPT suggestions are superior to human suggestions, but rather to demonstrate how ChatGPT suggestions can enhance traditional techniques for CDS maintenance and optimization. This approach aligns with the fundamental theorem of medical informatics, which emphasizes the creation of computer systems that augment human intelligence, leading to improved performance when humans and computers work together compared to humans alone.  4.3. The strengths and features of this study In this study, the capacity of ChatGPT was evaluated by 2 pediatricians for CDS in pediatrics. The study compared ChatGPT’s responses to 8 items on clinical symptoms commonly encountered by pediatricians, and 2 pediatricians assessed its answers using the Wright Map and KIDMAP to compare ChatGPT ability with other virtual participants in Rasch analysis. The study found that ChatGPT has the potential to improve the clinical workflow and the responsible use of clinical decision-making, and it demonstrated the feasibility of using ChatGPT for patients with symptoms commonly encountered in pediatrics. According to this study, ChatGPT has an average level (with grade C) of ability to answer pediatric questions, and the effectiveness of ChatGPT is determined by the judgers’ leniency and severity as pediatricians B and A with 0.90 and −0.83 logits, respectively. We suggest that the methods and visualizations used in this study can be replicated in future research uising RaschOnline.  The distinct features of this study include the following: RaschOnline,  which is based on Rasch RSM, was used to analyze the data. This allowed us to use 3 visualizations to display item features and person responses online: Wright Map with groups, DIF using forest plots, and KIDMAP. These visualizations had not been previously demonstrated in the literature and can be accessed on RaschOnline  for further details and demonstrations. ChatGPT has an average level of ability in answering pediatric questions using objective measurement of Rasch analysis. The effectiveness of ChatGPT is determined by the judges’ leniency and severity, and caution should be taken in future ChatGPT assessments. 4.4. Limitations and directions for future study This study has certain limitations that may motivate further research. The first concern is that the data were generated using Rasch simulation responses  based on the Rasch RSM model.  The real and virtual responses to the 8 pediatric items were compared. Second, RashOnline  has clearly been shown to be applicable in use  rather than traditional professional statistical software (e.g., WINSTEPS,  Quest,  ConQuest,  RUMM2030,  WINMIRA,  LPCM-Win,  and R-language Rasch software  ), and further research should be conducted to determine whether the visualizations generated using Google Maps in RaschOnline are more straightforward and easier to use for general researchers. Third, on the basis of the study sample size (n = 300 in this study), it is not possible to draw reliable and valid conclusions. For a reliable and accurate assessment, there is a need for a larger sample size in future research. Fourth, in this study, only 8 items were used. A test or assessment that contains more items will be more reliable. To assess ChatGPT ability, more items will be needed in the future. Fifth, in the case of an open-ended (OE) assessment, ChatGPT’s ability is dependent upon the judge’s leniency and severity. The results of our study revealed that the 2 pediatricians had distinctly different attitudes toward the responses provided by the ChatGPT. The conditions of leniency and severity in the assessment of ChatGPT should be stricter in the future. Sixth, in future studies, it is possible to recruit medical students in a classroom or pediatricians at an international conference to evaluate their perceptions of ChatGPT’s responses to the 8 items using Rasch analysis, in addition to the simulated data based on the Rasch model as did in this study. While this approach may involve higher costs and require more manpower compared to the current study, it holds potential for validating the results obtained. It is anticipated that the findings from such studies could yield similar results. Further validating the effectiveness of ChatGPT in the evaluated context based on their ratings on ChatGPT for acceptance, relevance, understanding, and usefulness is required in the future. Finally, while AI technologies such as ChatGPT have shown promise in aiding medical decision-making in specific areas, such as diagnosing certain conditions or interpreting medical images, they are not yet advanced enough to replace medical doctors in complex diagnoses or treatment planning. However, as technology continues to evolve, it is possible that AI could play an increasingly significant role in healthcare decision-making in the future. 4.5. Conclusions The results showed that ChatGPT had an average score of −0.89 logits and 0.90 logits with a grade of C, and there was a significant difference in performance assessments between pediatricians. This study has demonstrated the potential of ChatGPT as a valuable tool for CDS in pediatrics. The findings have shed light on ChatGPT’s performance in responding to clinical symptoms commonly encountered in pediatric practice, and its ability to provide accurate and contextually relevant information. By utilizing ChatGPT, healthcare professionals can benefit from improved clinical workflows and decision-making processes in the future. Furthermore, the use of RaschOnline is recommended for visalizations in assessments based on Rasch analysis because it has an easy copy and paste interface.   According to 2 pediatricians’ assessments, ChatGPT with a grade of C had an average score of −0.89 logits and 0.90 logits. There was a significant difference in in pediatrician bias of ChatGPT between pediatricians ( P < .05), with scores of −0.89 (SE = 0.37) and 0.90 (SE = 0.41) in a unit of log odds (logits in Rasch analysis), respectively. However, the ICC (=0.61) shows that the moderate reliability (within 0.5–0.74)  was observed between the 2 pediatricians in terms of assessment of consistency. Accordingly, 2 objectives have been achieved: ChatGPT’s ability as medium performance (grade C) and differences in the assessment of ChatGPT between pediatricians.   The performance of a large language model called ChatGPT was evaluated on the United States Medical Licensing Exam (USMLE), which includes 3 exams: Step 1, Step 2CK, and Step 3.  The results showed that ChatGPT performed at or near the passing threshold for all 3 exams, without any specialized training or reinforcement. In a study,  a total of 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts were analyzed by 5 clinicians. Among the top 20 suggestions based on the survey scores, 9 were generated by ChatGPT. The AI-generated suggestions were noted for providing unique perspectives and were evaluated as highly understandable and relevant. However, their usefulness was deemed moderate, with low acceptance due to concerns regarding bias, inversion, and redundancy, similar to the finding in our study. Prior research on medical question answering has focused on evaluating model performance on specific tasks.  ChatGPT has achieved accuracy on various datasets, including answering yes-or-no questions from PubMed-available abstracts and questions on the US Medical Licensing Examination.  ChatGPT has also demonstrated high accuracy rates for breast cancer screening prompts and SATA prompts.  Additionally, ChatGPT has been graded as a minor for answering prompts about Kawasaki disease.  In comparison, a study reported only 29% accuracy on medical Multiple Choice Questions (MCQs).  This study seeks to determine ChatGPT grade C for questions in the field of pediatrics, similar results to previous studies.  The potential for ChatGPT and future virtual assistants become important tools for patients and healthcare providers based on assessments ranging from basic fact-based questions to complex clinical questions in a study.  Although ChatGPT was able to provide interpretable responses that reduced the likelihood of causing alarm compared to Google’s feature snippet, there is an urgent need for regulators and healthcare professionals to be involved in developing standards for minimum quality and to educate patients about the limitations of emerging AI assistants.  To raise awareness about the paradigm shift in medical education and research, the future of medical education and research is a blessing or a blight in disguise  and needs to be given careful consideration. Gilat and Cole  have raised concerns about ChatGPT’s reliability and fairness, in line with reports in the popular press regarding misinformation issues. The authors caution that ChatGPT may not always provide accurate information, and there are fears that it could be manipulated to spread false information  or produce “deepfakes.  ” Based on the results of this study, ChatGPT is capable of providing answers to pediatric-related questions with an average level (with grade C) of accuracy and consistency across the 8 prompts provided. According to the study, ChatGPT can serve as a useful tool for healthcare professionals in diagnosing and treating pediatric patients, but caution should be taken into consideration. As prompts to ChatGPT with item 8 (Do you think it is possible for ChatGPT AI to replace medical doctors at this moment or in the future?), ChatGPT does not have such personal beliefs or opinions about the question: while ChatGPT and other AI technologies have shown promise in aiding medical decision-making, they are not capable of replacing medical doctors at this moment or in the foreseeable future. This is because the field of medicine requires a human touch that includes empathy, ethical judgment, and communication skills, which are not yet replicable by AI or ChatGPT. Additionally, healthcare providers have ethical and legal responsibilities to provide the best possible care to their patients, which cannot be replaced by a machine or, said, ChatGPT. It should be noted that the study objective was not to prove that ChatGPT suggestions are superior to human suggestions, but rather to demonstrate how ChatGPT suggestions can enhance traditional techniques for CDS maintenance and optimization. This approach aligns with the fundamental theorem of medical informatics, which emphasizes the creation of computer systems that augment human intelligence, leading to improved performance when humans and computers work together compared to humans alone.    In this study, the capacity of ChatGPT was evaluated by 2 pediatricians for CDS in pediatrics. The study compared ChatGPT’s responses to 8 items on clinical symptoms commonly encountered by pediatricians, and 2 pediatricians assessed its answers using the Wright Map and KIDMAP to compare ChatGPT ability with other virtual participants in Rasch analysis. The study found that ChatGPT has the potential to improve the clinical workflow and the responsible use of clinical decision-making, and it demonstrated the feasibility of using ChatGPT for patients with symptoms commonly encountered in pediatrics. According to this study, ChatGPT has an average level (with grade C) of ability to answer pediatric questions, and the effectiveness of ChatGPT is determined by the judgers’ leniency and severity as pediatricians B and A with 0.90 and −0.83 logits, respectively. We suggest that the methods and visualizations used in this study can be replicated in future research uising RaschOnline.  The distinct features of this study include the following: RaschOnline,  which is based on Rasch RSM, was used to analyze the data. This allowed us to use 3 visualizations to display item features and person responses online: Wright Map with groups, DIF using forest plots, and KIDMAP. These visualizations had not been previously demonstrated in the literature and can be accessed on RaschOnline  for further details and demonstrations. ChatGPT has an average level of ability in answering pediatric questions using objective measurement of Rasch analysis. The effectiveness of ChatGPT is determined by the judges’ leniency and severity, and caution should be taken in future ChatGPT assessments.   This study has certain limitations that may motivate further research. The first concern is that the data were generated using Rasch simulation responses  based on the Rasch RSM model.  The real and virtual responses to the 8 pediatric items were compared. Second, RashOnline  has clearly been shown to be applicable in use  rather than traditional professional statistical software (e.g., WINSTEPS,  Quest,  ConQuest,  RUMM2030,  WINMIRA,  LPCM-Win,  and R-language Rasch software  ), and further research should be conducted to determine whether the visualizations generated using Google Maps in RaschOnline are more straightforward and easier to use for general researchers. Third, on the basis of the study sample size (n = 300 in this study), it is not possible to draw reliable and valid conclusions. For a reliable and accurate assessment, there is a need for a larger sample size in future research. Fourth, in this study, only 8 items were used. A test or assessment that contains more items will be more reliable. To assess ChatGPT ability, more items will be needed in the future. Fifth, in the case of an open-ended (OE) assessment, ChatGPT’s ability is dependent upon the judge’s leniency and severity. The results of our study revealed that the 2 pediatricians had distinctly different attitudes toward the responses provided by the ChatGPT. The conditions of leniency and severity in the assessment of ChatGPT should be stricter in the future. Sixth, in future studies, it is possible to recruit medical students in a classroom or pediatricians at an international conference to evaluate their perceptions of ChatGPT’s responses to the 8 items using Rasch analysis, in addition to the simulated data based on the Rasch model as did in this study. While this approach may involve higher costs and require more manpower compared to the current study, it holds potential for validating the results obtained. It is anticipated that the findings from such studies could yield similar results. Further validating the effectiveness of ChatGPT in the evaluated context based on their ratings on ChatGPT for acceptance, relevance, understanding, and usefulness is required in the future. Finally, while AI technologies such as ChatGPT have shown promise in aiding medical decision-making in specific areas, such as diagnosing certain conditions or interpreting medical images, they are not yet advanced enough to replace medical doctors in complex diagnoses or treatment planning. However, as technology continues to evolve, it is possible that AI could play an increasingly significant role in healthcare decision-making in the future.   The results showed that ChatGPT had an average score of −0.89 logits and 0.90 logits with a grade of C, and there was a significant difference in performance assessments between pediatricians. This study has demonstrated the potential of ChatGPT as a valuable tool for CDS in pediatrics. The findings have shed light on ChatGPT’s performance in responding to clinical symptoms commonly encountered in pediatric practice, and its ability to provide accurate and contextually relevant information. By utilizing ChatGPT, healthcare professionals can benefit from improved clinical workflows and decision-making processes in the future. Furthermore, the use of RaschOnline is recommended for visalizations in assessments based on Rasch analysis because it has an easy copy and paste interface.   We thank AJE (American Journal Experts at https://www.aje.com/ ) for the English language review of this manuscript. All authors declare no conflicts of interest.   Conceptualization: Hsu-Ju Kao, Wen-Chung Wang, Julie Chi Chow. Investigation: Willy Chou. Methodology: Tsair-wei Chien.      