Effects of Implementing Artificial Intelligence-Based Computer-Aided Detection for Chest Radiographs in Daily Practice on the Rate of Referral to Chest Computed Tomography in Pulmonology Outpatient Clinic
In recent years, artificial intelligence (AI) has achieved remarkable success in the evaluation of chest radiographs (CRs) . Owing to its excellent performance in experimental studies , AI is increasingly used in daily practice . Among the various applications of AIs, detecting various abnormalities to assist the interpretation of radiologists or clinicians (AI-based computer-aided detection, AI-CAD) is the most widely used application . Recently, several studies reported improved accuracy of CR interpretation following AI-CAD implementation in clinical practice . The ultimate goal of using AI-CAD is to improve the quality of patient management and patient outcomes . However, because clinical decision-making and patient outcomes are influenced by factors other than CR interpretation, the clinical impact of CR interpretation using AI-CAD beyond interpretation accuracy has rarely been investigated . Since CR is the initial screening examination for the presence of various respiratory diseases , the initial clinical decision influenced by the CR interpretation would be the need for a referral to chest computed tomography (CT) to confirm and characterize the CR findings. Hence, the influence of AI-CAD implementation on CR interpretation in chest CT referral could indicate whether AI-CAD implementation influences patient management. Therefore, we aimed to investigate the influence of the clinical implementation of AI-CAD for CR interpretation in daily practice on the frequency of referral to chest CT in patients visiting pulmonology outpatient clinics. 

Patients and CRs This single-center retrospective study was conducted at the Seoul National University Hospital in Republic of Korea after obtaining approval from the Institutional Review Board (approval number: D-2107-122-1236), with a waiver for obtaining informed consent from the patients. We included patients who were referred for CRs from outpatient pulmonology clinics between January 2019 and December 2020. Each CR was included in the study of patients who underwent multiple CRs during the study period . All CRs were obtained in the erect position using a posteroanterior projection with various scanners. AI-CAD and Clinical Implementation A commercialized, regulatory-approved AI-CAD system (Lunit INSIGHT CXR, version 2.0.2.0, Lunit Inc.) was used in clinical practice on January 1, 2020. The AI-CAD was designed to evaluate the presence of pulmonary nodules, pulmonary consolidation, and pneumothorax in a frontal CR and to display the location of abnormalities with a heat map when the predicted probability for the presence of abnormality was ≥ 15%  . Pulmonologists were informed that they could request CRs, including AI-CAD analyses, from January 1, 2020, using a dedicated examination protocol. The referring pulmonologist (fellowship trainee or attending pulmonologist with 2–28 years of experience) decided whether to request CR with or without AI-CAD analysis. When CRs with AI-CAD analyses were requested, AI-CAD analyses were automatically processed immediately after CR acquisition, and the results of AI-CAD analyses were posted on the institutional picture archiving and communication system along with the CR images. All CRs were interpreted by attending thoracic radiologists (1–20 years of experience), fellowship trainees in thoracic radiology, or residency trainees (1st–4th year in the training program) supervised by the attending thoracic radiologists. Radiologists interpreted the CRs after reviewing the AI-CAD results for examinations, including AI-CAD analyses. Formal reports were provided regardless of the availability of AI-CAD analysis results. Pulmonologists could also evaluate the CRs and AI-CAD analysis results immediately after CR acquisition, regardless of the presence of formal reports. Definitions of the Outcomes The primary outcome of our study was the CT referral rate. CT referral was defined as a chest CT request by the pulmonology department after CR acquisition, followed by CT within 30 days of CR acquisition . Exclusion criteria for the CT referral include CT requests from departments other than pulmonology and those before CR acquisition since these requests might not be related to the CR findings. Moreover, CT scans obtained 30 days after CR acquisition were also excluded from the referral because these CT requests may not have been used to evaluate the findings identified on the CRs. As secondary outcomes, we evaluated the frequency of positive and negative CT results followed by CT referrals (positive and negative CT referral rates, hereafter). To classify positive and negative CT results, a junior thoracic radiologist (W.H., one-year experience as an attending thoracic radiologist) reviewed the reports written by the attending thoracic radiologists. CT reports indicating findings requiring further evaluation or treatment were classified as positive (Supplementary Material). The positive and negative CT referral rates were defined as the referral frequency for CT with subsequent positive and negative CT results, respectively. We also investigated the referral frequency to CT before documentation of the formal report by the radiologist (referral to CT without a CR report, hereafter) as a secondary outcome. Multivariable Analyses To evaluate the effect of AI-CAD implementation after adjusting for potential confounding variables, we conducted multivariable logistic regression analyses to investigate the effect of AI-CAD application on study outcomes. CRs obtained before and after AI-CAD implementation were included in the multivariable analyses after excluding CRs requested by pulmonologists who had never requested AI-CAD analyses or those who had requested CRs only before- or after AI-CAD implementation . The following independent variables were included: 1) use of AI-CAD analyses (i.e., the main factor), 2) patient’s sex, 3) patient’s age, 4) CR acquisition before and after AI-CAD implementation, 5) referring pulmonologist, 6) nature of the CR examination (baseline versus follow-up CR), and 7) availability of radiology reports at the time of the pulmonology visit after CR. Subgroup Analysis 1: Experience of Referring Pulmonologist To investigate whether the effect of AI-CAD analysis differed according to the experience of the pulmonologists, we conducted multivariable analyses for study outcomes in subgroups of CRs requested by senior pulmonologists (attending pulmonologists with ≥ 10 years of experience) and junior pulmonologists (fellowship trainees and attending pulmonologists with < 10 years of experience). Subgroup Analysis 2: AI-CAD Effect Over Time after Implementation In order to investigate the change in the effect of AI-CAD analysis over time after its implementation, we conducted multivariable logistic regression analyses for the study outcomes in a subgroup of CRs obtained after AI-CAD implementation. The CRs were classified into those obtained in the early (January to June 2020) and later (July to December 2020) 6-month periods after the AI-CAD implementation. The following independent variables were included: 1) use of AI-CAD analyses; 2) sex; 3) age; 4) CRs obtained in the early versus later 6 months after AI-CAD implementation; 5) referring pulmonologist; 6) baseline versus follow-up CR; and 7) availability of radiology reports at the time of pulmonology visit after CR. We added an interaction term between the request for AI-CAD analyses and the time of CR acquisition in the regression model to evaluate whether the effect of AI-CAD analyses was different between CRs obtained in the early versus later 6 months after AI-CAD implementation. Subgroup Analysis 3: AI-CAD Results We conducted subgroup analyses of the CRs using AI-CAD to evaluate the effect of the AI-CAD analysis results on the study outcomes. The AI-CAD results were classified as positive or negative based on the abnormalities identified by AI-CAD. We conducted multivariable analyses for the study outcomes using the following independent variables: 1) positive versus negative AI-CAD results, 2) patient sex, 3) patient age, 4) referring pulmonologist, 5) baseline versus follow-up CR, and 6) availability of radiology reports at the time of pulmonology visit after CR. Statistical Analysis The crude study outcomes were compared between the CRs obtained before and after AI-CAD implementation and were analyzed using the Chi-square test. Subsequently, we conducted multivariable logistic regression analyses to investigate the association between AI-CAD use and the study outcomes. Generalized estimation equations were adopted to adjust for the effects of multiple CRs obtained from a single patient. All statistical analyses were conducted using R software (version 3.6.3, R Project for Statistical Computing), and P -value <0.05 were considered statistically significant. 

This single-center retrospective study was conducted at the Seoul National University Hospital in Republic of Korea after obtaining approval from the Institutional Review Board (approval number: D-2107-122-1236), with a waiver for obtaining informed consent from the patients. We included patients who were referred for CRs from outpatient pulmonology clinics between January 2019 and December 2020. Each CR was included in the study of patients who underwent multiple CRs during the study period . All CRs were obtained in the erect position using a posteroanterior projection with various scanners. 

A commercialized, regulatory-approved AI-CAD system (Lunit INSIGHT CXR, version 2.0.2.0, Lunit Inc.) was used in clinical practice on January 1, 2020. The AI-CAD was designed to evaluate the presence of pulmonary nodules, pulmonary consolidation, and pneumothorax in a frontal CR and to display the location of abnormalities with a heat map when the predicted probability for the presence of abnormality was ≥ 15%  . Pulmonologists were informed that they could request CRs, including AI-CAD analyses, from January 1, 2020, using a dedicated examination protocol. The referring pulmonologist (fellowship trainee or attending pulmonologist with 2–28 years of experience) decided whether to request CR with or without AI-CAD analysis. When CRs with AI-CAD analyses were requested, AI-CAD analyses were automatically processed immediately after CR acquisition, and the results of AI-CAD analyses were posted on the institutional picture archiving and communication system along with the CR images. All CRs were interpreted by attending thoracic radiologists (1–20 years of experience), fellowship trainees in thoracic radiology, or residency trainees (1st–4th year in the training program) supervised by the attending thoracic radiologists. Radiologists interpreted the CRs after reviewing the AI-CAD results for examinations, including AI-CAD analyses. Formal reports were provided regardless of the availability of AI-CAD analysis results. Pulmonologists could also evaluate the CRs and AI-CAD analysis results immediately after CR acquisition, regardless of the presence of formal reports. 

The primary outcome of our study was the CT referral rate. CT referral was defined as a chest CT request by the pulmonology department after CR acquisition, followed by CT within 30 days of CR acquisition . Exclusion criteria for the CT referral include CT requests from departments other than pulmonology and those before CR acquisition since these requests might not be related to the CR findings. Moreover, CT scans obtained 30 days after CR acquisition were also excluded from the referral because these CT requests may not have been used to evaluate the findings identified on the CRs. As secondary outcomes, we evaluated the frequency of positive and negative CT results followed by CT referrals (positive and negative CT referral rates, hereafter). To classify positive and negative CT results, a junior thoracic radiologist (W.H., one-year experience as an attending thoracic radiologist) reviewed the reports written by the attending thoracic radiologists. CT reports indicating findings requiring further evaluation or treatment were classified as positive (Supplementary Material). The positive and negative CT referral rates were defined as the referral frequency for CT with subsequent positive and negative CT results, respectively. We also investigated the referral frequency to CT before documentation of the formal report by the radiologist (referral to CT without a CR report, hereafter) as a secondary outcome. 

To evaluate the effect of AI-CAD implementation after adjusting for potential confounding variables, we conducted multivariable logistic regression analyses to investigate the effect of AI-CAD application on study outcomes. CRs obtained before and after AI-CAD implementation were included in the multivariable analyses after excluding CRs requested by pulmonologists who had never requested AI-CAD analyses or those who had requested CRs only before- or after AI-CAD implementation . The following independent variables were included: 1) use of AI-CAD analyses (i.e., the main factor), 2) patient’s sex, 3) patient’s age, 4) CR acquisition before and after AI-CAD implementation, 5) referring pulmonologist, 6) nature of the CR examination (baseline versus follow-up CR), and 7) availability of radiology reports at the time of the pulmonology visit after CR. 

To investigate whether the effect of AI-CAD analysis differed according to the experience of the pulmonologists, we conducted multivariable analyses for study outcomes in subgroups of CRs requested by senior pulmonologists (attending pulmonologists with ≥ 10 years of experience) and junior pulmonologists (fellowship trainees and attending pulmonologists with < 10 years of experience). 

In order to investigate the change in the effect of AI-CAD analysis over time after its implementation, we conducted multivariable logistic regression analyses for the study outcomes in a subgroup of CRs obtained after AI-CAD implementation. The CRs were classified into those obtained in the early (January to June 2020) and later (July to December 2020) 6-month periods after the AI-CAD implementation. The following independent variables were included: 1) use of AI-CAD analyses; 2) sex; 3) age; 4) CRs obtained in the early versus later 6 months after AI-CAD implementation; 5) referring pulmonologist; 6) baseline versus follow-up CR; and 7) availability of radiology reports at the time of pulmonology visit after CR. We added an interaction term between the request for AI-CAD analyses and the time of CR acquisition in the regression model to evaluate whether the effect of AI-CAD analyses was different between CRs obtained in the early versus later 6 months after AI-CAD implementation. 

We conducted subgroup analyses of the CRs using AI-CAD to evaluate the effect of the AI-CAD analysis results on the study outcomes. The AI-CAD results were classified as positive or negative based on the abnormalities identified by AI-CAD. We conducted multivariable analyses for the study outcomes using the following independent variables: 1) positive versus negative AI-CAD results, 2) patient sex, 3) patient age, 4) referring pulmonologist, 5) baseline versus follow-up CR, and 6) availability of radiology reports at the time of pulmonology visit after CR. 

The crude study outcomes were compared between the CRs obtained before and after AI-CAD implementation and were analyzed using the Chi-square test. Subsequently, we conducted multivariable logistic regression analyses to investigate the association between AI-CAD use and the study outcomes. Generalized estimation equations were adopted to adjust for the effects of multiple CRs obtained from a single patient. All statistical analyses were conducted using R software (version 3.6.3, R Project for Statistical Computing), and P -value <0.05 were considered statistically significant. 

Patient Characteristics Among the 54434 CRs from 20060 patients (mean age, 67 years ± 13 [standard deviation]; 9910 males), 28546 CRs from 14565 patients (mean age 67 years ± 13; 7130 males) were obtained before the AI-CAD implementation, while 25888 CRs from 12929 patients (mean age 67 years ± 13; 6435 males) were obtained after the AI-CAD implementation . A total of 7434 patients underwent CRs both before and after AI-CAD implementation. Among the CRs obtained after AI-CAD implementation, 3421 CRs (13.2%) were requested for AI-CAD analyses. Baseline CRs (10.9% vs 9.9%; P < 0.001) and positive CR reports (reports indicating the presence of any findings requiring further evaluation or treatment; 9.9% vs 8.7%; P < 0.001) were less frequent after AI-CAD implementation . shows the clinical diagnoses of the patients. Comparison of the Study Outcomes before and after AI-CAD Implementation Among the 4257 patients who underwent chest CTs within 30 days after CR acquisition, CTs requested from departments other than pulmonology (1277) and those requested before CR acquisition (889) were excluded from the CT referral. CT referrals were observed in 2091 (3.8%) patients with CRs. Among them, 785 (37.5%) and 1306 (62.5%) CT scans exhibited positive and negative findings, respectively . Comparing the crude study outcomes between CRs obtained before and after AI-CAD implementation, the CT referral rate (4.6% vs. 3.2%; P < 0.001), positive (1.7% vs. 1.2%; P < 0.001) and negative (2.9% vs. 2.0%; P < 0.001) CT referral rates, and the frequency of referral to CT without CR report (2.2% vs. 1.8%; P < 0.001) were higher in CRs obtained after AI-CAD implementation . Association between the Use of AI-CAD and Study Outcomes For multivariable logistic regression analyses, 21055 CRs in 9191 patients were included . In the logistic regression analyses to identify CT referral predictors, the use of AI-CAD was independently associated with increased CT referral (odds ratio [OR], 1.33; 95% confidence interval [CI], 1.08–1.65; P = 0.008) . The AI-CAD analysis was independently associated with increased negative CT referral (OR, 1.46; 95% CI, 1.12–1.91; P = 0.005) but not an independent factor for the positive CT referral (OR, 1.08; 95% CI, 0.77–1.54; P = 0.647) nor for the referral to CT without CR report (OR, 1.10; 95% CI, 0.82–1.47; P = 0.542) . Subgroup Analysis 1: Experience of Referring Pulmonologist In the subgroup of CRs requested by senior pulmonologists, the crude CT referral rate (2.6% [591/22772] vs. 3.9% [694/17933]; P < 0.001) and other secondary study outcomes were higher in the CRs obtained after AI-CAD implementation. Meanwhile, in the subgroup of CRs requested by junior pulmonologists, the crude CT referral rate (5.5% [318/5774] vs. 6.1% [488/7955]; P = 0.123) and secondary study outcomes did not significantly differ between CRs obtained before and after AI-CAD implementation . Similarly, in the multivariable analyses, the use of AI-CAD analysis was independently associated with increased CT referral (OR, 1.45; 95% CI, 1.07–1.96; P = 0.004) and increased negative CT referral (OR, 1.72; 95% CI, 1.16–2.54; P = 0.006) in the subgroup of CRs requested by senior pulmonologists. Meanwhile, using AI-CAD analysis was not an independent factor for any of the study outcomes in CRs requested by the junior pulmonologists . Subgroup Analysis 2: AI-CAD Effect Over Time after Implementation The crude CT referral rate was higher in CRs with AI-CAD analyses than in those without AI-CAD analyses in both the CRs obtained in the early (7.9% [102/1287] vs. 4.4% [500/11377]; P < 0.001) and later (9.5% [203/2134] vs. 3.4% [372/11090]; P < 0.001) 6 months after AI-CAD implementation . In the multivariable analysis, we found a significant interaction between using AI-CAD and the time of CR acquisition to predict CT referral (OR, 1.53; 95% CI, 1.03–2.28; P = 0.037). In other words, the effect of AI-CAD analyses on CT referral was more substantial in the CRs obtained in the later six months after AI-CAD implementation . Subgroup Analysis 3: AI-CAD Results In the subgroup of CRs with AI-CAD analysis (3373 CRs from 2518 patients), positive AI-CAD results were observed in 69.2% of patients (2334/3373). The CT referral rate was 8.9% (301/3373). The positive and negative CT referral rates were 2.7% (92/3373) and 6.2% (209/3373), respectively. We did not find evidence of a difference in the CT referral rate between CRs with positive (8.4%) and negative (10.2%) AI-CAD results ( P = 0.084). In the logistic regression analyses, a positive AI-CAD result was not an independent factor for CT referral (OR, 1.17; 95% CI, 0.88–1.55; P = 0.288) . 

Among the 54434 CRs from 20060 patients (mean age, 67 years ± 13 [standard deviation]; 9910 males), 28546 CRs from 14565 patients (mean age 67 years ± 13; 7130 males) were obtained before the AI-CAD implementation, while 25888 CRs from 12929 patients (mean age 67 years ± 13; 6435 males) were obtained after the AI-CAD implementation . A total of 7434 patients underwent CRs both before and after AI-CAD implementation. Among the CRs obtained after AI-CAD implementation, 3421 CRs (13.2%) were requested for AI-CAD analyses. Baseline CRs (10.9% vs 9.9%; P < 0.001) and positive CR reports (reports indicating the presence of any findings requiring further evaluation or treatment; 9.9% vs 8.7%; P < 0.001) were less frequent after AI-CAD implementation . shows the clinical diagnoses of the patients. 

Among the 4257 patients who underwent chest CTs within 30 days after CR acquisition, CTs requested from departments other than pulmonology (1277) and those requested before CR acquisition (889) were excluded from the CT referral. CT referrals were observed in 2091 (3.8%) patients with CRs. Among them, 785 (37.5%) and 1306 (62.5%) CT scans exhibited positive and negative findings, respectively . Comparing the crude study outcomes between CRs obtained before and after AI-CAD implementation, the CT referral rate (4.6% vs. 3.2%; P < 0.001), positive (1.7% vs. 1.2%; P < 0.001) and negative (2.9% vs. 2.0%; P < 0.001) CT referral rates, and the frequency of referral to CT without CR report (2.2% vs. 1.8%; P < 0.001) were higher in CRs obtained after AI-CAD implementation . 

For multivariable logistic regression analyses, 21055 CRs in 9191 patients were included . In the logistic regression analyses to identify CT referral predictors, the use of AI-CAD was independently associated with increased CT referral (odds ratio [OR], 1.33; 95% confidence interval [CI], 1.08–1.65; P = 0.008) . The AI-CAD analysis was independently associated with increased negative CT referral (OR, 1.46; 95% CI, 1.12–1.91; P = 0.005) but not an independent factor for the positive CT referral (OR, 1.08; 95% CI, 0.77–1.54; P = 0.647) nor for the referral to CT without CR report (OR, 1.10; 95% CI, 0.82–1.47; P = 0.542) . 

In the subgroup of CRs requested by senior pulmonologists, the crude CT referral rate (2.6% [591/22772] vs. 3.9% [694/17933]; P < 0.001) and other secondary study outcomes were higher in the CRs obtained after AI-CAD implementation. Meanwhile, in the subgroup of CRs requested by junior pulmonologists, the crude CT referral rate (5.5% [318/5774] vs. 6.1% [488/7955]; P = 0.123) and secondary study outcomes did not significantly differ between CRs obtained before and after AI-CAD implementation . Similarly, in the multivariable analyses, the use of AI-CAD analysis was independently associated with increased CT referral (OR, 1.45; 95% CI, 1.07–1.96; P = 0.004) and increased negative CT referral (OR, 1.72; 95% CI, 1.16–2.54; P = 0.006) in the subgroup of CRs requested by senior pulmonologists. Meanwhile, using AI-CAD analysis was not an independent factor for any of the study outcomes in CRs requested by the junior pulmonologists . 

The crude CT referral rate was higher in CRs with AI-CAD analyses than in those without AI-CAD analyses in both the CRs obtained in the early (7.9% [102/1287] vs. 4.4% [500/11377]; P < 0.001) and later (9.5% [203/2134] vs. 3.4% [372/11090]; P < 0.001) 6 months after AI-CAD implementation . In the multivariable analysis, we found a significant interaction between using AI-CAD and the time of CR acquisition to predict CT referral (OR, 1.53; 95% CI, 1.03–2.28; P = 0.037). In other words, the effect of AI-CAD analyses on CT referral was more substantial in the CRs obtained in the later six months after AI-CAD implementation . 

In the subgroup of CRs with AI-CAD analysis (3373 CRs from 2518 patients), positive AI-CAD results were observed in 69.2% of patients (2334/3373). The CT referral rate was 8.9% (301/3373). The positive and negative CT referral rates were 2.7% (92/3373) and 6.2% (209/3373), respectively. We did not find evidence of a difference in the CT referral rate between CRs with positive (8.4%) and negative (10.2%) AI-CAD results ( P = 0.084). In the logistic regression analyses, a positive AI-CAD result was not an independent factor for CT referral (OR, 1.17; 95% CI, 0.88–1.55; P = 0.288) . 

The impact of the clinical implementation of AI-CAD for interpreting CRs in clinical decision-making has not been thoroughly investigated. In this study, we investigated the influence of implementing AI-CAD for CR interpretation on the CT referral rate in pulmonology patients. The CT referral rate increased after AI-CAD implementation (3.2% to 4.6%; before versus after AI-CAD implementation), and using AI-CAD was an independent factor for CT referral (OR, 1.33; P = 0.008). Additionally, referral to negative chest CT also increased after the AI-CAD implementation (2.0% to 2.9 %; before versus after the AI-CAD implementation) and was independently associated with using AI-CAD (OR, 1.46; P = 0.005). Although previous studies have reported improved diagnostic accuracy of CR interpretation following AI-CAD implementation , its impact on patient management has rarely been reported. For example, Hwang et al.  reported that despite the increased detection yield of new metastases on CR after AI-CAD implementation, most metastases had already been identified on CT. Hong et al.  reported that although the sensitivity for post-biopsy pneumothorax increased after AI-CAD implementation, the drainage catheter insertion time did not change. Compared to previous studies investigating AI-CAD for CRs, the most important strength of our study is that we investigated the impact of AI-CAD implementation for CR interpretation on clinical decision-making. The increased CT referral rate following AI-CAD implementation observed in our study provides substantial evidence that AI-CAD for CR interpretation can influence patient management. Increased detection of subtle but relevant abnormalities could be the primary goal of the clinical implementation of AI-CAD . However, using AI-CAD could be a double-edged sword because it could result in false-positive detection of abnormalities, leading to unnecessary additional examinations or treatments. In this study, AI-CAD was independently associated with CT referral and negative CT referral. Although we could not confirm the reasons for CT referrals, increased identification of false-positive findings may be the leading cause of increased overall CT referrals and referrals to negative CT findings. We cannot confirm that a negative CT referral was clinically unnecessary because the pulmonologist may have decided to obtain chest CT images based on reasonable clinical backgrounds, and confirmation of the absence of clinically significant abnormalities would provide valuable information for patient management. However, increased referrals to negative CT should be considered as an important adverse effect of AI-CAD implementation. If AI-CAD contributed to the detection of clinically significant abnormalities, it might have led to an increased frequency of confirmation of abnormalities on CT scans. However, contrary to expectations, using AI-CAD was not associated with a referral for positive CT. Our results seem contradictory considering the results of previous studies that reported improved detection performance with AI-CAD . Although the reason for this observation is unclear, it could be that chest CTs were requested for most patients with significant respiratory diseases, regardless of the AI-CAD analyses. Since the decision for CT referral could be influenced by various factors, pulmonologists may have requested chest CT scans for patients with a high probability of significant respiratory diseases regardless of CR findings. Timely interpretation of the CR by a radiologist may not always be possible in clinical practice. AI-CAD may aid clinicians in CR interpretation and subsequent clinical decision-making in such situations. In our study, radiology reports were unavailable until the next pulmonology visit in 38.7% of CRs, and 51.5% of CT referrals were made without a radiology report. However, because these clinicians are relatively less familiar with AI-CAD than radiologists, their clinical decisions could be more susceptible to AI-CAD results . The influence of false-positive AI-CAD results on the pulmonologists’ interpretation of CRs may contribute to the increased negative CT referral results. Interestingly, increased CT referral and negative CT referral rates were observed in CRs requested by senior pulmonologists but not those requested by junior pulmonologists, which may be related to the lower baseline CT referral rate among senior pulmonologists (2.6% vs. 5.5%; before AI-CAD implementation group). The additional detection of abnormalities by AI-CAD may have a greater influence on senior pulmonologists, who tend to request chest CT more judiciously. This suggests that familiarity with the CR interpretation does not correspond to an appropriate interpretation of the AI-CAD results. Therefore, appropriate education and guidance from clinicians regarding AI-CAD use may be necessary. In addition to the intrinsic shortcomings of a single-center retrospective study, our study has several limitations. First, AI-CAD has not been universally applied after its clinical implementation, and we conducted multivariable logistic regression analyses to minimize bias due to the direct comparison between the two groups. Second, the authors arbitrarily defined CT referral, and it was difficult to confirm whether the CT referral was due to an abnormality identified in the CRs. Third, the CT findings were classified as positive or negative results based on the evaluation of a single thoracic radiologist. Finally, the influence of AI-CAD on patient management, beyond the decision to obtain CT images, was not evaluated. In conclusion, implementing AI-CAD for CR increased overall CT referrals in patients admitted to the pulmonology clinic. Referrals to chest CT scans with negative results (i.e., without significant abnormalities) also increased after AI-CAD implementation. Further investigation is warranted to evaluate whether AI-CAD implementation influences patient management and outcomes. 

