Three-dimensional markerless surface topography approach with convolutional neural networks for adolescent idiopathic scoliosis screening
Adolescent idiopathic scoliosis (AIS) is a three-dimensional spine deformity marked by a lateral deviation and often accompanied by vertebral rotation . The prevalence of AIS can be up to 5% and has a higher incidence in girls . AIS is diagnosed through radiographic examination, which involves taking a standing posterior-anterior x-ray to assess the curvature using the Cobb angle . AIS is confirmed when the measured Cobb angle is at least 10° . Observation is recommended for curves less than 25°, and bracing is prescribed for curves between 25° and 45° . Severe curves greater than 45° may require corrective spinal surgery . Moreover, the likelihood of curve progression increases during puberty and the rapid growth phase , . Screening can play a crucial role in the early detection of scoliosis among adolescents, often performed during routine school screenings or in a clinical setting , . Detecting scoliosis early allows for intervention to prevent the spinal curve from worsening , . However, controversies about screenings in schools exist. Clinical screening methods have high false positive rates, leading to referrals for x-ray imaging that exposes children to unnecessary radiation , – . Further screening can involve stress from examinations and can lead to psychological side effects . However, the recommendations in favour of screening rely on evidence of moderate quality, whereas those opposing screening are founded on low quality evidence . The effect of discontinued screening programs has been evaluated in Canada, resulting in 32% of confirmed cases of AIS being considered late referrals requiring immediate treatment by bracing or surgery . The Adam’s forward bend test is a standard and simple test where the examiner observes the back from the rear while the participant is bent forward at the waist, looking for any asymmetry, humps, or other irregularities in the contour of the spine, which could indicate a higher likelihood of the presence of scoliosis . The Adam’s forward bend test is subjective and not recommended on its own because the results often depend on the experience of the examiner. The test has high sensitivity, ranging from 92 to 84.4%, but shows a high risk of false positives and a low specificity of 68% , . The scoliometer is a handheld instrument used in the assessment of scoliosis to measure the degree of axial trunk rotation of the hump revealed by the Adam’s test . The scoliometer measures the angle of trunk rotation (ATR) and adds an objective measure to the Adams test . When a cutoff ATR angle of 5° is used, it has a sensitivity of around 100% and a specificity of 47% , . Specificity increases to 86% while sensitivity decreases to 83% with a cutoff ATR angle of 7° , . The positive predictive value can range from 28 to 57% when a cutoff angle of 5° is used , . Likewise, the positive predictive value when using a cutoff ATR of 7° can range from 40 to 71% , . Additionally, the use of multiple tests increases sensitivity and specificity , , . Surface topography (ST) can be an alternative screening approach. The markerless ST asymmetry analysis method reflects the 3D geometry of the torso around the best plane of symmetry to identify the external areas of asymmetry in a deviation color map (DCM) image – . ST has been previously shown to have a medium to strong agreement with estimating curve location and predicting the curve severity , . Markerless ST can be used as an innovative and non-invasive approach for scoliosis screening, providing a three-dimensional assessment of the back’s surface without markers or radiation that may improve traditional screening approaches. However, differences in trunk asymmetries between individuals with scoliosis and typically developing individuals are not fully understood. Convolutional neural network (CNN) is a supervised machine learning algorithm commonly applied to computer vision tasks such as image classification, localization, detection, and segmentation , . The neural network algorithm learns and captures features from inputs while preserving spatial relationships . CNN processes grid-like topology data, such as image data, as a 2D or 3D grid of pixels . In research on scoliosis, machine learning algorithms, specifically CNNs, have been implemented in radiological images to detect spinal curvature and to automatically measure the Cobb angle , . CNN algorithms have also been implemented to develop scoliosis screening methods relying on various inputs from surface torso scans to predict spinal curve features , . Algorithms based on 2D back images to predict curve severity have been proposed. However, the model performance compared to the scoliometer device has shown to have a lower specificity in detecting non scoliotic individuals . Back asymmetry analysis of participants in the forward bend position has also been proposed to predict the scoliosis curve. Compared to the scoliometer test, the model showed a lower sensitivity in detecting individuals with scoliosis . To improve the overall accuracy of identifying AIS and typical spine growth cases, the CNN approach may be applied for AIS detection based on the ST asymmetry technique. This study aimed to develop and evaluate the efficacy of a classification model aimed at AIS detection using the markerless ST asymmetry technique. 

Data acquisition The dataset in this study included torso surface scans of participants with radiographically confirmed AIS and volunteers with a typical spine development screened negative for AIS from the scoliometer test, yielding a total of 558 participants. The dataset contained 285 individuals diagnosed with AIS and 273 with typically developing spines. All participants in the study were between 10 and 18 years old. Participants with AIS had curves between 10° and 45° at baseline. In addition to the baseline scan, a subset of participants with AIS underwent follow-up scans. The total number of surface scans for participants with AIS, including baseline and follow-up scans, amounted to 693. Similarly, some participants without AIS also had baseline and follow-up surface scans. The total number of surface scans for participants without AIS, including baseline and follow-up scans, amounted to 298. This study reports a secondary analysis of data collected across multiple centers, specifically in Edmonton, Canada, where 782 scans were collected and Campo Grande, Brazil, where 209 scans were collected. Ethical approval, granted by the University of Alberta Health Research Ethics Board, was obtained with approval numbers Pro00118643 and Pro00117065. Participants provided consent in the original studies. Surface topography analysis The ST analysis technique, previously documented in the literature, involved obtaining surface scans of the entire torso – . The data contained torso surface scans acquired using 4 Minolta cameras (Fig. a), capturing the torso’s front, back, right, and left sides. The captured views of the torso were imported into Geomagic Control 2015 (3D Systems, North Carolina, USA) and merged. Additionally, the dataset contained surface scans captured using a Structure sensor (Structure Sensor Pro) attached to an iPad (Fig. b). With the handheld device, a circular path was performed around the participants with the camera focused on the torso to obtain 3D depth information of the torso. The 3D mesh model is then extracted from the scanning app for refinement. The operators collecting the scans were experienced in using the device and the scans were collected in a controlled environment with uniform lighting to reduce the impact of external factors on scan quality. The Minolta cameras were used to collect 782 scans, while the structure sensors were used to collect 209 surface scans. For data collected with the stationary cameras, participants were standing within a frame, with their arms positioned at 90-degree elevation. For scans collected with the structure scanner, the frame was not used; participants were instructed to position the arms at 90-degree elevation by using sticks to maintain their arms in a consistently elevated position throughout the duration of obtaining the scan. All the dataset’s scans were cropped to isolate and maintain a full 3D model of the torso. For asymmetry analysis, the torso model was duplicated and mirrored along the midsagittal plane. The reflected torso was aligned with the original to minimize the distance between the two models (Fig. c). The roto-inversion plane associated with this alignment was termed the plane of best symmetry . Torso asymmetry was evaluated by obtaining the distances between each point on the original torso and its corresponding point on the reflected torso (Fig. d). With a specific focus on the back region of the torso, only 180 degrees of the back’s span, with its deviation information, remained (Fig. e). The deviations following scaling, and their corresponding position on the torso were mapped onto a grid of 102 columns by 102 rows (Fig. f). The depth, representing the distance between the back surface and the coronal plane, was also mapped onto a 102 × 102 grid (Fig. g). The deviation and depth grid maps were the input channels to a convolutional neural network (CNN). Convolutional neural network The CNN architecture is presented in Fig. . The architecture consisted of multiple convolutional layers with max-pooling layers in between, and fully connected layers for binary classification . A rectified linear activation function was applied at each convolutional layer. A sigmoid function was applied to the final output in the architecture. Network weights were updated using the Adam optimizer algorithm , . The outcomes of the CNN model resulted in an output prediction of 1 (positive for AIS) or 0 (negative for AIS). When developing the CNN model, a 10-fold cross-validation was performed. A testing set, which accounted for 20% of the data, was held out from all training for the final evaluation of the model. Data augmentation on the training and validation sets used during cross-validation was conducted to increase the dataset artificially. The data was increased by randomly rotating the input grid maps between − 10 and 10 degrees in the coronal plane. The non-AIS group data was further increased by reflecting along the y-axis (medial-lateral). The number of epochs was limited to 500 during training. A batch size of 32 and a learning rate of 0.001 were also applied. Since the distribution between the AIS and control groups was unbalanced, the non-AIS group cases were oversampled during training to ensure a balanced number of cases in each batch. Specifically, random oversampling was applied, where samples from the minority class (control group) were chosen at random to be duplicated and added to the training dataset. For the testing set, AIS and control cases were randomly selected from the dataset. Half of the AIS group in the testing set had maximum curves with Cobb angle less than 25°, and the remaining half of the AIS group had Cobb angles greater than 25°. This stratification by curve severity was conducted to evaluate the model’s sensitivity in detecting AIS. Statistical analysis Sensitivity, which is the ability of the model to correctly identify AIS, was determined and calculated from the prediction outcomes of true positives (TP) and false negatives (FN) (Table ). Likewise, specificity, which is the ability to correctly identify negative cases, was determined. The true negatives (TN) and false positives (FP) outcomes are used to calculate specificity (Table ). The model performance was also evaluated using the false positive rate and false negative rate. Additional measures to evaluate the model were the positive likelihood ratio (PLR) and the negative likelihood ratio (NLR) (Table ). The testing set not used during training was used to evaluate the model’s performance. The model with the best accuracy was selected from the models generated during 10-fold cross-validation. A receiver-operating characteristic (ROC) analysis was conducted to determine the ability of the classification model to discern ST maps between positive AIS and negative AIS cases. The ROC curve is obtained by plotting the sensitivity over the false positive rate at various model output thresholds The area under the curve (AUC) measuring the overall accuracy of the classifier compared to the ground truth is determined from the ROC curve. An AUC of 1.0 represents high sensitivity and minimal false positive rate . 

The dataset in this study included torso surface scans of participants with radiographically confirmed AIS and volunteers with a typical spine development screened negative for AIS from the scoliometer test, yielding a total of 558 participants. The dataset contained 285 individuals diagnosed with AIS and 273 with typically developing spines. All participants in the study were between 10 and 18 years old. Participants with AIS had curves between 10° and 45° at baseline. In addition to the baseline scan, a subset of participants with AIS underwent follow-up scans. The total number of surface scans for participants with AIS, including baseline and follow-up scans, amounted to 693. Similarly, some participants without AIS also had baseline and follow-up surface scans. The total number of surface scans for participants without AIS, including baseline and follow-up scans, amounted to 298. This study reports a secondary analysis of data collected across multiple centers, specifically in Edmonton, Canada, where 782 scans were collected and Campo Grande, Brazil, where 209 scans were collected. Ethical approval, granted by the University of Alberta Health Research Ethics Board, was obtained with approval numbers Pro00118643 and Pro00117065. Participants provided consent in the original studies. 

The ST analysis technique, previously documented in the literature, involved obtaining surface scans of the entire torso – . The data contained torso surface scans acquired using 4 Minolta cameras (Fig. a), capturing the torso’s front, back, right, and left sides. The captured views of the torso were imported into Geomagic Control 2015 (3D Systems, North Carolina, USA) and merged. Additionally, the dataset contained surface scans captured using a Structure sensor (Structure Sensor Pro) attached to an iPad (Fig. b). With the handheld device, a circular path was performed around the participants with the camera focused on the torso to obtain 3D depth information of the torso. The 3D mesh model is then extracted from the scanning app for refinement. The operators collecting the scans were experienced in using the device and the scans were collected in a controlled environment with uniform lighting to reduce the impact of external factors on scan quality. The Minolta cameras were used to collect 782 scans, while the structure sensors were used to collect 209 surface scans. For data collected with the stationary cameras, participants were standing within a frame, with their arms positioned at 90-degree elevation. For scans collected with the structure scanner, the frame was not used; participants were instructed to position the arms at 90-degree elevation by using sticks to maintain their arms in a consistently elevated position throughout the duration of obtaining the scan. All the dataset’s scans were cropped to isolate and maintain a full 3D model of the torso. For asymmetry analysis, the torso model was duplicated and mirrored along the midsagittal plane. The reflected torso was aligned with the original to minimize the distance between the two models (Fig. c). The roto-inversion plane associated with this alignment was termed the plane of best symmetry . Torso asymmetry was evaluated by obtaining the distances between each point on the original torso and its corresponding point on the reflected torso (Fig. d). With a specific focus on the back region of the torso, only 180 degrees of the back’s span, with its deviation information, remained (Fig. e). The deviations following scaling, and their corresponding position on the torso were mapped onto a grid of 102 columns by 102 rows (Fig. f). The depth, representing the distance between the back surface and the coronal plane, was also mapped onto a 102 × 102 grid (Fig. g). The deviation and depth grid maps were the input channels to a convolutional neural network (CNN). 

The CNN architecture is presented in Fig. . The architecture consisted of multiple convolutional layers with max-pooling layers in between, and fully connected layers for binary classification . A rectified linear activation function was applied at each convolutional layer. A sigmoid function was applied to the final output in the architecture. Network weights were updated using the Adam optimizer algorithm , . The outcomes of the CNN model resulted in an output prediction of 1 (positive for AIS) or 0 (negative for AIS). When developing the CNN model, a 10-fold cross-validation was performed. A testing set, which accounted for 20% of the data, was held out from all training for the final evaluation of the model. Data augmentation on the training and validation sets used during cross-validation was conducted to increase the dataset artificially. The data was increased by randomly rotating the input grid maps between − 10 and 10 degrees in the coronal plane. The non-AIS group data was further increased by reflecting along the y-axis (medial-lateral). The number of epochs was limited to 500 during training. A batch size of 32 and a learning rate of 0.001 were also applied. Since the distribution between the AIS and control groups was unbalanced, the non-AIS group cases were oversampled during training to ensure a balanced number of cases in each batch. Specifically, random oversampling was applied, where samples from the minority class (control group) were chosen at random to be duplicated and added to the training dataset. For the testing set, AIS and control cases were randomly selected from the dataset. Half of the AIS group in the testing set had maximum curves with Cobb angle less than 25°, and the remaining half of the AIS group had Cobb angles greater than 25°. This stratification by curve severity was conducted to evaluate the model’s sensitivity in detecting AIS. 

Sensitivity, which is the ability of the model to correctly identify AIS, was determined and calculated from the prediction outcomes of true positives (TP) and false negatives (FN) (Table ). Likewise, specificity, which is the ability to correctly identify negative cases, was determined. The true negatives (TN) and false positives (FP) outcomes are used to calculate specificity (Table ). The model performance was also evaluated using the false positive rate and false negative rate. Additional measures to evaluate the model were the positive likelihood ratio (PLR) and the negative likelihood ratio (NLR) (Table ). The testing set not used during training was used to evaluate the model’s performance. The model with the best accuracy was selected from the models generated during 10-fold cross-validation. A receiver-operating characteristic (ROC) analysis was conducted to determine the ability of the classification model to discern ST maps between positive AIS and negative AIS cases. The ROC curve is obtained by plotting the sensitivity over the false positive rate at various model output thresholds The area under the curve (AUC) measuring the overall accuracy of the classifier compared to the ground truth is determined from the ROC curve. An AUC of 1.0 represents high sensitivity and minimal false positive rate . 

Data characteristics of training and validation sets during cross-validation as well as the testing set are shown in Table . Participants’ age, height, weight, and Cobb angle of the largest curve were balanced between training + validation and testing sets. The data size during training were increased from 796 to 1860 following data augmentation. Average accuracy of 100% and 94% were obtained from the 10-fold training and validation sets, respectively. Additional results are shown for each cross-validation fold in Table . The average testing set across the 10 folds yielded an accuracy of 93%, a sensitivity of 96%, and a specificity of 87%. The model selection process involved choosing the fold with the highest training accuracy. The best-performing model during cross-validation was evaluated with the testing set presented in Fig. . Model prediction of participants with curves less than 25° was compared to the ground truth (Fig. ). The sensitivity of detecting curves was less than 25° was 96%. From the ROC curve, an AUC of 0.957 (95% CI 0.922; 0.992) was obtained by comparing the CNN classifier outputs to the ground truth. Participants with curves greater than 25° were also compared to the ground truth, and a sensitivity of 99% was achieved. The AUC for the data with only moderate curves (Cobb > 25°) was 0.985 (95% CI 0.965;1 .00). The sensitivity of the combined AIS data was 97%, with a false negative rate of 2.9%. Consequently, when comparing the model prediction of non-scoliosis participants against the ground truth, a specificity of 90% was obtained, with a false positive rate of 10%. The AUC of the combined data obtained from ROC analysis was 0.971 (95% CI 0.945;0.996). The accuracy for the combined data in Fig. was 95%. The positive likelihood ratio (PLR) was 9.7, and the negative likelihood ratio (NLR) was 0.032. 

The purpose of this study was to develop a classification model and to investigate the viability of markerless ST as a screening tool for AIS. A CNN-based binary classification model was developed using deviations and depth information of the back surface of the torso. Classification results from the CNN model indicate a high overall accuracy of 95% in distinguishing between individuals with and without scoliosis based on torso asymmetry. A high sensitivity of 97% was observed using the proposed method, which can minimize false negatives. Additionally, a high specificity of 90% can minimize false positives compared to standard screening methods. The achieved positive likelihood ratio of 9.7 indicates that the odds of obtaining a positive result in individuals with AIS are approximately 10 times higher than those without the condition. The low negative likelihood ratio of 0.032 suggests that individuals with AIS obtaining a negative result are nearly negligible. The observed likelihood ratios highlight the potential of the ST markerless technique as a reliable screening tool with strong discriminative capabilities. Our results exhibit notable advantages compared to conventional screening methods like the scoliometer and Adam’s forward bend test. The model based on ST maps demonstrated a lower false positive rate of 10% compared to the scoliometer test, with a false positive rate of 19.3% . The decrease in the false positive rate leads to a decrease in unnecessary x-ray referrals that expose children to ionizing radiation. While these findings suggest a promising approach for scoliosis screening, conducting a comprehensive validation study directly comparing the performance of the CNN model and clinical screening tools with the ground truth obtained from x-ray examination is needed. Future work can include a cross-sectional study evaluating the screening performance of the ST approach and current clinical screening tests in a school screening program. Assessing the reliability and generalizability of the ST approach would contribute to the practical implications the proposed method in clinical settings. Additionally, the cost effectiveness of the ST approach should be assessed in future studies to determine whether the improved accuracy justifies the potential increase in time and resource requirements. The proposed model using ST analysis is automatic and avoids human error, while the accuracy of the standard tools depends on the experience of the operator. Studies have evaluated the use of torso asymmetries to predict scoliosis. Kokabu et al. proposed an automatic system using a depth sensor to scan individuals back in Adam’s forward bend position to detect AIS. The proposed method relies on determining the asymmetry index based on the deviations between the reflected and original point cloud of the back following the best alignment algorithm. The cutoff value of the asymmetry index to predict Cobb angle greater than 15° was predicted based on receiver operating characteristic (ROC) analysis. The authors reported that the cutoff value yielded an accuracy of 84% to predict cases with Cobb greater than 15°. The authors also obtained a sensitivity of 79% and a specificity of 92% . However, their study did not analyze the cutoff value for predicting Cobb greater than 10°, which is the minimum angle to be classified as scoliosis. The study included younger participants, and the sample size was much smaller in Kokabu et al.’s study compared to our present analysis, with 76 total participants suspected of having scoliosis based on an x-ray examination. Yang et al. developed a deep-learning algorithm to screen for scoliosis using unclothed back images. The binary classification algorithm for detecting scoliotic images from non-scoliosis images exhibited an average area under the curve (AUC) of 0.946 from an external validation set (scoliotic images n = 300, normal controls n = 100). The authors reported a sensitivity of 87.5% and a specificity of 83.5% . This study focused exclusively on unclothed back photographs for classifying scoliosis and non-scoliosis cases. In contrast, our approach incorporated the ST technique to evaluate back asymmetry comprehensively. In addition, our study also considered depth information of the back to capture variations in spatial dimensions of the torso. Chowanska et al. used Moiré topography to compute the surface trunk rotation parameter as a scoliosis screening method. However, their study’s findings reported unsatisfactory sensitivity and specificity . The primary challenge stemmed from the difficulty in determining an optimal surface trunk rotation parameter cutoff value, ultimately resulting in diminished screening accuracy (sensitivity of 77.4% and specificity of 71.1%) . Additionally, their study did not use advanced learning algorithms to distinguish between scoliosis and non-scoliosis images. The results from our proposed screening model using ST were similar to other studies evaluating torso asymmetries to detect and classify AIS. The overall accuracy of the CNN classifier (95%) was greater than that of the accuracy results using the asymmetry index of participants in the forward bend position (84%) proposed by Kokabu et al.. Our accuracy performance (sensitivity of 97%, specificity of 90%) was also greater than that of the deep learning model developed by classifying photographs of participants’ back torso surfaces (sensitivity of 87.5%, specificity of 83.5%), which was proposed by Yang et al.. Likewise, the Chowanska et al. classification threshold to screen for scoliosis had lower accuracy (sensitivity of 77.4%, specificity of 71.1%) than the model proposed in this study. Several limitations were identified in this study. The CNN-based binary approach, while effective for initial screening purposes, does not provide information about the severity of the scoliosis curve. Future work can include developing regression-based models to predict curve magnitude. Building such models would require additional data, with wide ranging cobb angles. The sample size between classification groups were unbalanced, introducing potential bias and can impact the generalizability of the model. Surface torso scans were collected with two different acquisition system. A higher resolution is observed with the Minolta scanners compared to the Structure sensor. However, the latter provides a more affordable handheld system which can easily be moved around the participants’ trunk. Multiple operators collected the surface scans, and thus inter-and intra reliability analysis should be conducted when using the Structure sensor system. Excellent reliability has previously been observed with the Minolta system for classifying curve severity based on the ST approach . Another limitation is that the torso scans were cropped around the posterior superior iliac spine. The exclusion of the pelvis in the data implies that the pelvis rotation often seen in AIS individuals was not accounted for. Additionally, the study acknowledges a limitation in sample size, especially concerning the application of CNN algorithms and other neural networks, which typically require larger datasets to mitigate the risk of overfitting. These limitations highlight the need for cautious interpretation and future refinement to enhance the robustness and applicability of the proposed method. 

The study aimed to ascertain the viability of markerless ST as a screening tool for AIS. The results presented in this paper provide promising evidence supporting the effectiveness of this method in assessing torso asymmetry and identifying individuals with scoliosis using a CNN algorithm. Our findings demonstrate the potential of markerless surface topography as a reliable and non-invasive means of screening for AIS. The significance of using the ST method in clinics and community screening is that it can enhance the efficiency and accuracy of scoliosis detection, allowing for earlier diagnosis, potentially preventing the condition’s progression, thereby improving patient outcomes. 

