Sex estimation using skull silhouette images from postmortem computed tomography by deep learning
In August 2023, wildfires in Maui killed numerous victims. Individual identification during the disaster was difficult because many bodies were severely damaged, and the disaster scene was extremely large and complex . Hence, it is important to investigate suitable approaches for personal identification in large-scale disasters. In 2007, Rutty et al. reported the use of mobile postmortem computed tomography (PMCT) for mass fatality incidents . Subsequently, PMCT was recognized as being a useful tool for disaster victim identification (DVI) . PMCT is a noninvasive technique to quickly obtain whole-body information. Furthermore, image data can be stored semi-permanently and shared worldwide . According to “The Program on Promotion of Policy about Death Investigation” enacted in Japan, postmortem imaging is a suitable method for performing scientific investigations concerning personal identification. This law also promotes the development of databases for personal identification . PMCT has been extensively studied for various aspects of forensic investigation, including the determination of cause of death and personal identification – . These studies underscore the importance of integrating advanced imaging techniques like PMCT into forensic practice to enhance the accuracy and efficiency of investigations. While the soft tissues of the human body decay over time after death, the bones undergo minimal change; therefore, useful information for personal identification is retained. In particular, the morphological information of the bone can be used for sex estimation, which is important for personal identification. Male bones are generally bigger, thicker, and more uneven than those of females from the same age group . The skull and pelvis have particularly obvious sex-related differences in shape. In 2008, Walker obtained 88% accuracy of sex estimation using a discriminant function from skull shapes assessed visually . In 2022, Garcovich et al. obtained 88.7% accuracy in males and 90.7% accuracy in females by performing geometric morphometric analysis of frontal bone landmarks in head X-ray images . Although these studies have yielded valuable results, they suffer from a reproducibility crisis because of the visual assessment of the skull shape or manual setting of landmarks. During DVI among a large number of corpses, it is necessary to achieve rapid and accurate personal identification. To make rapid and accurate sex estimation possible, it is important to investigate the possibility of using artificial intelligence (AI). The applicability of deep-learning methods, such as convolutional neural networks (CNNs), has been explored in various fields. In forensic anthropology and forensic pathology, CNNs are used to solve certain problems , such as sex and age estimation – , recognition of head measurement landmarks , and segmentation of head structures . In a study on sex estimation using deep learning, Cao et al. achieved more than 90% accuracy using pelvic CT images . In 2019, Bewes et al. reported 95% accuracy using three-dimensional (3D) skull reconstruction image from antemortem CT . In addition to sex estimation with antemortem CT, sex estimation with PMCT or X-ray imaging must be considered to develop a practical method that can be used worldwide . Recent advancements have proposed new sex estimation methods that combine the advantages of visual observation and the accuracy of measurement-based approaches. For example, Nikita and Michopoulou (2019) proposed a method that quantifies the shape of specific cranial regions to achieve a 94% accuracy in sex classification , . Yang (2019) achieved 96.7% accuracy by machine learning algorithms to 3D cranial models . However, these various prior studies still require a manual process to identify key features or provide 3D data, which limits the automation and convenience of sex estimation. In this study, two-dimensional (2D) silhouette images were obtained (hereafter referred to as silhouette images) from head PMCT only showing the skull outline and used for sex estimation in personal identification employing trained deep-learning models. Silhouette images can be acquired from both PMCT images and X-ray imaging, which is commonly used worldwide. To the best of our knowledge, this is the first study to report sex estimation using silhouette images.   Database This study included 264 cases (132 of each sex) from 337 head PMCT cases (203 males and 134 females) who underwent PMCT at autopsy. Patients with bone defects located on the outline of the skull in the silhouette image were excluded. Bone defects inside the skull outline can be covered by filling inside the skull outline using a labeling process, which is described later, and these cases were included. In addition, cases wherein the parietal lobe was not within the imaging range and children (under 18) were excluded. Furthermore, we randomly excluded some PMCT cases to obtain the same number of male and female images to avoid biasing the results of sex estimation using deep-learning models. The patients were aged 18–97 years old, with an average age of 58.5 years. Bodies were placed in bags without positioning. The study was approved by the Institutional Review Board of Kyushu University, Japan (approval number #2017-27-285). All methods were performed in accordance with relevant guidelines and regulations. Due to the retrospective nature of the study, Kyushu University Institutional Review Board waived the need of obtaining informed consent. PMCT images were obtained using a 16-row multidetector CT scanner (ECLOS, Hitachi Medical Corporation, Tokyo, Japan) in 2014–2019 at Department of Forensic Pathology and Sciences, Kyushu University (Table ), and were anonymized by removing patient IDs, names, and other personal information before usage. All image data were corrected using the skull angle correction developed by Kawazoe et al . so that the skull had the same position as that in the antemortem CT imaging. For the application of deep learning, we applied a four-fold cross-validation, and the 132 cases of each sex were divided into 99 training images (75%) and 33 test images (25%). Of the 99 training images, 29 were used as validation images.   This study included 264 cases (132 of each sex) from 337 head PMCT cases (203 males and 134 females) who underwent PMCT at autopsy. Patients with bone defects located on the outline of the skull in the silhouette image were excluded. Bone defects inside the skull outline can be covered by filling inside the skull outline using a labeling process, which is described later, and these cases were included. In addition, cases wherein the parietal lobe was not within the imaging range and children (under 18) were excluded. Furthermore, we randomly excluded some PMCT cases to obtain the same number of male and female images to avoid biasing the results of sex estimation using deep-learning models. The patients were aged 18–97 years old, with an average age of 58.5 years. Bodies were placed in bags without positioning. The study was approved by the Institutional Review Board of Kyushu University, Japan (approval number #2017-27-285). All methods were performed in accordance with relevant guidelines and regulations. Due to the retrospective nature of the study, Kyushu University Institutional Review Board waived the need of obtaining informed consent. PMCT images were obtained using a 16-row multidetector CT scanner (ECLOS, Hitachi Medical Corporation, Tokyo, Japan) in 2014–2019 at Department of Forensic Pathology and Sciences, Kyushu University (Table ), and were anonymized by removing patient IDs, names, and other personal information before usage. All image data were corrected using the skull angle correction developed by Kawazoe et al . so that the skull had the same position as that in the antemortem CT imaging. For the application of deep learning, we applied a four-fold cross-validation, and the 132 cases of each sex were divided into 99 training images (75%) and 33 test images (25%). Of the 99 training images, 29 were used as validation images.   Figure shows the procedure for creating silhouette images from head PMCT data. First, the patient table and body bag were removed from the PMCT volume data, which had been positioning-corrected using a semi-automatic method for readjusting the head position as proposed by Kawazoe et al . The soft tissues were removed via threshold processing with 300 Hounsfield units [HU], leaving only the bone. Subsequently, 2D images were created using maximum intensity projection by varying the yaw angles in seven different directions (0° (frontal view), 15°, 30°, 45°, 60°, 75°, and 90° (lateral view)). The seven directions were chosen to capture a comprehensive range of anatomical features: for example, the frontal view reveals the zygomatic bone, while the lateral view highlights the superciliary arch and the external occipital protuberance (EOP). By including intermediate angles, we aimed to ensure that the silhouette images represent various aspects of the skull’s morphology. The 2D images were converted into silhouette images using binarization processing. The caudal signal from the cervical vertebrae was removed to easily learn only the morphological information of the skull. In certain lateral cases, observing the mastoid is challenging owing to the overlap of the mandible and cervical vertebrae. To address this issue, the cutting position of the lowest skull coordinates in the lateral images was set to 15 slices above that for the other six directions. Subsequently, Noise removal was achieved by first using the contour detection functions from the Python library OpenCV to automatically identify the largest contour corresponding to the skull. Then, we filled the inside of the detected contour to remove internal noise and subsequently removed any remaining fine noise outside the skull, addressing background noise effectively. Finally, silhouette images for deep learning were created by cropping the image size to 256 × 256 pixels and centering them on the skull (Fig. ). A Python program was used for image processing of creating silhouette images. . We implemented data augmentation with horizontal flip and ± 5° rotation to increase the number of training and validation images four times (70 × 4 training and 29 × 4 validation images for each sex (560 training and 232 validation images)). The nearest neighbor method was selected as the interpolation method for image rotation. In addition, silhouette images were converted into red-green-blue images to apply transfer learning using ImageNet . This will be discussed later in this section.   AlexNet and VGG16 were used for sex estimation. All deep-learning models adopt transfer learning using the parameters learned by ImageNet, which is a large database with over 10 million color images. During training, the batch size was 64, the optimization algorithm was Adam, the activation function was SoftMax, and the loss function was cross-entropy. The learning rates were 1e-5 for AlexNet and 2e-6 for VGG16. We applied an early stopping function that stopped the training when the validation loss was minimized. Deep learning was performed using Python with an Intel Xeon Silver 4110 central processing unit, 32 GB of memory, and an NVIDIA TITAN RTX graphics processing unit.   The test results for sex estimation were evaluated for accuracy. Accuracy was defined as the percentage of images where the sex was correctly identified among 264 images (four times 33 test images for each sex) and was calculated for each deep-learning model and each projection angle. In addition, a majority vote based on the test results of multiple viewing angles, hereafter referred to as the majority vote, was conducted. A case identified correctly in more than half of the projection angles was regarded as true. In addition, we evaluated the performance of each model by calculating Matthews Correlation Coefficient (MCC). We computed precision, recall, and F1 score for each sex class (male and female) to provide a more comprehensive understanding of the models’ classification performance. To evaluate the trained deep-learning models, we created a receiver operating characteristic (ROC) curve by plotting the sensitivity on the vertical axis and 1 − specificity on the horizontal axis. Sensitivity was defined as the ratio of male images correctly identified as males, and specificity was defined as the ratio of female images correctly identified as females. We calculated the area under the ROC curve (AUC) as a quantitative index. Furthermore, gradient-weighted class activation mapping (Grad-CAM) , which is among the most commonly employed methods for explainable AI, was used to investigate which parts of the skull were focused on by the deep-learning models. We compared the Grad-CAM heatmap to areas with obvious sex differences (superciliary arch, forehead, parietal, and external occipital protuberance (EOP)) described in forensic anthropology and forensic pathology .   Performance of deep-learning models Table presents the average values of the training accuracy, training loss, validation accuracy, and validation loss in the seven directions for each deep-learning model. VGG16 achieved the highest value for validation accuracy and validation loss. Both models exhibited the highest validation accuracy in the lateral projection: AlexNet = 88.6% and VGG16 = 90.1%. Tables and present the values of precision, recall, accuracy, and MCC for sex estimation across seven angles for different deep-learning models. For AlexNet, the highest accuracy was achieved at the 90° projection angle, with an accuracy of 0.886. This angle also yielded the highest MCC value of 0.774, indicating a strong overall performance. The precision and recall values at this angle were 0.911 and 0.856 for males, and 0.864 and 0.917 for females, respectively. Similarly, for the VGG16 model, the lateral view achieved the highest accuracy of 0.898 and the highest MCC of 0.796. At this angle, the precision and recall were 0.913 and 0.879 for males, and 0.883 and 0.917 for females, respectively. Furthermore, Fig. shows that the ROC curve from the lateral view achieved the highest accuracy among all projection angles for each deep-learning model. The AUC values for AlexNet and VGG16 were 0.951 and 0.957, respectively. Notably, VGG16 consistently demonstrated higher precision, recall, and MCC compared to AlexNet in most angles, particularly excelling in the lateral view. These results suggest that VGG16 is generally more effective and reliable for sex estimation across various angles. Table presents the results of the majority vote in three directions (0°, 45°, and 90°) and seven directions (0–90°). The majority vote in three directions exhibited slightly higher accuracy in AlexNet (89.4%) and VGG16 (91.7%) than in only one direction, as shown in Tables and .   Table presents the average values of the training accuracy, training loss, validation accuracy, and validation loss in the seven directions for each deep-learning model. VGG16 achieved the highest value for validation accuracy and validation loss. Both models exhibited the highest validation accuracy in the lateral projection: AlexNet = 88.6% and VGG16 = 90.1%. Tables and present the values of precision, recall, accuracy, and MCC for sex estimation across seven angles for different deep-learning models. For AlexNet, the highest accuracy was achieved at the 90° projection angle, with an accuracy of 0.886. This angle also yielded the highest MCC value of 0.774, indicating a strong overall performance. The precision and recall values at this angle were 0.911 and 0.856 for males, and 0.864 and 0.917 for females, respectively. Similarly, for the VGG16 model, the lateral view achieved the highest accuracy of 0.898 and the highest MCC of 0.796. At this angle, the precision and recall were 0.913 and 0.879 for males, and 0.883 and 0.917 for females, respectively. Furthermore, Fig. shows that the ROC curve from the lateral view achieved the highest accuracy among all projection angles for each deep-learning model. The AUC values for AlexNet and VGG16 were 0.951 and 0.957, respectively. Notably, VGG16 consistently demonstrated higher precision, recall, and MCC compared to AlexNet in most angles, particularly excelling in the lateral view. These results suggest that VGG16 is generally more effective and reliable for sex estimation across various angles. Table presents the results of the majority vote in three directions (0°, 45°, and 90°) and seven directions (0–90°). The majority vote in three directions exhibited slightly higher accuracy in AlexNet (89.4%) and VGG16 (91.7%) than in only one direction, as shown in Tables and .   The heatmap outputs obtained using Grad-CAM are shown in Fig. . The model paid more attention to the areas shown in warm colors than those in cold colors. The deep-learning models identified specific anatomical features of the skull as most influential in determining sex. In males, the model primarily focused on the superciliary arch, particularly in the lateral (90 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$\:^\:$$ ) view, while in females, the emphasis was on the EOP in the lateral view and the zygomatic bones in the frontal (0 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$\:^\:$$ ) view. Figure shows that VGG16 consistently identified key anatomical features associated with sexual dimorphism. In female cases, the zygomatic bone was prominently recognized at 0°, while both the superciliary arch and the external occipital protuberance (EOP) were distinctly highlighted at 90°. AlexNet also recognized these features but with less precision, often focusing more broadly on internal skull regions and background areas. The agreement between the focus of the Grad-CAM and areas with obvious sex differences in forensic anthropology was investigated. Here, 33 test cases for each sex, whose accuracies were the closest to the overall accuracy in the four-fold cross-validation, were analyzed in lateral images of VGG16, which showed the highest accuracy (Table ). It was found that 93% of the images (27/29) correctly estimated as males were focused on the superciliary arch and all images (31/31) correctly estimated as females were focused on the EOP.   In this study, we created skull silhouette images from head PMCT and conducted sex estimations using two deep-learning models. Sex estimation using silhouette images achieved an accuracy of 89.8% (Table ). In addition, the majority vote in the three directions improved the accuracy to 91.7%. This performance is comparable to that of conventional sex estimation using head X-ray imaging , , and shows the feasibility of sex estimation by combining silhouette images and deep learning. Conventional methods for estimating sex from the skull have involved the use of discriminant functions with manual measurement values , or setting of landmarks in images . These traditional approaches rely on forensic anthropologists manually identifying anatomical features that typically exhibit sexual dimorphism, such as the more robust and larger structures found in male skulls, including prominent superciliary arch, thicker zygomatic arches, larger mastoid processes, marked EOP, a steeper forehead slope, and more pronounced parietal bossing (Fig. ). However, these methods are time-consuming and therefore unsuitable during DVI in large-scale disasters. To address these challenges, we investigated sex estimation using deep learning with 2D images obtained from PMCT and demonstrated the potential usefulness of our automatic and rapid approach. A comparison of the projection angles in the test results revealed the highest accuracy to be at 90° for AlexNet and VGG16 (Table ). Silhouette images likely reflect sex differences in the superciliary arch and EOP, which are clearly observable near the lateral direction. When comparing the deep-learning models, VGG16 showed the highest accuracy in four directions (0°, 30°, 45°, and 90°), as well as the highest AUC value (Fig. ). VGG16 contained more training parameters than AlexNet and could more effectively recognize points with large sex differences. Analysis of Grad-CAM heatmaps revealed sex differences aligned with forensic anthropology findings, notably in the superciliary arch, zygomatic bone, and (Fig. ). Among these, the VGG16 model demonstrated a particular focus on the outline of the skull, corroborating previous studies that reported VGG16’s superior reliability over Alexnet for visual assessment . Specifically, Grad-CAM analysis showed that in correctly identified cases, the superciliary arch was emphasized in 93% of male cases (27/29), while the EOP was recognized in all female cases (31/31), highlighting a clear sex difference. These results suggest that the deep learning models, particularly VGG16, rely on key anatomical features such as the superciliary arch for males and the EOP for females in sex estimation. Additionally, the zygomatic bone was also identified as influential features in some cases. This observation is consistent with traditional forensic methods, which also emphasize these regions for sex determination. However, it is important to note that these models, including VGG16, occasionally produced heatmaps with warm colors outside the skull, indicating potential areas of model uncertainty or error. In the majority vote, an odd number of projection angles were used because cases wherein over half the angles were identified were regarded as true (e.g., over four out of seven angles). The majority vote in the seven directions did not improve the performance (Table ). This may be because the decision points of deep-learning models were the same in five oblique directions (15–75°), wherein the heatmaps output by Grad-CAM represent similar positions (zygomatic bone). In fact, the test results in seven directions tended to misidentify the same cases, particularly in these five directions. The majority vote in three directions (0°, 45°, and 90°) improved the performances of AlexNet and VGG16, with a maximum accuracy of 91.7%. According to the Grad-CAM heatmaps, both deep-learning models recognized different points in these three directions (0°, zygomatic bone; 45°, zygomatic bone and mastoid process; 90°, superciliary arch and EOP), and the effects of the majority vote were evident. This improvement in accuracy indicates that it is important to train the models with the skull outline obtained from multiple projection angles, that is, the 3D skull shape. However, Matsunobu et al. [37] reported that 3D images contain more information than 2D images and therefore require more computation time, making them unsuitable for DVI. Therefore, developing a model that can learn multiple 2D contour shapes is required to improve accuracy. Our approach focuses on silhouette images, which capture the skull’s edge contours, including sexually dimorphic features like parietal bossing, supraorbital ridge elevation, and the external occipital protuberance. Silhouette images can also alleviate the effects of decay and partial defects in the body by filling the skull contours. This is particularly useful in forensic cases where bodies found during disasters often exhibit defects or damage to the skull. In fact, approximately 10% of the PMCT images used in this study had sphenoid and temporal bone defects. The area around the sphenoid bone, which is relatively thin, tends to perish easily and cannot be depicted well. Surface-rendered and volume-rendered images, which represent unevenness, depict defects that may affect deep learning. In contrast, 2D silhouette images represent only outline information, excluding internal contrast details that may be influenced by decay or damage, making them more robust for analysis even when parts of the skull are damaged. Therefore, compared to traditional manual landmark-setting methods, our deep learning approach using 2D silhouette images can significantly reduce the time required for sex estimation by automating the process. This makes it particularly valuable in forensic contexts, especially in large-scale disasters where quick and reliable identification is critical. Furthermore, the ability of silhouette images to effectively represent key features, even in cases of partial skull damage, highlights the utility of our method in various forensic scenarios. In the case of large-scale disasters, silhouette images can be created using a portable X-ray system, even if PMCT imaging is difficult to perform. This means that this study can be adapted for DVI even in countries where PMCT is unavailable. Accordingly, this study indicates that 2D silhouette images created from X-ray images are useful for estimating the probable sex of victims as a novel method in DVI. Integrating deep learning with PMCT offers significant advantages, especially in regions where traditional autopsies may be culturally or ethically challenging, such as Japan and Korea. PMCT provides a non-invasive alternative that is likely to be more acceptable in such contexts. Additionally, the combination of deep learning and PMCT enhances the efficiency and accuracy of sex estimation, which is crucial in scenarios requiring rapid identification, like large-scale disasters. While this approach is valuable for efficiently managing large volumes of cases and reducing the workload for forensic experts, it is essential to apply these advanced techniques with careful consideration of ethical implications, particularly concerning privacy and transparency, to ensure sensitive information is handled appropriately. Limitations Our study had several limitations. First, the dataset was collected from a single institution and consisted of 264 cases, which limits the generalizability of our findings across different populations and age groups. We did not explore how sex estimation accuracy might vary based on population diversity or age. However, individuals under 18 years of age, whose growth and secondary sexual characteristics were not fully developed, were excluded from this study. Future studies should aim to include a broader and more diverse dataset to assess the impact of population and age variation on sex estimation accuracy. Second, the entire skull was used to train deep-learning models. The proposed approach cannot be used to identify cases that differ significantly from the usual shape because cases with large defects on the skull outline were excluded in this study. Sex estimation from a part of the skull also must be considered in future studies. Finally, the number of training images was insufficient. The maximum accuracy was 89.8% and further improvements in performance are required for actual personal identification. Therefore, deep learning should be applied with additional training images to solve this problem. In addition, considering the rapid progress in deep learning , the development and assessment of AI models for personal identification is a future challenge.   Our study had several limitations. First, the dataset was collected from a single institution and consisted of 264 cases, which limits the generalizability of our findings across different populations and age groups. We did not explore how sex estimation accuracy might vary based on population diversity or age. However, individuals under 18 years of age, whose growth and secondary sexual characteristics were not fully developed, were excluded from this study. Future studies should aim to include a broader and more diverse dataset to assess the impact of population and age variation on sex estimation accuracy. Second, the entire skull was used to train deep-learning models. The proposed approach cannot be used to identify cases that differ significantly from the usual shape because cases with large defects on the skull outline were excluded in this study. Sex estimation from a part of the skull also must be considered in future studies. Finally, the number of training images was insufficient. The maximum accuracy was 89.8% and further improvements in performance are required for actual personal identification. Therefore, deep learning should be applied with additional training images to solve this problem. In addition, considering the rapid progress in deep learning , the development and assessment of AI models for personal identification is a future challenge.   This study demonstrates the potential of utilizing 2D silhouette images from postmortem CT scans for rapid and accurate sex estimation in forensic science. By applying deep learning techniques, key anatomical features such as the superciliary arch and external occipital protuberance were effectively identified, yielding high levels of accuracy. The proposed method is fast, robust, and provides valuable support for personal identification in various forensic contexts.   