Quality control stress test for deep learning-based diagnostic model in digital pathology
Digital pathology is an emerging transformation of diagnostic pathology . At that, histological slides can be digitized using histological scanner and reviewed on the monitor without a microscope. One of the promises of digital pathology is an automated analysis of pathological specimens using deep learning-based models . Deep learning (DL) is a powerful technology for image analysis based on convolutional neural networks . The latter use unsupervised feature generation by convolving images using multiple different filters and aggregate the corresponding representations in different ways to achieve a prediction. Many studies to date addressed the feasibility of DL for diagnostic applications with high accuracy  resulting in several commercial diagnostic products that are being developed for prostate cancer, breast cancer, skin diseases, and in other medical domains. However, this is still a work in progress. Importantly, histological slides may be very heterogeneous with regard to staining, thickness of sections, and are especially prone to artifacts during tissue processing, cutting, staining, and digitization. Such heterogeneity is prominent among different institutions and even within the same laboratory. The aim of the pathologist and of any assisting system (e.g., DL-based) is to provide top diagnostic accuracy irrespective of pre-analytical issues and artifacts. Only few studies investigate how DL-based models will operate in case of artifacts . Generalization of this information is difficult for models from different domains, tasks (classification, segmentation, detection), different neural network architectures, and models using different ground truth. However, this knowledge is of utmost importance for assisting systems to be implemented in the routine diagnostic workflow. In this work we attempt to digitally reproduce major types of histological artifacts and perform a systematic stress test for a DL-based tool for prostate cancer (PCA) detection in histological slides which previously showed high accuracy (>98%) using external validation sets . We investigate mechanism of misclassification in different types of artifacts and discuss strategies of how to prevent negative effects of artifacts on model accuracy during development and implementation of algorithm.   Model description The DL-based patch-level classification model for PCA detection used in the study was previously trained using a large-high-quality dataset based on the histological slides from The Cancer Genome Atlas (TCGA) PCA cohort containing ~1.5 million patches . Several convolutional network architectures were tested with NASNetLarge and InceptionResNetV2 showing similar highest accuracy. In current implementation the model is based on the completely retrained InceptionResNetV2 architecture with fully connected layer containing 256 units and classification layer for three classes (prostate glandular tissue, nonglandular tissue, and tumor tissue). The patch size is 300 × 300 pixels (px) corresponding to the whole slide image (WSI) region size of ~150 × 150 µm at ×20 objective magnification (accounting for µm/px parameter of different scanning systems). For further details see ref. . Patient cases Three independent patient cohorts and the original training cohort for control purposes were included in the study. Patients in all cohorts represented the full spectrum of PCA stages and Gleason grades from routine practice . Detailed characteristics of cohorts are outlined in Table . Digitization and datasets Six datasets (DS1-6) were generated (Table ). One cohort consisting of 51 patients (1 slide/patient) was digitized using three different scanners resulting in three datasets. For the final analysis, given very high computational costs, a random crop of each dataset with 120,000 patches was generated containing 50,000 patches with tumor tissue, 50,000 patches with nonneoplastic glandular prostate tissue, and 20,000 patches with nonglandular tissue (all properly classified by model at baseline, corresponding to reference F1 Score 1.0 for comparison with analysis in presence of artifacts). Generation of subsets was carried out randomly taking care for roughly similar number of patches per WSI for all three tissue classes. Artifacts Nine different types of artifacts common for routine histopathology practice were computationally generated. These artifacts included focus, elastic deformation, brightness, contrast, dark spots (e.g., dust, cover glass scratches, and other kinds of contamination), synthetic threads overlying tissue, contaminating squamous epithelia, greasy fingerprints on the slide surface, and hematoxylin-eosin (HE) staining scheme (Fig. ). Some of the artefacts (focus, JPEG compression, elastic deformation, brightness, and contrast) were generated through corresponding changes of the image patch at a pixel level, while others (dark spots, synthetic threads, squamous epithelia, and fingerprints) were synthetized through overlaying of the corresponding artifact on the original pixel content, leaving the latter intact. In addition, three image parameters, potentially affecting performance of DL algorithm, such as jpeg compression, rotation of patch, and flipping of patch, were tested (Fig. ). Three types of dark spots, ten variants of synthetic thread location, 20 variants of squamous epithelia, and a sample of oil drop (greasy fingerprints) were extracted from independent representative routine slides to be later overlaid on the single patches. Focus artifacts were generated using Gaussian blur with levels corresponding to 2 px increases in kernel size. Nine different staining schemes (Fig. , Supplementary Fig. ) with different visual perception of staining quality (from poor to good) were saved as representative images from routine slides for stain transfer using Macenko algorithm . Brightness normalization was routinely performed as the initial step of stain transfer. Representative images are provided in Fig. . Pipeline, software The model was initially trained to work with stain normalization via Macenko stain transfer  from one standard staining scheme outlined in Supplementary Fig. . Therefore, stain-normalized versions of datasets were used in all analyses with exception of those used for HE-staining analysis (where nine other staining schemes were used for tests, Fig. , Supplementary Fig. ). In a test pipeline, single artifacts were generated on every single patch of a corresponding dataset, and model classification was carried out on these modified patches to estimate model accuracy in presence of artifacts. Python (version 3.7.7) and Tensorflow (version 2.3) was used for model implementation. Staintools package was used for brightness standardization and Macenko stain transfer. Opencv-python and Pillow packages were used for image manipulations. Monai package (instrument: Rand2DElastic) was used for elastic deformations. Principle component analysis and t-distributed stochastic neighbor embedding (t-SNE) modules from scikit-learn package for Python were used for analysis of similarity of image patch content. For this purpose, convolutional base of the model was disconnected from classification head and connected to three consecutive global average pooling layers to reduce the dimensions of output vector; output of the last convolutional layer with dimensions (8,8,1536) was therefore transformed to a vector (1,1,1536). Principal component analysis was performed using 700 components as first step with further t-SNE using 2 components and 3000 iterations. Visualization was performed using matplotlib package. Gradient-weighted Class Activation Mapping (GradCAM) was implemented as described in  with outputs captured from activations of the last convolutional layer and opencv-python library for heatmap visualization. Hardware Experiments were performed in parallel on two workstations with 2x GPU Nvidia RTX 2080 Ti 11 Gb and 1x GPU Quatro P6000 24 Gb, respectively, as well as using a high performing computing cluster of the University of Cologne equipped with 12 Nvidia V100 32 Gb GPU cards. Statistical analysis Statistical analysis was performed using R version 4.0.3 (The R Foundation for Statistical Computing). Accuracy of the models was estimated using precision, recall, and F1 score metrics. Precision for tumor detection was calculated as TP/(TP + FP), recall as TP/(TP + FN) and F1 score as 2 × Precision × Recall/(Precision + Recall), where TP, FP, and FN are true positive, false positive and false negative patch classification results through model, respectively. Plots were created using ggplot2 and VennDiagram libraries for R.   The DL-based patch-level classification model for PCA detection used in the study was previously trained using a large-high-quality dataset based on the histological slides from The Cancer Genome Atlas (TCGA) PCA cohort containing ~1.5 million patches . Several convolutional network architectures were tested with NASNetLarge and InceptionResNetV2 showing similar highest accuracy. In current implementation the model is based on the completely retrained InceptionResNetV2 architecture with fully connected layer containing 256 units and classification layer for three classes (prostate glandular tissue, nonglandular tissue, and tumor tissue). The patch size is 300 × 300 pixels (px) corresponding to the whole slide image (WSI) region size of ~150 × 150 µm at ×20 objective magnification (accounting for µm/px parameter of different scanning systems). For further details see ref. .   Three independent patient cohorts and the original training cohort for control purposes were included in the study. Patients in all cohorts represented the full spectrum of PCA stages and Gleason grades from routine practice . Detailed characteristics of cohorts are outlined in Table .   Six datasets (DS1-6) were generated (Table ). One cohort consisting of 51 patients (1 slide/patient) was digitized using three different scanners resulting in three datasets. For the final analysis, given very high computational costs, a random crop of each dataset with 120,000 patches was generated containing 50,000 patches with tumor tissue, 50,000 patches with nonneoplastic glandular prostate tissue, and 20,000 patches with nonglandular tissue (all properly classified by model at baseline, corresponding to reference F1 Score 1.0 for comparison with analysis in presence of artifacts). Generation of subsets was carried out randomly taking care for roughly similar number of patches per WSI for all three tissue classes.   Nine different types of artifacts common for routine histopathology practice were computationally generated. These artifacts included focus, elastic deformation, brightness, contrast, dark spots (e.g., dust, cover glass scratches, and other kinds of contamination), synthetic threads overlying tissue, contaminating squamous epithelia, greasy fingerprints on the slide surface, and hematoxylin-eosin (HE) staining scheme (Fig. ). Some of the artefacts (focus, JPEG compression, elastic deformation, brightness, and contrast) were generated through corresponding changes of the image patch at a pixel level, while others (dark spots, synthetic threads, squamous epithelia, and fingerprints) were synthetized through overlaying of the corresponding artifact on the original pixel content, leaving the latter intact. In addition, three image parameters, potentially affecting performance of DL algorithm, such as jpeg compression, rotation of patch, and flipping of patch, were tested (Fig. ). Three types of dark spots, ten variants of synthetic thread location, 20 variants of squamous epithelia, and a sample of oil drop (greasy fingerprints) were extracted from independent representative routine slides to be later overlaid on the single patches. Focus artifacts were generated using Gaussian blur with levels corresponding to 2 px increases in kernel size. Nine different staining schemes (Fig. , Supplementary Fig. ) with different visual perception of staining quality (from poor to good) were saved as representative images from routine slides for stain transfer using Macenko algorithm . Brightness normalization was routinely performed as the initial step of stain transfer. Representative images are provided in Fig. .   The model was initially trained to work with stain normalization via Macenko stain transfer  from one standard staining scheme outlined in Supplementary Fig. . Therefore, stain-normalized versions of datasets were used in all analyses with exception of those used for HE-staining analysis (where nine other staining schemes were used for tests, Fig. , Supplementary Fig. ). In a test pipeline, single artifacts were generated on every single patch of a corresponding dataset, and model classification was carried out on these modified patches to estimate model accuracy in presence of artifacts. Python (version 3.7.7) and Tensorflow (version 2.3) was used for model implementation. Staintools package was used for brightness standardization and Macenko stain transfer. Opencv-python and Pillow packages were used for image manipulations. Monai package (instrument: Rand2DElastic) was used for elastic deformations. Principle component analysis and t-distributed stochastic neighbor embedding (t-SNE) modules from scikit-learn package for Python were used for analysis of similarity of image patch content. For this purpose, convolutional base of the model was disconnected from classification head and connected to three consecutive global average pooling layers to reduce the dimensions of output vector; output of the last convolutional layer with dimensions (8,8,1536) was therefore transformed to a vector (1,1,1536). Principal component analysis was performed using 700 components as first step with further t-SNE using 2 components and 3000 iterations. Visualization was performed using matplotlib package. Gradient-weighted Class Activation Mapping (GradCAM) was implemented as described in  with outputs captured from activations of the last convolutional layer and opencv-python library for heatmap visualization.   Experiments were performed in parallel on two workstations with 2x GPU Nvidia RTX 2080 Ti 11 Gb and 1x GPU Quatro P6000 24 Gb, respectively, as well as using a high performing computing cluster of the University of Cologne equipped with 12 Nvidia V100 32 Gb GPU cards.   Statistical analysis was performed using R version 4.0.3 (The R Foundation for Statistical Computing). Accuracy of the models was estimated using precision, recall, and F1 score metrics. Precision for tumor detection was calculated as TP/(TP + FP), recall as TP/(TP + FN) and F1 score as 2 × Precision × Recall/(Precision + Recall), where TP, FP, and FN are true positive, false positive and false negative patch classification results through model, respectively. Plots were created using ggplot2 and VennDiagram libraries for R.   Dependence of model accuracy on the presence of artifacts Six test datasets were used in this study, each containing 50,000 patches from tumor, 50,000 patches from benign glandular, and 20,000 patches from benign nonglandular prostate tissue classes (fat, stroma, muscle, vessels, etc.), all properly classified by model at baseline. Of those, Dataset 1 (details in Table ) is composed of training dataset patches and included for control purposes. All studied artifacts reduced the accuracy of the pre-trained deep learning model for PCA detection  used in this study (Figs. , , and ). Focus Sixteen focus levels were tested corresponding to increments of 2 pixels in Gaussian blur kernel filter (Fig. ). Even minimal focus deteriorations led to negative changes in model accuracy. Thus, with kernel sizes 1 px (level 1) and 3 px (level 2), still visually percepted as tolerable by pathologists, the number of false negative (FN) tumor patches was 372/50,000 (0.7%) and 2476/50,000 (4.95%), respectively, with progressive fall in accuracy thereafter (Fig. ). JPEG compression JPEG compression of 80% or equivalent is used by many scanner systems as a standard; therefore we used 80% as a baseline reference (F1 score = 1.0) and tested accuracy of the model at a range of compressions (Fig. ). We observed a slight fall in accuracy starting from 75% JPEG compression. However, rather high levels of accuracy (F1 score >0.95) were evident down to 15% compression level with sudden fall thereafter. Flip and rotation Deep learning-based models are known to be sensitive to presentation of information in the image patch. Although not a typical histological artifact, we aimed to test the model classification consistency in case of simple flips and rotations in random directions. These led to minor losses of accuracy compared to baseline, with F1 score >0.98 for flips and >0.976 for random rotations in tested datasets (Fig. ). Elastic deformation Elastic deformation of cells and tissue fibers has a mechanical distortion in its nature. It is a common artifact resulting from tissue cutting and tissue harvesting. We imitated elastic deformations using five sizes of grid cell staying in the visually realistic range of deformation. All types of deformation produced losses of accuracy with resulting F1 score ranging 0.959–0.980 and larger grid cell sizes associated with higher losses (Fig. ). Contrast and brightness Different scanner systems provide a very different quality of the resulting images with regard to color scheme, brightness and contrast. Analysis of patch content similarity shows scanner-related clustering even when the same slides are scanned by different systems (Supplementary Fig. ). Negative and positive ranges of brightness and contrast were tested, with increments/decrements of 0.1 (10%), respectively (Fig. ). Notably, even small changes in contrast and brightness produced losses in accuracy which became more prominent by further relaxation. Upregulation of brightness and contrast had prominent effects in Dataset 5. Further investigation of this unexpected finding revealed that the WSI scanner was calibrated by the operator according to own visual perception of good contrast and brightness of scanned slides. All other datasets were scanned using calibrated manufacturer setup. Dark spots Dark spots are probably the most common artifacts seen in digital pathology. They usually result from dust, scratches, and foreign objects on the surface of the histological slides or under the coverslip. These dark spots are often transparent and usually do not alter the focus. We reproduced 3 types of dark spots (Fig. ). Placement of these artifacts on the tested images was generated randomly. For every type of spot one, two, and three spots overlying original patch were tested. Number of spots was positively correlated with losses of accuracy (Fig. ). Most prominent losses were evident for spot type 3 (the biggest one), especially with several such spots overlying single patches at the same time reducing transparency. Threads Elongated subjects like synthetic threads are occasionally seen in digitized histological slides, usually accompanied by focal focus distortion. We reproduced 10 different positions of a thread/focus distortion with random selection of the position for single patches during tests (Fig. ). This produced severe accuracy losses with F1 score dropped down to 0.92 from the baseline F1 score of 1.0 (Fig. ). Squamous epithelia Squamous epithelia are one of the commonest artifacts resulting from contamination of the prostate biopsy specimen during tissue processing. They lie on the tissue’s surface, are to large extent transparent, and usually do not impair focus quality during scanning (Fig. ). We extracted 20 templates of squamous epithelia consisting of one or more epithelial cells (Fig. ). For each patch, one, two, or three random epithelial templates were overlaid before classification with position selected randomly. Accuracy loss was correlated with the number of overlaid epithelial complexes (Fig. ). We found that even a single squamous cell cluster produced small but non-negligible losses in accuracy in all datasets. Fingerprints (oil drops) Fingerprints are common and appear as small transparent fat drops in digitized histological slides. We extracted a representative fat drop from WSI and overlaid it on original patches during experiments to see how it can affect the classification (Fig. ). The classification accuracy was still high due to transparency of fat drops, however, non-negligible losses were evident (F1 score <0.98; Fig. ). Staining HE-staining heterogeneity or just a visually “bad” staining (too intensive or too weak) is a prominent source of inter- and intra-institutional heterogeneity in histological slides. Our PCA detection model was developed to work with brightness standardization and stain normalization using a reference staining scheme and Macenko algorithm  (Supplementary Fig. ). Baseline F1 score using this scheme is 1.0 in all datasets. We carried out model accuracy tests of native staining (without any normalization) as well as using a stain transfer from 9 visually different staining schemes including strong and weak staining for every single patch from test datasets (Fig. , Supplementary Fig. ). Substantial losses of accuracy were observed in case of staining scheme visually interpreted as “poor” (Fig. ). Additional level of color heterogeneity originates from the fact that every scanning system has a special color “touch” affecting original staining (Supplementary Fig. ). The similarity analysis of image patch content shows that even after Macenko stain transfer the clustering of image patches from different datasets is defined by the scanner system used (Supplementary Fig. ), even if this type of stain normalization allows for high model classification accuracy. Only style/stain transfer using generative adversarial network (GAN) (Supplementary Methods; Supplementary Fig. ) is able to override scanner-specific features and allow similarity clustering of single patches based on the histological content. Artifacts and tissue type The artifacts affect the model accuracy through false positive (FP) or false negative (FN) results. Thus, nonglandular tissue (stroma, fat, muscle, nerves, vessels, etc.) is a nonsignificant source of FP patches, i.e., benign classified as tumor. The greatest number of FP patches in this tissue class was 138/20,000 (0.69%) in Dataset 6 at the 5% JPEG compression level; while in all other scenarios it was <0.08%. Therefore, misclassification occurs mostly in tumor (FN) and benign glandular (FP) classes which were, however, affected in different ways (Fig. , detailed information in Supplementary Figs. and ). JPEG compression and fingerprints resulted in the similar numbers of FP and FN. Elastic deformation and HE staining produced both types of misclassification, however, predilection to FP and FN was depending on grid size and staining scheme used, respectively. Other artifacts also showed some propensity to certain types of misclassifications (Fig. ). Dataset-related heterogeneity Dataset 1 composed of patches from training dataset provided better accuracy in presence of all artifacts, implying that the model still remembers some features inherent for this particular dataset. The trends of misclassification were in general similar for all datasets. Importantly, during flips, rotations, elastic deformation, squamous epithelia, and upregulated contrast artifacts, Datasets 3–5 belonging to the same cohort, but scanned by different systems, showed slightly larger accuracy losses compared to Datasets 2 and 6. Deeper analysis of this phenomenon revealed that Datasets 3–5 showed up to 4 times more false-positive misclassifications compared to Datasets 2 and 6 with approximately the same number of false negative patches. Morphological analysis provides some clues and allows to attribute this to the fact that all benign patches in Datasets 3–5 were from regions immediately adjacent to tumor areas often carrying preneoplastic lesions (high-grade prostatic intraepithelial neoplasia, HGPIN) which contain cytological features of tumor tissue and, apparently, have lower threshold for misclassification. Benign class patches in Dataset 6 and, especially, dataset 2 stemmed to a large extent from completely “benign” slides or slides with a small amount of tumor. This is supported by the fact that baseline probability of being a tumor in glandular tissue class was statistically significant different between Datasets (DS) 2/6 and DS 3–5 (mean probability of being tumor for DS1 0.019, DS3 0.04, DS4 0.045, DS5 0.032, DS6 0.023; for any comparison: paired t -test < 1e−10). Moreover, F1 accuracy curves for single artifacts were often visually stratified by this baseline probability of being a tumor for benign patches. Misclassified patches As all studied datasets demonstrated the same patterns of model accuracy behavior in presence of artifacts, we selected a representative dataset (Dataset 3) for detailed investigation of misclassified patches. In this dataset intersections were evident among misclassified patches for different artifacts (Fig. ). The artifacts that act randomly and generate principally new content in patch (squamous epithelial cell, threads, dark spots) or substantially reduce the amount of existing information (focus, threads, dark spots) tend to show larger numbers of unique misclassified patches (Fig. ). GradCAM-powered analysis of class activation maps showed, however, that this new content is not being used for classification (new features) but rather obfuscating the existing features (Fig. ). However, some artifacts seem to genuinely change the feature constellation or feature prominence in a patch (elastic deformation, HE-staining; Figs. and ). This is supported by the fact that elastic deformations produce the highest numbers of false positive results (Supplementary Fig. ). Focus artifacts are probably the most important artifacts in digital pathology. We carried out similarity analysis using our model as encoder (see Methods) for focus quality-related misclassified patches. The latter were clustered using t-SNE principle and analyzed by the expert genitourinary pathologist (YT) to identify the morphological descriptive features. False negative tumor patches built following recognizable clusters (Fig. ): (1) very low tumor content in patch (single gland or less), (2) well differentiated tumor or carcinoma with pseudohyperplastic features. Among false positive classified benign patches several clusters were evident (Fig. ): (1) inflammation, (2) luminal content of glands, (3) preneoplastic changes (HGPIN), and (4) retraction artifacts (commonly seen in tumor tissue).   Six test datasets were used in this study, each containing 50,000 patches from tumor, 50,000 patches from benign glandular, and 20,000 patches from benign nonglandular prostate tissue classes (fat, stroma, muscle, vessels, etc.), all properly classified by model at baseline. Of those, Dataset 1 (details in Table ) is composed of training dataset patches and included for control purposes. All studied artifacts reduced the accuracy of the pre-trained deep learning model for PCA detection  used in this study (Figs. , , and ).   Sixteen focus levels were tested corresponding to increments of 2 pixels in Gaussian blur kernel filter (Fig. ). Even minimal focus deteriorations led to negative changes in model accuracy. Thus, with kernel sizes 1 px (level 1) and 3 px (level 2), still visually percepted as tolerable by pathologists, the number of false negative (FN) tumor patches was 372/50,000 (0.7%) and 2476/50,000 (4.95%), respectively, with progressive fall in accuracy thereafter (Fig. ).   JPEG compression of 80% or equivalent is used by many scanner systems as a standard; therefore we used 80% as a baseline reference (F1 score = 1.0) and tested accuracy of the model at a range of compressions (Fig. ). We observed a slight fall in accuracy starting from 75% JPEG compression. However, rather high levels of accuracy (F1 score >0.95) were evident down to 15% compression level with sudden fall thereafter.   Deep learning-based models are known to be sensitive to presentation of information in the image patch. Although not a typical histological artifact, we aimed to test the model classification consistency in case of simple flips and rotations in random directions. These led to minor losses of accuracy compared to baseline, with F1 score >0.98 for flips and >0.976 for random rotations in tested datasets (Fig. ).   Elastic deformation of cells and tissue fibers has a mechanical distortion in its nature. It is a common artifact resulting from tissue cutting and tissue harvesting. We imitated elastic deformations using five sizes of grid cell staying in the visually realistic range of deformation. All types of deformation produced losses of accuracy with resulting F1 score ranging 0.959–0.980 and larger grid cell sizes associated with higher losses (Fig. ).   Different scanner systems provide a very different quality of the resulting images with regard to color scheme, brightness and contrast. Analysis of patch content similarity shows scanner-related clustering even when the same slides are scanned by different systems (Supplementary Fig. ). Negative and positive ranges of brightness and contrast were tested, with increments/decrements of 0.1 (10%), respectively (Fig. ). Notably, even small changes in contrast and brightness produced losses in accuracy which became more prominent by further relaxation. Upregulation of brightness and contrast had prominent effects in Dataset 5. Further investigation of this unexpected finding revealed that the WSI scanner was calibrated by the operator according to own visual perception of good contrast and brightness of scanned slides. All other datasets were scanned using calibrated manufacturer setup.   Dark spots are probably the most common artifacts seen in digital pathology. They usually result from dust, scratches, and foreign objects on the surface of the histological slides or under the coverslip. These dark spots are often transparent and usually do not alter the focus. We reproduced 3 types of dark spots (Fig. ). Placement of these artifacts on the tested images was generated randomly. For every type of spot one, two, and three spots overlying original patch were tested. Number of spots was positively correlated with losses of accuracy (Fig. ). Most prominent losses were evident for spot type 3 (the biggest one), especially with several such spots overlying single patches at the same time reducing transparency.   Elongated subjects like synthetic threads are occasionally seen in digitized histological slides, usually accompanied by focal focus distortion. We reproduced 10 different positions of a thread/focus distortion with random selection of the position for single patches during tests (Fig. ). This produced severe accuracy losses with F1 score dropped down to 0.92 from the baseline F1 score of 1.0 (Fig. ).   Squamous epithelia are one of the commonest artifacts resulting from contamination of the prostate biopsy specimen during tissue processing. They lie on the tissue’s surface, are to large extent transparent, and usually do not impair focus quality during scanning (Fig. ). We extracted 20 templates of squamous epithelia consisting of one or more epithelial cells (Fig. ). For each patch, one, two, or three random epithelial templates were overlaid before classification with position selected randomly. Accuracy loss was correlated with the number of overlaid epithelial complexes (Fig. ). We found that even a single squamous cell cluster produced small but non-negligible losses in accuracy in all datasets.   Fingerprints are common and appear as small transparent fat drops in digitized histological slides. We extracted a representative fat drop from WSI and overlaid it on original patches during experiments to see how it can affect the classification (Fig. ). The classification accuracy was still high due to transparency of fat drops, however, non-negligible losses were evident (F1 score <0.98; Fig. ).   HE-staining heterogeneity or just a visually “bad” staining (too intensive or too weak) is a prominent source of inter- and intra-institutional heterogeneity in histological slides. Our PCA detection model was developed to work with brightness standardization and stain normalization using a reference staining scheme and Macenko algorithm  (Supplementary Fig. ). Baseline F1 score using this scheme is 1.0 in all datasets. We carried out model accuracy tests of native staining (without any normalization) as well as using a stain transfer from 9 visually different staining schemes including strong and weak staining for every single patch from test datasets (Fig. , Supplementary Fig. ). Substantial losses of accuracy were observed in case of staining scheme visually interpreted as “poor” (Fig. ). Additional level of color heterogeneity originates from the fact that every scanning system has a special color “touch” affecting original staining (Supplementary Fig. ). The similarity analysis of image patch content shows that even after Macenko stain transfer the clustering of image patches from different datasets is defined by the scanner system used (Supplementary Fig. ), even if this type of stain normalization allows for high model classification accuracy. Only style/stain transfer using generative adversarial network (GAN) (Supplementary Methods; Supplementary Fig. ) is able to override scanner-specific features and allow similarity clustering of single patches based on the histological content.   The artifacts affect the model accuracy through false positive (FP) or false negative (FN) results. Thus, nonglandular tissue (stroma, fat, muscle, nerves, vessels, etc.) is a nonsignificant source of FP patches, i.e., benign classified as tumor. The greatest number of FP patches in this tissue class was 138/20,000 (0.69%) in Dataset 6 at the 5% JPEG compression level; while in all other scenarios it was <0.08%. Therefore, misclassification occurs mostly in tumor (FN) and benign glandular (FP) classes which were, however, affected in different ways (Fig. , detailed information in Supplementary Figs. and ). JPEG compression and fingerprints resulted in the similar numbers of FP and FN. Elastic deformation and HE staining produced both types of misclassification, however, predilection to FP and FN was depending on grid size and staining scheme used, respectively. Other artifacts also showed some propensity to certain types of misclassifications (Fig. ).   Dataset 1 composed of patches from training dataset provided better accuracy in presence of all artifacts, implying that the model still remembers some features inherent for this particular dataset. The trends of misclassification were in general similar for all datasets. Importantly, during flips, rotations, elastic deformation, squamous epithelia, and upregulated contrast artifacts, Datasets 3–5 belonging to the same cohort, but scanned by different systems, showed slightly larger accuracy losses compared to Datasets 2 and 6. Deeper analysis of this phenomenon revealed that Datasets 3–5 showed up to 4 times more false-positive misclassifications compared to Datasets 2 and 6 with approximately the same number of false negative patches. Morphological analysis provides some clues and allows to attribute this to the fact that all benign patches in Datasets 3–5 were from regions immediately adjacent to tumor areas often carrying preneoplastic lesions (high-grade prostatic intraepithelial neoplasia, HGPIN) which contain cytological features of tumor tissue and, apparently, have lower threshold for misclassification. Benign class patches in Dataset 6 and, especially, dataset 2 stemmed to a large extent from completely “benign” slides or slides with a small amount of tumor. This is supported by the fact that baseline probability of being a tumor in glandular tissue class was statistically significant different between Datasets (DS) 2/6 and DS 3–5 (mean probability of being tumor for DS1 0.019, DS3 0.04, DS4 0.045, DS5 0.032, DS6 0.023; for any comparison: paired t -test < 1e−10). Moreover, F1 accuracy curves for single artifacts were often visually stratified by this baseline probability of being a tumor for benign patches.   As all studied datasets demonstrated the same patterns of model accuracy behavior in presence of artifacts, we selected a representative dataset (Dataset 3) for detailed investigation of misclassified patches. In this dataset intersections were evident among misclassified patches for different artifacts (Fig. ). The artifacts that act randomly and generate principally new content in patch (squamous epithelial cell, threads, dark spots) or substantially reduce the amount of existing information (focus, threads, dark spots) tend to show larger numbers of unique misclassified patches (Fig. ). GradCAM-powered analysis of class activation maps showed, however, that this new content is not being used for classification (new features) but rather obfuscating the existing features (Fig. ). However, some artifacts seem to genuinely change the feature constellation or feature prominence in a patch (elastic deformation, HE-staining; Figs. and ). This is supported by the fact that elastic deformations produce the highest numbers of false positive results (Supplementary Fig. ). Focus artifacts are probably the most important artifacts in digital pathology. We carried out similarity analysis using our model as encoder (see Methods) for focus quality-related misclassified patches. The latter were clustered using t-SNE principle and analyzed by the expert genitourinary pathologist (YT) to identify the morphological descriptive features. False negative tumor patches built following recognizable clusters (Fig. ): (1) very low tumor content in patch (single gland or less), (2) well differentiated tumor or carcinoma with pseudohyperplastic features. Among false positive classified benign patches several clusters were evident (Fig. ): (1) inflammation, (2) luminal content of glands, (3) preneoplastic changes (HGPIN), and (4) retraction artifacts (commonly seen in tumor tissue).   Emerging digital transformation of diagnostic pathology provides a possibility for automatization of some pathological tasks . Histological slides, digitized with high resolution, can be effectively analyzed using different computer vision algorithms . DL-based algorithms are especially accurate in this setting and were successfully implemented for tumor recognition, tumor grading, and extracting prognostic and predictive information from histological slides of patients with different types of cancer . In the real world setting histological slides are extremely heterogeneous even within single pathological laboratory and often contain multiple artifacts which can stem from natural reasons (mechanical or cauterization artifacts related to tissue sampling), preprocessing (embedding, fixation, cutting, staining) and digitization steps (focus, foreign objects, color scheme/brightness/contrast, level of image compression etc.) (Fig. ). Most DL algorithms published to date did not explicitly control for effects of artifacts and, therefore, might be biased through the initial selection of higher quality slides for training and testing phases. High accuracy of DL-based algorithms in the presence of artifacts is of utmost importance for their implementation in the clinical workflow. In this study, we employed a recently published DL-based model for PCA detection in histological slides developed on large-high-quality training dataset and validated on two external datasets with high overall accuracy of >98% . We reproduced 12 types of common histological artifacts or situations (Fig. ) and systematically tested how model accuracy may depend on their presence and severity using 6 datasets representing different institutes and different scanning systems (Table ). As it was anticipated, all tested artifacts indeed affected model accuracy as they reduced the amount of information or distorted the information within an image (Figs. and ). Noteworthy, glandular tissue classes (tumor and benign) were the main source of artifacts, while nonglandular tissue contributed much less significantly. Two main misclassification mechanisms (false positive tumor classification for benign tissue and false negative benign classification for tumor regions) differed among tested artifacts (Fig. ) with a slight predominance of false negatives, which are naturally worse and can have poor diagnostic consequences as model fails silently. False-positive misclassifications will be normally highlighted by model and reviewed/controlled by pathologist operator. Several published studies systematically addressed the influence of histological artifacts on performance of DL-based models in diagnostic pathology. A solid basis of evidence is related to focus problem in digital pathology . Most of the studies, however, concentrate on quantitative detection of out-of-focus regions using simple DL-based models trained on synthetic data  or other principles . Kohlberger et al.  investigated accuracy losses of breast cancer metastasis detection model based on publicly available dataset and produced conclusions concordant with the results of our study. The advantage of our study is that we employed a validated model with high accuracy and were able to test it using 6 different datasets digitized with different WSI scanners. Wrong focus is an artifact with a clear mechanism of action through degradation of image features primarily associated with false-negative misclassifications of tumor regions which are dangerous as model fails silently (Fig. , Supplementary Fig. ). Therefore, major digital pathology guidelines (The Royal College of Pathologists , College of American Pathologists ) require no objective threshold for focus quality, and pathologists decide whether the image sharpness is sufficient for unequivocal diagnosis. Our results show that substantial losses of accuracy can occur when the focus quality levels are still visually perceptible as adequate by pathologists (Fig. ). We also provide morphological classification of image regions most susceptible to misclassification in case of focus artifacts (Fig. ). Image compression effects were investigated by several groups showing accuracy deterioration of models for metastatic cancer detection and segmentation, nuclei segmentation, and lymphocyte detection . Image compression is a fully controllable parameter set up manually during scanning (usually at a level of 80% or more). It normally does not undergo any changes after complete and successful validation of digital pathology system. However, lower levels of JPEG compression can be manually set up by WSI operator to reduce needs for a storage space or in research setting. Interestingly, the human eye does not recognize image alteration within a wide range of compression levels (30–90%). The results of the current study and other published studies  show that this might have potential consequences for accuracy of analytical models trained on datasets with lower compression levels, should they be applied to such images. Our findings show that any compression levels under 80% can result in accuracy deterioration and should be avoided. Hematoxylin-eosin staining is a factor which is extremely difficult (or even impossible) to standardize at pre-analytical stage. WSIs from the single laboratory and, especially, from different institutions have very different HE-staining schemes due to multitude of factors, such as variation of stains, reagents and manufacturers, protocols, room conditions, and many more  (Supplementary Fig. ). Additional layer of heterogeneity are color schemes of WSI scanners from different manufactures which imprint every scanned slide . The same slides from one institution digitized by three different scanners (Datasets 3–5) in our study are highly heterogenous in terms of color scheme, brightness and contrast (Supplementary Fig. ), corroborating the results of the recent study by Schmitt et al. . Obviously, staining is a parameter that should be controlled for at time of WSI analysis by diagnostic models. Substantial losses in accuracy dependent on HE-staining quality, brightness, and contrast were evident in our study (Fig. ). Many methods were suggested for stain and color normalization with some of the most popular being Macenko , Vahadane , Bejnordi , and sparse autoencoder-based algorithms . Also, DL-based pix2pix GANs have been extensively studied most recently providing not only stain normalization, but also style transfer . Style transfer using GANs might be a promising approach to tailor model performance to single institutions where it should be implemented. In comparison to stain vector transfer using Macenko principle, pix2pix GANs can allow for a more advanced adaptation to the target domain and elimination of scanner-specific features  (see comparison in Supplementary Fig. ). Other approaches to address this problem are using extensive augmentation of training dataset via color/staining changes , using staining invariant features for model training , and using large training datasets from different institutions to compensate for the necessity of augmentation . To the best of our knowledge, other artifacts investigated in our study (dark spots, overlying squamous epithelia, synthetic threads, elastic deformations, fingerprints) were not addressed systematically in the pathology domain and, therefore, represent the novelty of this study. All these artifacts might result in accuracy deterioration and warrant quality control measures. Several strategies might be implemented to sustain accuracy of DL-based models in context of artifacts. Firstly, implementation of pre-analytical quality control tools (e.g. HistoQC ) is of utmost importance before models are even being applied to WSIs. Secondly, “knowing own models” principle is important to predict in which situations the former can fail and to achieve a balance between additional QC-related computational load and inference speed. Next, augmentation of training dataset using generation of synthetic artifacts can be a reasonable strategy to improve model accuracy. In addition, uncertainty measurements for a model classification might help to identify visual contexts to which the model was not exposed during training, such as artifacts . And lastly, more standardization is warranted for digital pathology systems, particularly in terms of color calibration and image compression levels . This study had certain limitations that require further elaboration. Although our model was trained using high-quality dataset, it is used only for detection of PCA. Generalizability of the results to other tumor types and DL models warrant additional investigations. We used the lowest levels of JPEG compression of 80% as reference. Some authors show that even lower compression levels (>90%) might be necessary for higher accuracy , although, we believe that it is a matter of compression level of images in training dataset (80% in our case). We do not reproduce certain types of artifacts like tissue folds (because of the technical difficulty of generation) and pen marks as the latter are basically not present in diagnostic setting (slides are being scanned before pathologists receive them) and can be easily avoided through slide cleaning before scanning in other situations. While our results are important for DL-based classification algorithms, it is not clear what impact the artifacts will have on non-DL-based computer vision algorithms (not addressed in this work). We believe that our stress-testing pipeline might be a necessary step for any diagnostic model/algorithm in course of clinical validation. While our study was performed entirely on HE-stained slides, we may predict that histological artifacts play the similar detrimental role in more advanced staining techniques, such as histochemistry, immunohistochemistry, and immunofluorescence, particularly multiplexed. Immunostaining and immunofluorescence are widely utilized as objects for DL in pathology and impact of tissue artifacts should be investigated in future studies. In our study, we reproduced twelve types of common histological artifacts or analytical situations. We systematically tested the effects of these artifacts and revealed mechanisms of misclassification using 6 datasets from several institutions, digitized by different scanning systems, using a validated pre-trained model for PCA detection with high accuracy. We here provide evidence that any histological artifact can lead to substantial loss of accuracy in DL model performance. We discuss the strategies for prevention of model accuracy losses in context of the artifacts. Stress-testing of diagnostic models using synthetically generated artifacts might be an important step during clinical validation of DL algorithms.   Supplementary Data   