A unified framework harnessing multi-scale feature ensemble and attention mechanism for gastric polyp and protrusion identification in gastroscope imaging
Gastric polyps and protrusions are prevalent conditions that may elevate stomach cancer risk, particularly due to mutation or enlargement. Polyps arise from inward gastric mucosa growth and are often asymptomatic but can cause pain or bleeding in larger cases. Studies show a 0.46% detection rate with a predominance among females and an average age of 58.1 years . Gastric protrusions, encompassing various pathologies like tumors and vascular anomalies, can be benign or malignant, especially gastrointestinal stromal tumors(GISTs) and neuroendocrine tumors – . Accurate diagnosis and multidisciplinary management are essential for successful treatment. The accurate diagnosis of gastric polyps and protrusions is primarily achieved through endoscopy, but their similar appearances can lead to confusion. Accurate initial diagnosis is essential for treatment success. Additional tests like endoscopic ultrasound and contrast-enhanced CT scans are often needed for differential diagnosis, but their high cost and limited availability in primary hospitals pose challenges. The need for cost-effective diagnostic strategies is urgent. Experienced gastroenterologists can usually differentiate the conditions, but junior doctors and general practitioners may struggle, potentially leading to inappropriate treatment choices and increased risks of recurrence or malignancy. The lack of advanced diagnostic equipment in primary institutions further complicates accurate diagnosis, potentially causing unnecessary treatment and psychological distress, and affecting long-term prognosis – . Deep learning, especially CNNs, has revolutionized gastrointestinal endoscopy by improving disease detection and intervention support . CNNs excel in detecting and classifying colonic polyps, diagnosing Helicobacter pylori without invasive procedures, and identifying early gastric cancer and Barrett’s esophagus dysplasia – . Despite these advances, distinguishing gastric polyps from protrusions using deep learning remains unexplored, highlighting a need for leveraging AI to enhance diagnostic accuracy in this area , – . In our research, we confront the challenge of applying deep learning to medical image analysis, particularly with limited datasets . To mitigate overfitting and enhance generalization, we introduce a novel model, MultiAttentiveScopeNet, integrating multi-layer feature ensemble and attention mechanisms , – . This model adeptly captures multi-scale features and focuses on critical image regions, optimizing its learning from smaller datasets. The MultiAttentiveScopeNet demonstrates its efficacy in digestive endoscopic image analysis, offering a promising approach for medical image analysis tasks and potential for broader clinical applications (Figs. and ). The main contributions of this study are as follows: The MultiAttentiveScopeNet model, which integrates an attention mechanism, enhances both predictive accuracy and model interpretability. This mechanism highlights critical image areas, aiding clinicians in making informed decisions and bolstering the model’s practical clinical applicability. Our model, Multi-layer Feature Ensemble, excels in gastroscopy image analysis by integrating a multi-layer feature ensemble. This strategy captures a comprehensive set of features, enhancing the model’s robustness in processing complex images. The approach ensures a nuanced understanding of image characteristics, crucial for accurate gastroscopy analysis. Our model leverages a multi-class gastroscopy image dataset, constructed through a weakly supervised labeling strategy, to enhance model training and validation. This dataset facilitates precise differentiation among normal, polyp, and protrusion images, crucial for model optimization. Our model, designed to differentiate gastric protrusions from polyps, has shown superior diagnostic precision in head-to-head trials against junior gastroenterologists. This model could be instrumental in supporting primary healthcare institutions and junior doctors in their clinical decision-making processes. The subsequent sections of this article are organized as follows: Section “ ” presents the experimental results along with empirical evaluations; Section “ ” discusses the motivation and innovation behind this study; Section “ ” provides a comprehensive description of the dataset construction, algorithmic framework, and evaluation methods; finally, Section “ ” concludes the paper and outlines future directions of work.   Dataset Table clearly and unambiguously presents the category distribution of the total dataset and its various sub-datasets. These sub-datasets are composed of five folds, each of which includes a training set and a validation set. In addition, the category distribution of the test set is also incorporated. The category distribution of each sub-dataset is detailed, allowing us to precisely grasp the distribution of categories across different datasets. This comprehensive display of category distribution not only aids us in understanding the structure and characteristics of the data but also provides an important reference for subsequent model training and validation. By comparing the category distributions across the sub-datasets, we can ensure that the data distribution is reasonable, thereby guaranteeing reliable and representative results in the later stages of model training and validation. Here, the term “later stages” refers specifically to the deeper layers of the neural network rather than stages in the training process. As the input data progresses through these deeper layers, the model captures more abstract and high-level features, which are crucial for distinguishing between different categories. Throughout the manuscript, any further references to “later stages” should be understood in this context. Model comparison on validation set In this study, we conducted a series of comprehensive comparative experiments to explore and validate the effectiveness of various attention mechanism modules and feature ensemble methods. We experimented with multiple attention mechanism modules, including SE, CBAM, and scSE, while also exploring various feature ensemble methods such as simple multi-layer feature output ensemble and multi-layer feature output ensemble enhanced with FPN , . The SE (Squeeze-and-Excitation) module improves the network’s representational capacity by adaptively recalibrating channel-wise feature responses, thereby emphasizing the most informative channels. CBAM (Convolutional Block Attention Module) refines feature maps by applying both channel and spatial attention, allowing the model to focus on crucial regions and improve feature extraction. Meanwhile, the scSE (Spatial and Channel Squeeze-and-Excitation) module combines spatial and channel attention to enhance spatial details while preserving channel relevance, which aids in distinguishing fine-grained patterns. To further enhance feature extraction, we incorporated the FPN (Feature Pyramid Network) into our ensemble approach, leveraging its ability to combine low-level spatial features with high-level semantic information across different scales, thereby improving multi-scale feature representation. To accurately evaluate the advantages of different method combinations, we designed a variety of experimental schemes, including the use of individual attention mechanism modules, the use of different feature output ensemble methods, and comparisons with the results of the original ResNet50. Through five-fold cross-validation, we conducted an in-depth analysis of the performance of models under various configurations (refer to Fig. ). The results indicate that the model incorporating both the SE module and the simple feature output ensemble method demonstrated superior performance across key evaluation metrics, including Accuracy, Precision, Recall, F1-Score, Matthews Correlation Coefficient (MCC), and Area Under Curve (AUC). These findings provide us with valuable guidance. Based on this, we decided to use the entire training set to retrain our MultiAttentiveScopeNet with this optimized configuration, and we plan to deploy it ultimately. Considering the potential advantages of ensemble learning, we also selected the combination of FPN and scSE, which ranked second in accuracy, for additional training. We aim to integrate it with the best configuration to further enhance the model’s performance in practical applications. Assessment on test set and human–machine comparison This study aims to explore the impact of different attention mechanism modules and feature ensemble methods on the performance of MultiAttentiveScopeNet through experiments. For this purpose, we retrained two different configurations of MultiAttentiveScopeNet using the entire training dataset. One configuration used the SE module and a simple multi-layer feature ensemble method (optimal configuration), while the other used the scSE module and FPN method (suboptimal configuration). After completing the training, we evaluated the performance of these two configurations on the test set to understand the impact of different configurations on model performance. To further optimize model performance, we explored two common enhancement techniques: Test Time Augmentation (TTA) and ensemble learning. TTA involves applying various transformations (e.g., rotations, flips, and shifts) to test images during inference to generate multiple predictions, which are then averaged to improve model robustness and accuracy. Ensemble learning, on the other hand, combines predictions from multiple models to reduce errors and enhance overall performance, thereby increasing result reliability. In addition to evaluating the base models, we assessed the effects of ensemble learning with these configurations and analyzed the performance differences before and after applying TTA within the ensemble learning framework. At the same time, to verify the practicality and accuracy of the model, we compared the diagnostic results of the model with those of two junior doctors (refer to Fig. ). The ROC curve visually presents the performance of the four models in different diagnostic categories (in Fig. .) From Fig. , it is evident that the diagnostic performance of all models surpasses that of the two junior doctors, with the model using ensemble learning and TTA showing the most outstanding performance. This superior performance can be attributed to several factors. First, the models demonstrate consistency and precision in analyzing every image, as they are not subject to fatigue, distractions, or variability in experience levels, which often affect human reviewers, especially in complex or ambiguous cases. Additionally, by leveraging large-scale datasets, the models can recognize subtle patterns and features that may not be immediately apparent to less experienced practitioners, enabling them to detect nuances that might be overlooked by junior doctors. Furthermore, the use of advanced techniques such as Test Time Augmentation (TTA) and ensemble learning enhances prediction accuracy and robustness, resulting in better overall performance compared to human assessments. These combined strengths contribute to the models’ superior diagnostic accuracy as demonstrated in our results. The confusion matrix (refer to Fig. ) provides us with a clear perspective to observe the performance of each model and doctor in recognizing each category. Particularly in diagnosing gastric protrusion, the junior doctors encountered considerable difficulty, whereas our proposed MultiAttentiveScopeNet successfully addressed this issue, demonstrating its superiority. In addition, the radar chart provides us with an intuitive way to compare the performance of each model and the human junior doctors across all evaluation metrics after normalization, and the kappa score heatmap reveals the consistency between the model’s predictions and those of the junior doctors (refer to Fig. ). This not only helps us understand the strengths and weaknesses of each model but also provides us with a clear direction on how to further optimize the model to enhance diagnostic performance in future research.By analyzing the performance of various models using techniques such as GradCAM, t-SNE, and UMAP, we identified which models performed better in distinguishing specific categories (e.g., gastric polyps vs. protrusions). For example, our model were particularly effective in identifying subtle variations indicative of polyps, while models utilizing FPN (Feature Pyramid Network) faced challenges in distinguishing protrusions from polyps due to their multi-scale feature focus, which could introduce noise in areas with homogeneous tissue structures. Activation value dimensionality reduction analysis In experiments involving deep learning and machine learning, visualization techniques are often utilized to interpret and analyze the internal mechanisms and characteristics of models. This is particularly crucial for data with high dimensions, as visualization after dimensionality reduction can offer researchers an abundance of valuable information. In this study, we have specifically chosen two widely acclaimed dimensionality reduction visualization methods: T-SNE and UMAP (refer to Fig. ). T-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear dimensionality reduction technique that focuses on preserving the local structure of data points by minimizing the divergence between probability distributions of pairwise similarities. This makes it particularly effective for visualizing high-dimensional data in a low-dimensional space, typically 2D, thereby enabling better interpretation of cluster patterns. On the other hand, UMAP (Uniform Manifold Approximation and Projection) aims to preserve both global and local structures within the data, making it a powerful tool for identifying patterns and clusters, especially in large datasets. Our goal is to more accurately represent the output characteristics of the MultiAttentiveScopeNet at various stages. To our delight, through the visual representations provided by these two methods, we were able to observe an intriguing phenomenon: the model’s ability to distinguish between categories in the later stages significantly surpassed that in the earlier stages. This result not only lends strong support to our previous decisions but also solidifies our belief that giving greater weight to the later stages of the model is the right choice. By assigning higher weights to these deeper layers, the model can more effectively focus on learning the complex patterns necessary for distinguishing between categories. This adjustment significantly improved classification performance on the test set and optimized the model’s utilization of critical features. This finding highlights the importance of considering model depth and complexity when designing weight allocation strategies, ensuring that the most important layers are prioritized to enhance diagnostic accuracy. Model interpretability analysis In contemporary deep learning research, interpreting and analyzing models is an indispensable step. This is especially true for complex deep networks, as a deep understanding of their working mechanisms is crucial for further optimization and adjustment of the models. Therefore, we have chosen to analyze using GradCAM, a technique widely embraced by the academic community (refer to Figs. and ). A series of experimental validations revealed that when the model successfully captures the key features in the data, its predictions tend to be more accurate. Further observation showed that as the depth of the model increases, the class activation maps and their outputs begin to exhibit clear characteristics of weakly supervised localization. This provides us with clues on how the model gradually focuses on the key information in the data. In contrast, the earlier stages of the model tend to focus more on regions with lower semantic significance, which could be a major reason why their predictive performance is not as strong as that of the later stages. These observations not only offer us valuable lessons but also validate that our previous decisions regarding model weight allocation were indeed correct.   Table clearly and unambiguously presents the category distribution of the total dataset and its various sub-datasets. These sub-datasets are composed of five folds, each of which includes a training set and a validation set. In addition, the category distribution of the test set is also incorporated. The category distribution of each sub-dataset is detailed, allowing us to precisely grasp the distribution of categories across different datasets. This comprehensive display of category distribution not only aids us in understanding the structure and characteristics of the data but also provides an important reference for subsequent model training and validation. By comparing the category distributions across the sub-datasets, we can ensure that the data distribution is reasonable, thereby guaranteeing reliable and representative results in the later stages of model training and validation. Here, the term “later stages” refers specifically to the deeper layers of the neural network rather than stages in the training process. As the input data progresses through these deeper layers, the model captures more abstract and high-level features, which are crucial for distinguishing between different categories. Throughout the manuscript, any further references to “later stages” should be understood in this context.   In this study, we conducted a series of comprehensive comparative experiments to explore and validate the effectiveness of various attention mechanism modules and feature ensemble methods. We experimented with multiple attention mechanism modules, including SE, CBAM, and scSE, while also exploring various feature ensemble methods such as simple multi-layer feature output ensemble and multi-layer feature output ensemble enhanced with FPN , . The SE (Squeeze-and-Excitation) module improves the network’s representational capacity by adaptively recalibrating channel-wise feature responses, thereby emphasizing the most informative channels. CBAM (Convolutional Block Attention Module) refines feature maps by applying both channel and spatial attention, allowing the model to focus on crucial regions and improve feature extraction. Meanwhile, the scSE (Spatial and Channel Squeeze-and-Excitation) module combines spatial and channel attention to enhance spatial details while preserving channel relevance, which aids in distinguishing fine-grained patterns. To further enhance feature extraction, we incorporated the FPN (Feature Pyramid Network) into our ensemble approach, leveraging its ability to combine low-level spatial features with high-level semantic information across different scales, thereby improving multi-scale feature representation. To accurately evaluate the advantages of different method combinations, we designed a variety of experimental schemes, including the use of individual attention mechanism modules, the use of different feature output ensemble methods, and comparisons with the results of the original ResNet50. Through five-fold cross-validation, we conducted an in-depth analysis of the performance of models under various configurations (refer to Fig. ). The results indicate that the model incorporating both the SE module and the simple feature output ensemble method demonstrated superior performance across key evaluation metrics, including Accuracy, Precision, Recall, F1-Score, Matthews Correlation Coefficient (MCC), and Area Under Curve (AUC). These findings provide us with valuable guidance. Based on this, we decided to use the entire training set to retrain our MultiAttentiveScopeNet with this optimized configuration, and we plan to deploy it ultimately. Considering the potential advantages of ensemble learning, we also selected the combination of FPN and scSE, which ranked second in accuracy, for additional training. We aim to integrate it with the best configuration to further enhance the model’s performance in practical applications.   This study aims to explore the impact of different attention mechanism modules and feature ensemble methods on the performance of MultiAttentiveScopeNet through experiments. For this purpose, we retrained two different configurations of MultiAttentiveScopeNet using the entire training dataset. One configuration used the SE module and a simple multi-layer feature ensemble method (optimal configuration), while the other used the scSE module and FPN method (suboptimal configuration). After completing the training, we evaluated the performance of these two configurations on the test set to understand the impact of different configurations on model performance. To further optimize model performance, we explored two common enhancement techniques: Test Time Augmentation (TTA) and ensemble learning. TTA involves applying various transformations (e.g., rotations, flips, and shifts) to test images during inference to generate multiple predictions, which are then averaged to improve model robustness and accuracy. Ensemble learning, on the other hand, combines predictions from multiple models to reduce errors and enhance overall performance, thereby increasing result reliability. In addition to evaluating the base models, we assessed the effects of ensemble learning with these configurations and analyzed the performance differences before and after applying TTA within the ensemble learning framework. At the same time, to verify the practicality and accuracy of the model, we compared the diagnostic results of the model with those of two junior doctors (refer to Fig. ). The ROC curve visually presents the performance of the four models in different diagnostic categories (in Fig. .) From Fig. , it is evident that the diagnostic performance of all models surpasses that of the two junior doctors, with the model using ensemble learning and TTA showing the most outstanding performance. This superior performance can be attributed to several factors. First, the models demonstrate consistency and precision in analyzing every image, as they are not subject to fatigue, distractions, or variability in experience levels, which often affect human reviewers, especially in complex or ambiguous cases. Additionally, by leveraging large-scale datasets, the models can recognize subtle patterns and features that may not be immediately apparent to less experienced practitioners, enabling them to detect nuances that might be overlooked by junior doctors. Furthermore, the use of advanced techniques such as Test Time Augmentation (TTA) and ensemble learning enhances prediction accuracy and robustness, resulting in better overall performance compared to human assessments. These combined strengths contribute to the models’ superior diagnostic accuracy as demonstrated in our results. The confusion matrix (refer to Fig. ) provides us with a clear perspective to observe the performance of each model and doctor in recognizing each category. Particularly in diagnosing gastric protrusion, the junior doctors encountered considerable difficulty, whereas our proposed MultiAttentiveScopeNet successfully addressed this issue, demonstrating its superiority. In addition, the radar chart provides us with an intuitive way to compare the performance of each model and the human junior doctors across all evaluation metrics after normalization, and the kappa score heatmap reveals the consistency between the model’s predictions and those of the junior doctors (refer to Fig. ). This not only helps us understand the strengths and weaknesses of each model but also provides us with a clear direction on how to further optimize the model to enhance diagnostic performance in future research.By analyzing the performance of various models using techniques such as GradCAM, t-SNE, and UMAP, we identified which models performed better in distinguishing specific categories (e.g., gastric polyps vs. protrusions). For example, our model were particularly effective in identifying subtle variations indicative of polyps, while models utilizing FPN (Feature Pyramid Network) faced challenges in distinguishing protrusions from polyps due to their multi-scale feature focus, which could introduce noise in areas with homogeneous tissue structures.   In experiments involving deep learning and machine learning, visualization techniques are often utilized to interpret and analyze the internal mechanisms and characteristics of models. This is particularly crucial for data with high dimensions, as visualization after dimensionality reduction can offer researchers an abundance of valuable information. In this study, we have specifically chosen two widely acclaimed dimensionality reduction visualization methods: T-SNE and UMAP (refer to Fig. ). T-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear dimensionality reduction technique that focuses on preserving the local structure of data points by minimizing the divergence between probability distributions of pairwise similarities. This makes it particularly effective for visualizing high-dimensional data in a low-dimensional space, typically 2D, thereby enabling better interpretation of cluster patterns. On the other hand, UMAP (Uniform Manifold Approximation and Projection) aims to preserve both global and local structures within the data, making it a powerful tool for identifying patterns and clusters, especially in large datasets. Our goal is to more accurately represent the output characteristics of the MultiAttentiveScopeNet at various stages. To our delight, through the visual representations provided by these two methods, we were able to observe an intriguing phenomenon: the model’s ability to distinguish between categories in the later stages significantly surpassed that in the earlier stages. This result not only lends strong support to our previous decisions but also solidifies our belief that giving greater weight to the later stages of the model is the right choice. By assigning higher weights to these deeper layers, the model can more effectively focus on learning the complex patterns necessary for distinguishing between categories. This adjustment significantly improved classification performance on the test set and optimized the model’s utilization of critical features. This finding highlights the importance of considering model depth and complexity when designing weight allocation strategies, ensuring that the most important layers are prioritized to enhance diagnostic accuracy.   In contemporary deep learning research, interpreting and analyzing models is an indispensable step. This is especially true for complex deep networks, as a deep understanding of their working mechanisms is crucial for further optimization and adjustment of the models. Therefore, we have chosen to analyze using GradCAM, a technique widely embraced by the academic community (refer to Figs. and ). A series of experimental validations revealed that when the model successfully captures the key features in the data, its predictions tend to be more accurate. Further observation showed that as the depth of the model increases, the class activation maps and their outputs begin to exhibit clear characteristics of weakly supervised localization. This provides us with clues on how the model gradually focuses on the key information in the data. In contrast, the earlier stages of the model tend to focus more on regions with lower semantic significance, which could be a major reason why their predictive performance is not as strong as that of the later stages. These observations not only offer us valuable lessons but also validate that our previous decisions regarding model weight allocation were indeed correct.   Gastric polyps and protrusions, common gastric conditions, pose a health risk due to their potential to induce malignancies. Traditional endoscopic examination is the primary diagnostic method but is prone to misjudgment due to their similar appearances. Advanced diagnostic techniques like ultrasound gastroscopy and enhanced CT scans are effective but costly and not widely accessible. To address this, our study introduces a deep learning model, MultiAttentiveScopeNet, leveraging multi-scale feature ensemble and attention mechanisms to differentiate these conditions effectively. This model aims to bridge the research gap and provide a more accessible diagnostic tool. In our pursuit to enhance medical image recognition, particularly in gastroscopy, we developed MultiAttentiveScopeNet, which integrates attention mechanisms and multi-scale feature ensembles to optimize traditional CNNs for image classification. By focusing on critical image areas, it mitigates overfitting and ensures comprehensive feature extraction, leveraging outputs from each deep learning stage with weighted predictions to capture both detailed and holistic features. However, while exploring different configurations, we found that incorporating FPN was suboptimal for our specific research. Although effective for object detection and segmentation tasks requiring multi-scale feature representation, its hierarchical feature fusion introduced unnecessary complexity and noise when applied to gastroscope images focused on subtle, localized features in homogeneous regions. This diluted the model’s ability to extract fine-grained details, leading to diminished performance. In contrast, we found TTA significantly improved model robustness by applying transformations like flips, rotations, and scaling to test images, generating multiple predictions that were averaged for more stable results. By combining these optimization techniques, along with ensemble learning, our model demonstrates improved diagnostic accuracy and robustness, making it well-suited for reliable clinical deployment. To evaluate our medical image recognition model, a comparative study was conducted with junior gastroenterologists. This experiment aimed to gauge the potential of deep learning in under-resourced medical settings. The task involved complex multi-class classification, challenging both the doctors and our model. Results indicated that while junior doctors’ performance matched our model in identifying normal and polyp images, they fell short in recognizing gastric protrusions, a critical diagnostic need. The Kappa score highlighted inconsistencies in their diagnoses, underscoring the need for standardized diagnostic tools. Our model, by integrating deep learning, offers a reliable diagnostic aid, reducing diagnostic variance and enhancing patient trust in medical services. Deep learning models, despite their remarkable performance, often suffer from a lack of transparency, known as the ‘black box’ problem. To address this, our study conducted an in-depth interpretability analysis focusing on the model’s intermediate layers, employing t-SNE, UMAP, and GradCAM to enhance transparency and build trust with medical professionals. GradCAM was specifically chosen because it highlights the regions in input images that most influence the model’s predictions, providing a visual explanation of what the model is “looking at” when making diagnostic decisions. This allows clinicians to verify that the model’s focus aligns with clinically relevant features, thereby increasing confidence in its reliability-an essential factor in clinical settings. Additionally, t-SNE and UMAP were used to visualize high-dimensional activation values in a lower-dimensional space, illustrating how the model clusters different categories based on learned features. This helps demonstrate the robustness of the model’s internal representations by confirming that similar cases are grouped correctly. Together, these methods promote greater transparency, making the model’s decision-making process more understandable and trustworthy for medical professionals, which is vital for the adoption of AI in clinical practice. Our findings indicate that later layers are pivotal for capturing semantic information, while earlier layers focus on detail and texture. This supports our strategy of multi-scale feature integration, with the output layer sections showing enhanced semantic properties crucial for classification. Our model, even with limited annotations, demonstrated robust localization capabilities, crucial for AI-assisted diagnosis.To further enhance model performance and reduce misclassification, we plan to implement a multi-stage task approach. Specifically, we propose first employing an object detection model to accurately identify and localize lesion regions. Once the lesions are detected, a subsequent classification model will be used to perform fine-grained categorization of the identified lesions (e.g., distinguishing between polyps and protrusions). This two-step process can help prevent the model from focusing on irrelevant regions of the image, thereby reducing noise and enhancing diagnostic accuracy. By isolating the regions of interest, we aim to optimize the model’s ability to differentiate subtle characteristics that are critical for accurate classification. Although our study has yielded significant advancements, we recognize the need for further refinement. The current dataset’s origin from a single medical center may limit the model’s adaptability. We anticipate that expanding our data sources to include multiple medical centers will not only address these constraints but also bolster the model’s robustness. Through this enriched dataset, we aim to achieve a comprehensive understanding of the model’s performance across varied clinical settings, thereby enhancing its practical utility in clinical decision-making.   Ethical approval and consent All experiments involving human participants and/or the use of human tissue samples were conducted in strict accordance with the ethical standards of the institutional and national research committee. The study was approved by the GUANGXI ZHUANG AUTONOMOUS REGION PEOPLE’S HOSPITAL Medical Ethics Committee, with the approval number [KY-ZC-2023-227]. Informed consent was obtained from all individual participants included in the study. The methods were carried out in accordance with the approved guidelines and regulations. Dataset and preprocessing The dataset for this study was sourced from the Gastrointestinal Endoscopy Center of the People’s Hospital of Guangxi Zhuang Autonomous Region, spanning gastroscopy images from January 2021 to August 2023. It includes 2379 normal, 1939 gastric protrusion, and 1988 gastric polyp images. Two experienced gastroenterologists reviewed the images to ensure quality, excluding poor ones. The images were divided into training, validation, and testing sets to evaluate the deep-learning model’s performance accurately. Model training In the realm of gastroscopy image analysis, rapid and precise differentiation among normal, gastric protrusion, and polyp images is crucial. Our model leverages the ResNet50 architecture, a well-established CNN, as its foundation. Initially, we pre-trained the model on the extensive ImageNet dataset, which comprises 1000 categories and over 1.28 million images. This pre-training phase endowed the model with a solid initial parameter set, accelerating convergence towards optimal performance. Post pre-training, we transferred the learned weights (excluding the final fully connected layer) to our model, utilizing them as starting parameters for further training and fine-tuning on our gastroscopy dataset. This transfer learning strategy , not only hastened convergence and improved classification accuracy but also curbed overfitting issues. In the design of contemporary deep learning models, particularly for medical image classification tasks, effectively extracting and utilizing features within the images is one of the critical challenges. To address this issue, we have innovatively improved the backbone of our model by integrating attention mechanisms and multi-feature layer ensemble prediction techniques, resulting in the successful construction of a model that not only preserves a wealth of features but also highlights the essential ones. We have named this model MultiAttentiveScopeNet. The primary enhancements of this new model include: The addition of attention modules after each of the four stages of the backbone, aims to extract vital features from each stage and ensure the model captures important feature information at all levels. To standardize the channel numbers of feature maps outputted from different stages, we utilized convolutional layers with a kernel size of 1 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$$$ 1, successfully unifying the channel numbers to 256. This approach reduces computational burden while maintaining feature diversity. After standardizing the channel numbers, we processed the output using convolutional layers with a kernel size of 3 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$$$ 3, a stride of 1, and padding of 1, enabling effective spatial information interaction among features from different stages and thus enriching the spatial information. Subsequently, we performed global average pooling and flattening operations on the output feature maps of each stage, resulting in four feature vectors of length 256. This step simplifies the subsequent processing workflow while retaining the main feature information from each stage. Using a classification head with shared parameters, we obtained the prediction results for each stage. This classification head is a fully connected layer, accepting input vectors of length 256 and outputting vectors of length 3, effectively transforming feature vectors into classification predictions. Finally, we assigned different weight coefficients (1, 2, 3, 4 in order) to each stage and calculated the weighted sum of the prediction results to obtain the final prediction. Higher weights were given to later stages as they possess stronger semantics, benefiting the classification task more substantially. This design approach ensures that the model effectively enhances classification accuracy and overall performance when handling gastroscopy image classification tasks. To enhance the model’s capability for representation learning when processing complex data, we have made key improvements to its head structure. Specifically, we introduced a novel parallel adapter module built upon the existing structure (refer to Fig. ). The primary purpose of this module is to augment the model’s ability to capture and comprehend data features through the improved structure, thereby demonstrating superior performance across various data types. In our study, we meticulously allocated 20% of the dataset for model evaluation and human–computer comparison, with the remaining 80% reserved for training. To optimize model configuration, we randomly divided the training set into five subsets, using one as the validation set and the others for training. This process was repeated five times, each with a different validation subset. The final model was retrained using the entire training set and tested on the test set, as depicted in Fig. . We employed independent random proportional splitting to ensure unbiased sample distribution across subsets, enhancing the reliability of our results. Evaluation of the model and the diagnostic performance of human doctors The MultiAttentiveScopeNet model was rigorously tested using a diverse gastroscopy image dataset. The test set, divided during the initial phase, included 476 normal gastroscope images from 176 patients, 388 gastric protrusion images from 340 patients, and 398 gastric polyp images from 342 patients. This comprehensive dataset enabled an objective assessment of the model’s capability in identifying various gastric conditions. In this study, rigorous experimental protocols were established to ensure objective outcomes. The two junior gastroenterologists classified the test set images by following a standardized protocol, evaluating each image solely based on visual diagnostic criteria commonly used in gastrointestinal endoscopy, such as mucosal surface patterns, lesion morphology, and the presence of abnormal protrusions or polyps. Notably, they conducted this assessment without access to any additional clinical information or prior diagnoses to ensure an unbiased evaluation. The test set comprised 1,262 endoscopic images, selected to represent a balanced distribution across three categories: Class 0 (“Normal”), Class 1 (“Protrusion”), and Class 2 (“Polyp”). Both gastroenterologists independently reviewed and classified all the images, with their assessments recorded as 0, 1, or 2, aligning with the categories used by the algorithm. Evaluation metrics In this study, we utilized a range of evaluation metrics to conduct a comprehensive performance assessment of the model. These metrics include Accuracy, Precision, Recall, F1-Score, Matthews Correlation Coefficient (MCC) , , and Area Under the Receiver Operating Characteristic curve (AUC). The formulas for these metrics are as follows: 1 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Accuracy}= &  +  {TN}}{ {TP} +  {TN} +  {FP} +  {FN}} $$ 2 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Precision}= & }{ {TP} +  {FP}} $$ 3 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Recall}= & }{ {TP} +  {FN}} $$ 4 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {F1-Score}= &    {Recall}}{ {Precision} +  {Recall}} $$ 5 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {MCC}= &    {TN} -  {FP}   {FN}}{ +  {FP})( {TP} +  {FN})( {TN} +  {FP})( {TN} +  {FN})}} $$ 6 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {AUC}= &  _{0}^{1} TPR( {FPR}^{-1}(x)) dx $$ In these formulas, TP, TN, FP, and FN respectively stand for True Positives, True Negatives, False Positives, and False Negatives. TPR represents the True Positive Rate (which is the same as Recall), and [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$ {FPR}^{-1}$$ is the inverse of the False Positive Rate. All of these evaluation metrics are calculated in two ways: at the micro and macro levels. On the micro level, the TP, TN, FP, and FN of all categories are combined to calculate a unified evaluation metric. On the macro level, we calculate the evaluation metrics for each category individually and then take the average, to assess the model’s average performance across different categories. Dimensionality reduction visualization of activation values and analysis of predictive result interpretability In our study, the model analysis segment plays a crucial role. To fully understand and evaluate the performance of the model, we have adopted a comprehensive analysis strategy. Initially, we focused on the output of different feature layers, performing global average pooling operations to transform the resulting 256-dimensional feature vectors into a 2-dimensional space, thereby facilitating a visual representation of the feature layer outputs. This step aims to intuitively observe and assess the distinguishing ability of the activation values of different feature layers for various categories. To minimize the randomness and bias that might arise from using a single dimensionality reduction method, we selected two well-established algorithms: T-SNE , and UMAP , . We intended to provide a more objective and comprehensive assessment of the distinguishing abilities of the activation values of the feature layer outputs for different object categories by visualizing them through these two dimensionality reduction algorithms. In addition to analyzing the model’s ability to distinguish between different categories of images from a macro perspective, we also focused on the interpretability analysis of individual samples. Using the widely verified and applied GradCAM algorithm , , we displayed the class activation maps for individual samples at different stages, allowing for an intuitive analysis of the rationality and reliability of the model’s inference results. In this way, we were able to evaluate the performance of the model from a macro perspective while also gaining a deeper understanding of the model’s working mechanisms and inference processes at a micro level, providing valuable references and foundations for subsequent model optimization and improvement.   All experiments involving human participants and/or the use of human tissue samples were conducted in strict accordance with the ethical standards of the institutional and national research committee. The study was approved by the GUANGXI ZHUANG AUTONOMOUS REGION PEOPLE’S HOSPITAL Medical Ethics Committee, with the approval number [KY-ZC-2023-227]. Informed consent was obtained from all individual participants included in the study. The methods were carried out in accordance with the approved guidelines and regulations.   The dataset for this study was sourced from the Gastrointestinal Endoscopy Center of the People’s Hospital of Guangxi Zhuang Autonomous Region, spanning gastroscopy images from January 2021 to August 2023. It includes 2379 normal, 1939 gastric protrusion, and 1988 gastric polyp images. Two experienced gastroenterologists reviewed the images to ensure quality, excluding poor ones. The images were divided into training, validation, and testing sets to evaluate the deep-learning model’s performance accurately.   In the realm of gastroscopy image analysis, rapid and precise differentiation among normal, gastric protrusion, and polyp images is crucial. Our model leverages the ResNet50 architecture, a well-established CNN, as its foundation. Initially, we pre-trained the model on the extensive ImageNet dataset, which comprises 1000 categories and over 1.28 million images. This pre-training phase endowed the model with a solid initial parameter set, accelerating convergence towards optimal performance. Post pre-training, we transferred the learned weights (excluding the final fully connected layer) to our model, utilizing them as starting parameters for further training and fine-tuning on our gastroscopy dataset. This transfer learning strategy , not only hastened convergence and improved classification accuracy but also curbed overfitting issues. In the design of contemporary deep learning models, particularly for medical image classification tasks, effectively extracting and utilizing features within the images is one of the critical challenges. To address this issue, we have innovatively improved the backbone of our model by integrating attention mechanisms and multi-feature layer ensemble prediction techniques, resulting in the successful construction of a model that not only preserves a wealth of features but also highlights the essential ones. We have named this model MultiAttentiveScopeNet. The primary enhancements of this new model include: The addition of attention modules after each of the four stages of the backbone, aims to extract vital features from each stage and ensure the model captures important feature information at all levels. To standardize the channel numbers of feature maps outputted from different stages, we utilized convolutional layers with a kernel size of 1 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$$$ 1, successfully unifying the channel numbers to 256. This approach reduces computational burden while maintaining feature diversity. After standardizing the channel numbers, we processed the output using convolutional layers with a kernel size of 3 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$$$ 3, a stride of 1, and padding of 1, enabling effective spatial information interaction among features from different stages and thus enriching the spatial information. Subsequently, we performed global average pooling and flattening operations on the output feature maps of each stage, resulting in four feature vectors of length 256. This step simplifies the subsequent processing workflow while retaining the main feature information from each stage. Using a classification head with shared parameters, we obtained the prediction results for each stage. This classification head is a fully connected layer, accepting input vectors of length 256 and outputting vectors of length 3, effectively transforming feature vectors into classification predictions. Finally, we assigned different weight coefficients (1, 2, 3, 4 in order) to each stage and calculated the weighted sum of the prediction results to obtain the final prediction. Higher weights were given to later stages as they possess stronger semantics, benefiting the classification task more substantially. This design approach ensures that the model effectively enhances classification accuracy and overall performance when handling gastroscopy image classification tasks. To enhance the model’s capability for representation learning when processing complex data, we have made key improvements to its head structure. Specifically, we introduced a novel parallel adapter module built upon the existing structure (refer to Fig. ). The primary purpose of this module is to augment the model’s ability to capture and comprehend data features through the improved structure, thereby demonstrating superior performance across various data types. In our study, we meticulously allocated 20% of the dataset for model evaluation and human–computer comparison, with the remaining 80% reserved for training. To optimize model configuration, we randomly divided the training set into five subsets, using one as the validation set and the others for training. This process was repeated five times, each with a different validation subset. The final model was retrained using the entire training set and tested on the test set, as depicted in Fig. . We employed independent random proportional splitting to ensure unbiased sample distribution across subsets, enhancing the reliability of our results.   The MultiAttentiveScopeNet model was rigorously tested using a diverse gastroscopy image dataset. The test set, divided during the initial phase, included 476 normal gastroscope images from 176 patients, 388 gastric protrusion images from 340 patients, and 398 gastric polyp images from 342 patients. This comprehensive dataset enabled an objective assessment of the model’s capability in identifying various gastric conditions. In this study, rigorous experimental protocols were established to ensure objective outcomes. The two junior gastroenterologists classified the test set images by following a standardized protocol, evaluating each image solely based on visual diagnostic criteria commonly used in gastrointestinal endoscopy, such as mucosal surface patterns, lesion morphology, and the presence of abnormal protrusions or polyps. Notably, they conducted this assessment without access to any additional clinical information or prior diagnoses to ensure an unbiased evaluation. The test set comprised 1,262 endoscopic images, selected to represent a balanced distribution across three categories: Class 0 (“Normal”), Class 1 (“Protrusion”), and Class 2 (“Polyp”). Both gastroenterologists independently reviewed and classified all the images, with their assessments recorded as 0, 1, or 2, aligning with the categories used by the algorithm.   In this study, we utilized a range of evaluation metrics to conduct a comprehensive performance assessment of the model. These metrics include Accuracy, Precision, Recall, F1-Score, Matthews Correlation Coefficient (MCC) , , and Area Under the Receiver Operating Characteristic curve (AUC). The formulas for these metrics are as follows: 1 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Accuracy}= &  +  {TN}}{ {TP} +  {TN} +  {FP} +  {FN}} $$ 2 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Precision}= & }{ {TP} +  {FP}} $$ 3 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {Recall}= & }{ {TP} +  {FN}} $$ 4 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {F1-Score}= &    {Recall}}{ {Precision} +  {Recall}} $$ 5 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {MCC}= &    {TN} -  {FP}   {FN}}{ +  {FP})( {TP} +  {FN})( {TN} +  {FP})( {TN} +  {FN})}} $$ 6 [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$  {AUC}= &  _{0}^{1} TPR( {FPR}^{-1}(x)) dx $$ In these formulas, TP, TN, FP, and FN respectively stand for True Positives, True Negatives, False Positives, and False Negatives. TPR represents the True Positive Rate (which is the same as Recall), and [12pt]{minimal} 				 				  				  				  				 				 				 				 				$$ {FPR}^{-1}$$ is the inverse of the False Positive Rate. All of these evaluation metrics are calculated in two ways: at the micro and macro levels. On the micro level, the TP, TN, FP, and FN of all categories are combined to calculate a unified evaluation metric. On the macro level, we calculate the evaluation metrics for each category individually and then take the average, to assess the model’s average performance across different categories.   In our study, the model analysis segment plays a crucial role. To fully understand and evaluate the performance of the model, we have adopted a comprehensive analysis strategy. Initially, we focused on the output of different feature layers, performing global average pooling operations to transform the resulting 256-dimensional feature vectors into a 2-dimensional space, thereby facilitating a visual representation of the feature layer outputs. This step aims to intuitively observe and assess the distinguishing ability of the activation values of different feature layers for various categories. To minimize the randomness and bias that might arise from using a single dimensionality reduction method, we selected two well-established algorithms: T-SNE , and UMAP , . We intended to provide a more objective and comprehensive assessment of the distinguishing abilities of the activation values of the feature layer outputs for different object categories by visualizing them through these two dimensionality reduction algorithms. In addition to analyzing the model’s ability to distinguish between different categories of images from a macro perspective, we also focused on the interpretability analysis of individual samples. Using the widely verified and applied GradCAM algorithm , , we displayed the class activation maps for individual samples at different stages, allowing for an intuitive analysis of the rationality and reliability of the model’s inference results. In this way, we were able to evaluate the performance of the model from a macro perspective while also gaining a deeper understanding of the model’s working mechanisms and inference processes at a micro level, providing valuable references and foundations for subsequent model optimization and improvement.   This study presents a deep learning model adept at identifying gastric polyps and protrusions, leveraging attention mechanisms, multi-scale feature ensemble, TTA, and ensemble learning. The model outperforms junior gastroenterologists in diagnosis, highlighting its potential for clinical application. However, further multi-center validation is needed to enhance its generalizability.   