Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy
Endoscopy is an essential tool in the diagnosis and treatment of gastrointestinal diseases . However, monocular endoscopes based on 2D imaging may compromise surgical and diagnostic tasks due to the lack of depth perception and detailed spatial information. Detailed 3D surgical scene reconstruction technology with enhanced depth perception can not only improve the accuracy and quality of surgery, but also play an important role in downstream tasks such as medical education and training, and surgical planning. However, robust algorithms are severely lacking in medical environments, where capturing high-fidelity diseased tissue is of the utmost importance. Therefore, developing a new method for reconstructing deformable tissue scenes is a pivotal technological and intellectual challenge in the field of Computer-Aided Diagnosis (CAD). In early research on medical image reconstruction, discrete representation methods (such as point clouds and grids) were often used to model and reconstruct three-dimensional structures. The advent of deep learning has shown great promise in many other fields. Researchers have attempted to directly apply deep learning methods to discrete 3D data representations and have developed end-to-end 3D depth reconstruction algorithms  based on point clouds and grids, which can be used for various types of medical imaging data. However, in endoscopic scenes, traditional 3D discrete representation methods struggle to reconstruct relatively complete geometric structures of scenes. With the great success of NeRFs in scene reconstruction using continuous representations , several research works are gradually introducing NeRF-based implicit modeling methods into the 3D reconstruction of endoscopic images. For example, the development of EndoNeRF  and Endosurf  has attracted widespread attention due to their significant progress in enhancing the accuracy and detail fidelity of endoscopic image reconstruction. However, these methods primarily handle stereo endoscopic image sequences with a fixed viewpoint, which limits their applicability to monocular endoscopes with varying perspectives. As shown in , reconstructing high-fidelity deformable tissue from endoscopic images is a challenging and arduous task due to three key issues: First, the endoscopic images exhibit invalid occlusions caused by non-soft tissue regions, such as surgical tools, surrounding black and green pixels, and metadata information. The green pixels typically represent regions used for displaying navigation information or providing localization references during endoscopic surgeries. These pixels may overlay the endoscopic image to aid in guiding the surgeon. The surrounding black pixels arise from areas outside the endoscope’s field of view. These black areas are inherent to the imaging setup and do not provide useful information for 3D reconstruction. On the other hand, metadata refers to additional information overlaid on the image, such as real-time metrics (e.g., video bitrate, dropped frames) or textual annotations like video identifiers. Occlusion of invalid regions can interfere with the model’s reconstruction of geometric information, resulting in misalignment or artifacts in the reconstructed scenes. Manually drawing masks to accurately identify soft tissue regions is time-consuming and complex, increasing both the algorithm’s time cost and the operational burden on doctors. At present, there is no dataset specifically distinguishing between soft tissue regions and non-soft tissue regions (i.e., invalid regions). Second, soft tissues have non-rigid properties, and accurately capturing their dynamic deformation over time and with changing viewpoints is a critical challenge in reconstruction. Third, due to lighting limitations and the inherent characteristics of NeRFs, rendered endoscopic images often suffer from poor quality, including uneven lighting, low contrast, and blurry textures, which affect the reconstruction quality of endoscopic scenes. To address the above issues, we propose a novel high-fidelity deformable soft tissue reconstruction framework based on neural radiance fields. First, to resolve the issue of invalid pixel occlusion in endoscopic images, we constructed a dataset specifically for segmenting soft tissue regions. Based on this dataset, we fine-tune SAM  and develop an enhanced version of the model, called SAM-EndoTissue. SAM-EndoTissue enables automatic and robust segmentation and generates soft tissue masks by excluding invalid regions from endoscopic images. Then, we propose a novel tissue mask-guided ray sampling strategy by incorporating tissue masks into the efficient dynamic neural radiance field, Tensor4D . This strategy selectively reconstructs deformable tissue regions, while excluding invalid regions. Additionally, to address blurred textures in rendered views, we incorporate the previously proposed EDAU-Net (Encoder Dual Attention U-Net Network) to enhance the rendered views. The view optimization pipeline is seamlessly integrated with the reconstruction framework to build an end-to-end solution for high-fidelity reconstruction of 3D endoscopic scenes. 

Endoscopic imaging typically relies on discrete representation methods such as point clouds  and mesh grids . These methods use sparse structures to efficiently generate surface models of simple scenes while utilizing additional warp fields to adapt to the dynamic changes of soft tissues. However, discrete representation methods face significant limitations and challenges in complex and dynamic endoscopic scenes. Endoscopic images captured within the human intestine often contain low-texture or no-texture areas that lack matching features. This sparse characteristic may lead to the loss of tissue details during the reconstruction process, thereby reducing reconstruction accuracy. In addition, discrete methods are difficult to adapt to irregular topological structures when dealing with dynamic soft tissues. The continuous representation methods model 3D scenes through implicit or continuous functions, which can capture the fine details and complex dynamic structures in the scene. Neural radiance fields (NeRFs)  have become a typical representative of continuous representation methods due to their efficient reconstruction capability and excellent detail performance. With the development of NeRFs and their extensions , there is enormous potential for novel view generation and 3D reconstruction in dynamic real-world scenarios to support downstream applications such as telemedicine, surgical path planning and simulation, medical education, and surgical training. In medical image processing of CT scans or MRI, NeRFs can reconstruct complex anatomical structures with rich details, potentially reducing patients’ exposure to multiple high-level ionizing radiation or high-resolution scans . For example, MedNeRF  and UMedNeRF  have successfully achieved the rendering of high-quality CT projections from a given few or even a single-view X-ray using a NeRF-based architecture. Iddrisu et al.  reconstructed the 3D geometry and appearance of brain structures from 2D brain MRI images, aiding doctors to more intuitively understand the morphology and location of the lesions. In X-ray image processing, Maas et al.  achieved 3D reconstruction of blood vessels from 2D X-ray angiography images with sparse views and limited angles. Although NeRFs have brought revolutionary progress to the reproduction of 3D scenes in CT, MRIs, and X-ray images, the above methods usually deal with static scenes. For endoscopic images, dynamic soft tissue structures need to be captured and reconstructed. Therefore, these methods still face certain limitations in the 3D reconstruction of endoscopic images. Currently, research in this field focuses on adapting NeRFs to better fit the unique requirements of endoscopy data. Some research works have achieved significant success in reconstructing single-viewpoint stereoscopic endoscopic images by using variants of deformable radiation fields. For example, Wang et al.  proposed EndoNeRF, which first applied D-NeRF  to the 3D reconstruction of binocular endoscopic images, restoring high-quality details of surgical scenes. Considering the constant changes in surgical scenes caused by instrument movement, Endosurf  designed a novel NeRF-based method to reconstruct accurate 3D information from single-viewpoint RGBD images. These models are capable of reconstructing deformable tissue scenes within the human body. However, they are limited to input views with a single viewpoint and require paired binocular endoscopic image data to obtain stereo-depth information. Moreover, these methods are usually time-consuming during both training and rendering. EndoGaussian  utilized the emerging 3D Gaussian Splatting  to improve the efficiency of soft tissue reconstruction in stereoscopic endoscopic videos. However, this method relies on local data points to estimate the overall structure and requires high-quality data to reconstruct accurate 3D models effectively. Soft tissues with limited viewpoints and poor surface features may lead to significant holes or errors in the reconstructed images. Currently, there are few reports on the reconstruction of monocular endoscopic images. Recent advancements demonstrate that modified NeRFs can reconstruct 3D scenes with different objectives, including dynamic scene representation , modeling for sparse views , and optimizing the performance of scene reconstruction or rendering . Due to the unique characteristics of endoscopic images, existing methods are not fully applicable to the 3D reconstruction of such images. Based on the framework of NeRFs, new reconstruction methods are designed for complex scenarios within human cavities and are expected to promote the further development of medical-assisted diagnosis technology. 

shows the framework of our proposed method for high-fidelity deformable soft tissue reconstruction based on NeRFs, which consists of three main modules: tissue region acquisition, deformable soft tissue reconstruction, and view optimization. (1) The tissue region acquisition module mainly utilizes the constructed dataset EndoTissue to fine-tune and retrain SAM to obtain an extended version, named SAM-EndoTissue. SAM-EndoTissue can quickly predict the tissue masks from input views, which directly contributes to the accuracy and efficiency of the deformable soft tissue reconstruction module. (2) The deformable soft tissue reconstruction module uses tissue masks to perform tissue mask-guided ray sampling on Tensor4D to ensure that the trained NeRFs only reconstruct valid dynamic soft tissue scenes. This targeted sampling improves the accuracy and quality of 3D reconstruction in deformable endoscopic scenes. (3) The view optimization module introduces the endoscopic image enhancement network EDAU-Net to enhance the rendered views of the dynamic implicit field. This module improves contrast, detail richness, and visual clarity, making the rendered images more interpretable and clinically applicable. Finally, we can obtain enhanced views. 3.1. Tissue Region Acquisition Currently, there is no dataset used to segment soft tissue regions other than the invalid areas in endoscopic images. The SAM is a state-of-the-art segmentation model designed to segment objects in images with high precision. SAM performs automatic and interactive segmentation, guided by prompts such as points, boxes, or text, enabling flexible application to various scenarios. Unlike standard image processing techniques, SAM benefits from its rich feature representation, trained on a massive dataset of over 1 billion segmentation masks across diverse image domains, which provides robust generalization capabilities. Standard image processing methods can be sensitive to noise, lighting changes, and occlusions. These factors are common in endoscopic environments. SAM’s architecture is specifically designed to generalize well to such complex scenes, maintaining segmentation accuracy even under challenging conditions. SAM is robust and stable for complex scenarios, making it an ideal choice for segmenting specific regions in minimally invasive surgical environments. Moreover, SAM’s fully automatic segmentation mode effectively reduces labor costs. However, SAM is not suitable for most medical image segmentation tasks . In order to train a deep learning network that can accurately and automatically segment soft tissues, this paper constructs the first dataset, EndoTissue, specifically designed for the segmentation of soft tissue regions from endoscopic images. Additionally, a simple fine-tuning strategy is designed to apply SAM to the segmentation task of soft tissue regions of medical images. This ensures that the method does not require additional operations by doctors during the surgical process, thereby avoiding any extra burden. As illustrated in , the invalid regions primarily consist of surgical tools, surrounding black and green areas, and metadata information. For surgical tools, some scholars have constructed datasets specifically for surgical tool segmentation. As shown in a, the dataset includes 10,360 real endoscopic images and corresponding surgical tool segmentation masks (ground truth, GT) selected from the open-source medical imaging datasets: AnnotatedImages , ART-Net Dataset , CholecSeg8k , kvasir-instrument , EndoVis15 , and RoboTool . The images in these datasets are all from endoscopic surgery scenes with invalid regions occluded by surgical tools. Therefore, we no longer need to manually annotate the regions where surgical tools are located. For the surrounding black and green regions, as well as metadata content, the dataset EndoTissue includes a total of 16209 real endoscopic images collected from the following open-source medical imaging dataset, including PolypGen , Nerthus , kvasir , Hyper-Kvasir , Gastrolab , Gastrointestinal-Bleeding , FPPD-13 , ETIS-LaribPolypDB , CVC-EndoSceneStill , CholecSeg8k , and BKAI-IGH NeoPolyp . The images in these datasets are obscured by invalid pixels, such as large green areas and black regions surrounding the image. We invited four experts majoring in digital media technology from universities to manually annotate the soft tissue regions of these endoscopic images using labelme to obtain the corresponding segmentation masks (GT). All ground truths in the dataset EndoTissue are black-and-white binary images. Pixels in the soft tissue regions are set to black, while those in invalid regions are set to white. By observing the endoscopic images, we can see that the sequence images of each scene have the same invalid regions, meaning only one GT needs to be manually annotated by experts for each scene. The labeling process followed strict annotation guidelines, including detailed instructions for the precise labeling of soft tissue regions using the labelme tool and clear definitions of invalid areas in endoscopic images to ensure consistency and accuracy. Selecting images from multiple datasets with different scenes provides rich training samples for the segmentation of soft tissue regions, ensuring that the EndoTissue dataset can adapt to a wide range of medical image segmentation tasks. As shown in b, the invalid regions in endoscopic images consist of two main types. The first type consists of areas occluded by surgical tools. The second type includes surrounding black and green regions as well as metadata information. These two types of datasets are used to fine-tune and retrain the SAM model, respectively, to improve the accuracy of segmenting soft tissue regions, resulting in two specialized models: SAM-EndoTissue1 and SAM-EndoTissue2 (for simplicity, they are collectively referred to as SAM-EndoTissue). To predict the region of interest (ROI) of endoscopic image sequences, we input the images into SAM-EndoTissue1 and SAM-EndoTissue2. The outputs from these two models are then merged to generate the final soft tissue masks that comprehensively represent the valid regions of the endoscopic images. SAM-EndoTissue not only removes black pixels but also excludes green areas, surgical tools, and metadata information, thereby achieving accurate reconstruction of valid soft tissue regions in endoscopic images. SAM-EndoTissue selects SAM’s fully automatic segmentation mode and fine-tunes the mask decoder, freezing the image encoder and prompt encoder. This mode does not require manually adding a bounding box for the selected ROI. Instead, the model automatically generates bounding box prompts of the same size as the original image to achieve segmentation. The fully automatic segmentation mode can improve the segmentation performance and reduce the cost of manual labeling. The tissue masks are automatically obtained by feeding the endoscopic images into the trained SAM-EndoTissue model. 3.2. Deformable Soft Tissue Reconstruction The structures and morphologies in endoscopic image sequences usually change to varying degrees due to factors such as organ peristalsis, respiration, and changes in the force applied by surgical tools. This increases the challenge of reconstructing deformable tissues from endoscopic images. Tensor4D is an advanced method for reconstructing dynamic scenes efficiently and effectively. It is specifically designed to improve computational efficiency compared to the traditional NeRFs. It adopts 4D decomposition for D-NeRF in monocular dynamic scenarios, efficiently capturing the dynamic scenes inside the human body. By adopting hierarchical tri-projection decomposition, Tensor4D reduces memory consumption and achieves a compact representation of 4D spatio-temporal fields. The coarse-to-fine strategy also balances the efficiency of training and the quality of reconstruction, ensuring rapid convergence in the early training stages while preserving fine-grained details in the final output. However, occlusion of invalid regions may limit Tensor4D’s depth perception of objects in the scene. To further improve the accuracy of Tensor4D for endoscopic scene reconstruction, we propose a tissue mask-guided ray sampling strategy. This strategy enhances the reconstructed details and accuracy of the dynamic implicit field by focusing on the sampling of specific soft tissue regions in endoscopic images and further reduces computational overhead by avoiding unnecessary computations in invalid regions. The overall flow of deformable soft tissue reconstruction is shown in . Scene Representation : Tensor4D uses multilayer perceptron (MLP) network E f , E g , and E c to encode a dynamic 3D scene. For a point X = ( x , y , z ) in the scene at time t , we first decompose the 4D tensor into three feature volumes: f z , f y , and f x . These feature volumes are further decomposed into nine flow feature planes, which are input into the flow MLP E f to predict the position offset Δ X = ( Δ x , Δ y , Δ z ) from the starting time to the specified input time t . Then, the position offset is decomposed into three LR and HR feature planes. These feature planes and Δ X = ( Δ x , Δ y , Δ z ) are fed into the geometry MLP E g to obtain the high-dimensional feature H and the volume density σ . Finally, the high-dimensional feature H and the perspective d = ( θ , φ ) are input into the color MLP E c to generate the reconstructed color c = ( r , g , b ) . Tissue Mask-Guided Ray Sampling : The ray casting of Tensor4D is applied to any pixel p in the entire image (i.e., the entire image scene participates in the reconstruction process), with the direction of the ray emitted from the camera center o passing through that pixel denoted as d . The point on the imaging ray corresponding to this pixel is defined as X ( h ) = o + d · h , where h represents the positional parameter of the sampling point along the ray. This method of directly reconstructing the entire image will affect the accuracy of reconstructing the region where the soft tissue is located. The binary mask can locate specific regions of the image, preventing them from participating in the subsequent reconstruction of implicit NeRFs . To overcome the redundant sampling problem in the original Tensor4D reconstruction, this paper proposes a tissue mask-guided ray sampling strategy to optimize the ray sampling process of Tensor4D. Specifically, assuming n images I are input, we first cast rays from pixels randomly selected from these images. The tissue mask I m a s k = 1 acquired by the SAM-EndoTissue model can efficiently distinguish the soft tissue regions and invalid regions of the image. Based on the tissue mask, we determine whether a pixel is located in an invalid region or a valid soft tissue region. If I m a s k i = 0 , the i-th pixel ray is sampled. If I m a s k i = 1 , the i-th pixel ray is not sampled. In the ray casting process, each ray is defined as X ( h ) = o + ω ′ · d · h , where ω ′ represents the weighting factors that guide the rays. The formula for ω ′ is given below: (1) ω ′ = ω ∥ ω ∥ F ω = 1 + ∑ i = 1 n I mask i ∑ i = 1 n I mask i F × 1 − I mask i where ∥ • ∥ F represents the Frobenius normalization. Volume Rendering : Tensor4D predicts the color value C ( p , t ) of the corresponding pixel p in the 2D image based on the color c and density σ of the sampled points in the scene at time t by using the volume rendering Equation : (2) C ( p , t ) = ∫ h n h f τ ( h , t ) σ ( p ( h , t ) ) c ( p ( h , t ) , d ) d h , (3) where p ( h , t ) = X ( h ) + Ψ θ t ( X ( h ) , t ) , (4) [ c ( p ( h , t ) , d ) , σ ( p ( h , t ) ) ] = Ψ θ x ( p ( h , t ) , d ) , (5) and τ ( h , t ) = exp − ∫ h n h σ ( p ( s , t ) ) d s . where τ ( h , t ) is the cumulative transmittance of the ray from the nearest point h n to the farthest point h f . p ( h , t ) represents the 3D point on the camera ray X ( h ) transformed to the canonical space using the deformation network Ψ θ t . Network Training : To train the dynamic NeRF network for the target scene, this paper uses three loss functions to evaluate the difference between the output and GT: feature regularization loss L r , surface constraint loss L e , and color loss L r g b . The total loss function is expressed as follows: (6) L m = λ 1 L r + λ 2 L e + λ 3 L r g b where the coefficients λ 1 , λ 2 , and λ 3 are set to 0.01, 0.2, and 1.0, respectively. L r g b is used to calculate the mean squared error between the pixel values of the rendered image and those of the original image, thereby completing the training of the dynamic NeRFs for the target scene in a self-supervised manner. The loss function L r g b is formulated as follows: (7) L r g b = 1 N s ∑ i = 1 N s C ( p , t ) − C g t ( p , t ) 2 2 where C ( p , t ) is the RGB color value of the image rendered by the canonical network. C g t ( p , t ) is the RGB color value of the input image. N s represents the set of sampling rays in the input view. By minimizing the difference between adjacent elements in the same feature plane, L r can preserve the sparse structural characteristic of the feature plane. The equation is defined below: (8) L r = ∑ T ∑ i , j ( T i + 1 , j − T i , j ) 2 + ( T i , j + 1 − T i , j ) 2 L e can optimize the geometric representation by enhancing the smoothness of the surface. It can be represented by Equation : (9) L e = ∇ S ( x , y , z , t ) 2 − 1 2 where ∇ S denotes the gradient of the surface at position ( x , y , z , t ) , which measures how much a surface varies in space. 3.3. Render View Optimization Although the dynamic NeRFs learned in can effectively reconstruct 3D scenes from endoscopic images, the input endoscopic images usually suffer from issues such as uneven illumination and low contrast due to the unique characteristics of the endoscopic imaging environment and equipment . In addition, NeRFs rely on optimizing the shape and appearance of each 3D spatial location along a light ray solely based on individual pixel RGB values. This inherent limitation may result in the loss of edge details in the rendered views . Therefore, images directly rendered using fully trained NeRFs can sometimes not meet the clinical needs. To further improve the fidelity of rendering views on the inner walls of human body cavities, this paper introduces our previous research work, the Encoder Dual Attention U-Net Network (EDAU-Net) , to enhance the rendered views. EDAU-Net is a novel deep learning-based global image enhancement network specifically designed to enhance endoscopic images. It improves the overall quality of the images by incorporating two innovative components into the U-Net framework: the Detail Attention Map module and the Luminance Attention Map module. The Detail Attention Map module is used to restore fine-grained details, while the Luminance Attention Map module is used to enhance the illumination of endoscopic images. EDAU-Net not only stably improves image texture and detail information but also effectively eliminates non-uniform luminance through supervised training on the large-scale endoscopic image enhancement dataset. The trained dynamic NeRFs generate the rendered image I R e n d e r e d by using the aforementioned volume rendering equation. Then, I R e n d e r e d is input into EDAU-Net to generate the enhanced image I E n h a n c e d : (10) I E n h a n c e d = EDAU - Net ( I R e n d e r e d ; θ ) where θ is the network parameter of EDAU-Net. illustrates the comparison results of rendering views before and after optimization. For the rendered views, the invalid regions excluded during the deformable soft tissue reconstruction phase are reintegrated into the image to provide a comprehensive representation of the surgical scene. After processing by EDAU-Net, the enhanced rendering view shows significant improvements in details, brightness, and contrast, while effectively suppressing noise and artifacts, as shown in . Therefore, the view optimization module not only improves the quality of rendered endoscopic images, but also further optimizes the reconstruction performance of the dynamic NeRFs. 

Currently, there is no dataset used to segment soft tissue regions other than the invalid areas in endoscopic images. The SAM is a state-of-the-art segmentation model designed to segment objects in images with high precision. SAM performs automatic and interactive segmentation, guided by prompts such as points, boxes, or text, enabling flexible application to various scenarios. Unlike standard image processing techniques, SAM benefits from its rich feature representation, trained on a massive dataset of over 1 billion segmentation masks across diverse image domains, which provides robust generalization capabilities. Standard image processing methods can be sensitive to noise, lighting changes, and occlusions. These factors are common in endoscopic environments. SAM’s architecture is specifically designed to generalize well to such complex scenes, maintaining segmentation accuracy even under challenging conditions. SAM is robust and stable for complex scenarios, making it an ideal choice for segmenting specific regions in minimally invasive surgical environments. Moreover, SAM’s fully automatic segmentation mode effectively reduces labor costs. However, SAM is not suitable for most medical image segmentation tasks . In order to train a deep learning network that can accurately and automatically segment soft tissues, this paper constructs the first dataset, EndoTissue, specifically designed for the segmentation of soft tissue regions from endoscopic images. Additionally, a simple fine-tuning strategy is designed to apply SAM to the segmentation task of soft tissue regions of medical images. This ensures that the method does not require additional operations by doctors during the surgical process, thereby avoiding any extra burden. As illustrated in , the invalid regions primarily consist of surgical tools, surrounding black and green areas, and metadata information. For surgical tools, some scholars have constructed datasets specifically for surgical tool segmentation. As shown in a, the dataset includes 10,360 real endoscopic images and corresponding surgical tool segmentation masks (ground truth, GT) selected from the open-source medical imaging datasets: AnnotatedImages , ART-Net Dataset , CholecSeg8k , kvasir-instrument , EndoVis15 , and RoboTool . The images in these datasets are all from endoscopic surgery scenes with invalid regions occluded by surgical tools. Therefore, we no longer need to manually annotate the regions where surgical tools are located. For the surrounding black and green regions, as well as metadata content, the dataset EndoTissue includes a total of 16209 real endoscopic images collected from the following open-source medical imaging dataset, including PolypGen , Nerthus , kvasir , Hyper-Kvasir , Gastrolab , Gastrointestinal-Bleeding , FPPD-13 , ETIS-LaribPolypDB , CVC-EndoSceneStill , CholecSeg8k , and BKAI-IGH NeoPolyp . The images in these datasets are obscured by invalid pixels, such as large green areas and black regions surrounding the image. We invited four experts majoring in digital media technology from universities to manually annotate the soft tissue regions of these endoscopic images using labelme to obtain the corresponding segmentation masks (GT). All ground truths in the dataset EndoTissue are black-and-white binary images. Pixels in the soft tissue regions are set to black, while those in invalid regions are set to white. By observing the endoscopic images, we can see that the sequence images of each scene have the same invalid regions, meaning only one GT needs to be manually annotated by experts for each scene. The labeling process followed strict annotation guidelines, including detailed instructions for the precise labeling of soft tissue regions using the labelme tool and clear definitions of invalid areas in endoscopic images to ensure consistency and accuracy. Selecting images from multiple datasets with different scenes provides rich training samples for the segmentation of soft tissue regions, ensuring that the EndoTissue dataset can adapt to a wide range of medical image segmentation tasks. As shown in b, the invalid regions in endoscopic images consist of two main types. The first type consists of areas occluded by surgical tools. The second type includes surrounding black and green regions as well as metadata information. These two types of datasets are used to fine-tune and retrain the SAM model, respectively, to improve the accuracy of segmenting soft tissue regions, resulting in two specialized models: SAM-EndoTissue1 and SAM-EndoTissue2 (for simplicity, they are collectively referred to as SAM-EndoTissue). To predict the region of interest (ROI) of endoscopic image sequences, we input the images into SAM-EndoTissue1 and SAM-EndoTissue2. The outputs from these two models are then merged to generate the final soft tissue masks that comprehensively represent the valid regions of the endoscopic images. SAM-EndoTissue not only removes black pixels but also excludes green areas, surgical tools, and metadata information, thereby achieving accurate reconstruction of valid soft tissue regions in endoscopic images. SAM-EndoTissue selects SAM’s fully automatic segmentation mode and fine-tunes the mask decoder, freezing the image encoder and prompt encoder. This mode does not require manually adding a bounding box for the selected ROI. Instead, the model automatically generates bounding box prompts of the same size as the original image to achieve segmentation. The fully automatic segmentation mode can improve the segmentation performance and reduce the cost of manual labeling. The tissue masks are automatically obtained by feeding the endoscopic images into the trained SAM-EndoTissue model. 

The structures and morphologies in endoscopic image sequences usually change to varying degrees due to factors such as organ peristalsis, respiration, and changes in the force applied by surgical tools. This increases the challenge of reconstructing deformable tissues from endoscopic images. Tensor4D is an advanced method for reconstructing dynamic scenes efficiently and effectively. It is specifically designed to improve computational efficiency compared to the traditional NeRFs. It adopts 4D decomposition for D-NeRF in monocular dynamic scenarios, efficiently capturing the dynamic scenes inside the human body. By adopting hierarchical tri-projection decomposition, Tensor4D reduces memory consumption and achieves a compact representation of 4D spatio-temporal fields. The coarse-to-fine strategy also balances the efficiency of training and the quality of reconstruction, ensuring rapid convergence in the early training stages while preserving fine-grained details in the final output. However, occlusion of invalid regions may limit Tensor4D’s depth perception of objects in the scene. To further improve the accuracy of Tensor4D for endoscopic scene reconstruction, we propose a tissue mask-guided ray sampling strategy. This strategy enhances the reconstructed details and accuracy of the dynamic implicit field by focusing on the sampling of specific soft tissue regions in endoscopic images and further reduces computational overhead by avoiding unnecessary computations in invalid regions. The overall flow of deformable soft tissue reconstruction is shown in . Scene Representation : Tensor4D uses multilayer perceptron (MLP) network E f , E g , and E c to encode a dynamic 3D scene. For a point X = ( x , y , z ) in the scene at time t , we first decompose the 4D tensor into three feature volumes: f z , f y , and f x . These feature volumes are further decomposed into nine flow feature planes, which are input into the flow MLP E f to predict the position offset Δ X = ( Δ x , Δ y , Δ z ) from the starting time to the specified input time t . Then, the position offset is decomposed into three LR and HR feature planes. These feature planes and Δ X = ( Δ x , Δ y , Δ z ) are fed into the geometry MLP E g to obtain the high-dimensional feature H and the volume density σ . Finally, the high-dimensional feature H and the perspective d = ( θ , φ ) are input into the color MLP E c to generate the reconstructed color c = ( r , g , b ) . Tissue Mask-Guided Ray Sampling : The ray casting of Tensor4D is applied to any pixel p in the entire image (i.e., the entire image scene participates in the reconstruction process), with the direction of the ray emitted from the camera center o passing through that pixel denoted as d . The point on the imaging ray corresponding to this pixel is defined as X ( h ) = o + d · h , where h represents the positional parameter of the sampling point along the ray. This method of directly reconstructing the entire image will affect the accuracy of reconstructing the region where the soft tissue is located. The binary mask can locate specific regions of the image, preventing them from participating in the subsequent reconstruction of implicit NeRFs . To overcome the redundant sampling problem in the original Tensor4D reconstruction, this paper proposes a tissue mask-guided ray sampling strategy to optimize the ray sampling process of Tensor4D. Specifically, assuming n images I are input, we first cast rays from pixels randomly selected from these images. The tissue mask I m a s k = 1 acquired by the SAM-EndoTissue model can efficiently distinguish the soft tissue regions and invalid regions of the image. Based on the tissue mask, we determine whether a pixel is located in an invalid region or a valid soft tissue region. If I m a s k i = 0 , the i-th pixel ray is sampled. If I m a s k i = 1 , the i-th pixel ray is not sampled. In the ray casting process, each ray is defined as X ( h ) = o + ω ′ · d · h , where ω ′ represents the weighting factors that guide the rays. The formula for ω ′ is given below: (1) ω ′ = ω ∥ ω ∥ F ω = 1 + ∑ i = 1 n I mask i ∑ i = 1 n I mask i F × 1 − I mask i where ∥ • ∥ F represents the Frobenius normalization. Volume Rendering : Tensor4D predicts the color value C ( p , t ) of the corresponding pixel p in the 2D image based on the color c and density σ of the sampled points in the scene at time t by using the volume rendering Equation : (2) C ( p , t ) = ∫ h n h f τ ( h , t ) σ ( p ( h , t ) ) c ( p ( h , t ) , d ) d h , (3) where p ( h , t ) = X ( h ) + Ψ θ t ( X ( h ) , t ) , (4) [ c ( p ( h , t ) , d ) , σ ( p ( h , t ) ) ] = Ψ θ x ( p ( h , t ) , d ) , (5) and τ ( h , t ) = exp − ∫ h n h σ ( p ( s , t ) ) d s . where τ ( h , t ) is the cumulative transmittance of the ray from the nearest point h n to the farthest point h f . p ( h , t ) represents the 3D point on the camera ray X ( h ) transformed to the canonical space using the deformation network Ψ θ t . Network Training : To train the dynamic NeRF network for the target scene, this paper uses three loss functions to evaluate the difference between the output and GT: feature regularization loss L r , surface constraint loss L e , and color loss L r g b . The total loss function is expressed as follows: (6) L m = λ 1 L r + λ 2 L e + λ 3 L r g b where the coefficients λ 1 , λ 2 , and λ 3 are set to 0.01, 0.2, and 1.0, respectively. L r g b is used to calculate the mean squared error between the pixel values of the rendered image and those of the original image, thereby completing the training of the dynamic NeRFs for the target scene in a self-supervised manner. The loss function L r g b is formulated as follows: (7) L r g b = 1 N s ∑ i = 1 N s C ( p , t ) − C g t ( p , t ) 2 2 where C ( p , t ) is the RGB color value of the image rendered by the canonical network. C g t ( p , t ) is the RGB color value of the input image. N s represents the set of sampling rays in the input view. By minimizing the difference between adjacent elements in the same feature plane, L r can preserve the sparse structural characteristic of the feature plane. The equation is defined below: (8) L r = ∑ T ∑ i , j ( T i + 1 , j − T i , j ) 2 + ( T i , j + 1 − T i , j ) 2 L e can optimize the geometric representation by enhancing the smoothness of the surface. It can be represented by Equation : (9) L e = ∇ S ( x , y , z , t ) 2 − 1 2 where ∇ S denotes the gradient of the surface at position ( x , y , z , t ) , which measures how much a surface varies in space. 

Although the dynamic NeRFs learned in can effectively reconstruct 3D scenes from endoscopic images, the input endoscopic images usually suffer from issues such as uneven illumination and low contrast due to the unique characteristics of the endoscopic imaging environment and equipment . In addition, NeRFs rely on optimizing the shape and appearance of each 3D spatial location along a light ray solely based on individual pixel RGB values. This inherent limitation may result in the loss of edge details in the rendered views . Therefore, images directly rendered using fully trained NeRFs can sometimes not meet the clinical needs. To further improve the fidelity of rendering views on the inner walls of human body cavities, this paper introduces our previous research work, the Encoder Dual Attention U-Net Network (EDAU-Net) , to enhance the rendered views. EDAU-Net is a novel deep learning-based global image enhancement network specifically designed to enhance endoscopic images. It improves the overall quality of the images by incorporating two innovative components into the U-Net framework: the Detail Attention Map module and the Luminance Attention Map module. The Detail Attention Map module is used to restore fine-grained details, while the Luminance Attention Map module is used to enhance the illumination of endoscopic images. EDAU-Net not only stably improves image texture and detail information but also effectively eliminates non-uniform luminance through supervised training on the large-scale endoscopic image enhancement dataset. The trained dynamic NeRFs generate the rendered image I R e n d e r e d by using the aforementioned volume rendering equation. Then, I R e n d e r e d is input into EDAU-Net to generate the enhanced image I E n h a n c e d : (10) I E n h a n c e d = EDAU - Net ( I R e n d e r e d ; θ ) where θ is the network parameter of EDAU-Net. illustrates the comparison results of rendering views before and after optimization. For the rendered views, the invalid regions excluded during the deformable soft tissue reconstruction phase are reintegrated into the image to provide a comprehensive representation of the surgical scene. After processing by EDAU-Net, the enhanced rendering view shows significant improvements in details, brightness, and contrast, while effectively suppressing noise and artifacts, as shown in . Therefore, the view optimization module not only improves the quality of rendered endoscopic images, but also further optimizes the reconstruction performance of the dynamic NeRFs. 

The experiments were performed in the following computing environment: Windows 10, CPU Intel(R) Xeon(R) E5-2620, CUDA 10.2, Python 3.7, PyTorch 1.5.0, and GPU Nvidia Titan Xp. In our implementation, we train our model using the Adam optimizer and set the iteration number to 200K. The batch size of rays is set to 512, and each is sampled 64 times along the ray. In this paper, we select four different scenes (A, B, C, and D) from the publicly available open-source endoscopic imaging datasets, the Nerthus dataset  and the Gastrolab Image Gallery , for qualitative and quantitative evaluation. Each scene has the characteristics of being obscured by invalid regions and dynamically changing in soft tissues. Moreover, each scene contains 37 to 48 sequential frames, with the size of either 720 × 576 or 640 × 480. illustrates the camera positions corresponding to the images of scenes A-D in 3D space obtained using COLMAP. In this figure, red markers represent the camera positions of the sequence frames used for training. The blue rectangular markers specifically highlight the camera positions of the sequence frames used for testing. We follow community standards  by holding out every eighth image as a test set for evaluating the effectiveness of the reconstruction. Additionally, we can observe the number of images captured from each scene and the spatial trajectory of the camera in . 4.1. Qualitative Evaluation To verify the effectiveness of our proposed method, we qualitatively compare it with state-of-the-art reconstruction methods, including NeRF , D-NeRF , DietNeRF , DS-NeRF , TiNeuVox , NRFF , 4D-GS , and Tensor4D . shows the reconstruction results of various methods for Scene A at two different time points, which further verifies their performance in solving the problems of invalid region occlusion and soft tissue deformation in endoscopic images. The reconstruction results of NeRF reveal noticeable blurring and artifacts when dealing with complex dynamic soft tissue scenes. Similarly, D-NeRF exhibits prominent blurring and artifacts, with minimal improvement compared to NeRF. This is because D-NeRF relies on dense temporal sampling to learn the continuity of dynamic changes. In the endoscopy scenarios, the non-rigid deformation of soft tissues and the interference of invalid pixels make it difficult to accurately capture subtle dynamic changes. DietNeRF struggles to effectively handle the interference caused by black boundaries and surgical tools, resulting in multiple artifacts in the image and blurring in the soft tissue regions, making it difficult to accurately reconstruct the structures of the soft tissues. DS-NeRF incorporates depth information, which helps alleviate the issue of being obstructed by invalid regions to some extent. However, it still produces blurred structures and textures in the synthesized views when handling soft tissue deformations in the scene. TiNeuVox and NRFF struggle to handle the issue of being occluded by complex invalid pixels and cannot accurately reconstruct the geometric structure of the scene, resulting in significant misalignment. Although 4D-GS demonstrates some advantages in handling dynamic scenes, it exhibits a prominent “hollow” phenomenon in the synthesized views at both time points. This limitation arises because the 4D-GS model has sparse Gaussians in regions with few feature points. Tensor4D performs well in handling soft tissue deformation but still cannot avoid the loss of detailed information. In addition, 4D-GS and Tensor4D exhibit pixel misalignment across the rendered image, meaning that noticeable shifts in overall pixel positions are observed in some areas. Our proposed method significantly improves the reconstruction quality of the soft tissue regions. Meanwhile, the proposed method can capture the dynamic changes in the soft tissues and achieve high-fidelity reconstruction of dynamic scenes. shows the reconstructed results of our method and state-of-the-art comparative methods at different viewpoints in scene B. From , we can observe that the soft tissue surfaces reconstructed by NeRF and NRFF exhibit varying degrees of blurring, indicating their limited adaptability to deformations. D-NeRF and TiNeuVox struggle to reconstruct the complete structural information of scene B, resulting in poor visual effects. 4D-GS still cannot reconstruct geometric structures with rich details due to its inherent limitations. The reconstructed soft tissue surfaces by DietNeRF and Tensor4D lack texture details. On the other hand, Tensor4D occasionally produces artifacts in regions undergoing rapid changes. Although DietNeRF performs relatively well in Scene B, it still has limitations in endoscopic scenes with non-rigid deformation of soft tissues and invalid pixel occlusion. Our proposed method achieves superior performance across all viewpoints compared with other reconstruction methods. Specifically, our method demonstrates significant advantages in the visual quality of rendered views, especially in generating texture details on the inner wall surface of the cavity. Moreover, our method can more accurately reconstruct the shape and appearance of the scene. To evaluate the effectiveness of EDAU-Net used in this paper, we compared EDAU-Net with classical 2D image enhancement methods, namely HE , AGCWD , SRIE , and EndoMLE . We designed two experimental setups for comparison. In the first step, 2D enhancement methods were used to pre-process the dataset, followed by using Tensor4D to reconstruct endoscopic tissues, referred to as “* + Tensor4D”. In the second step, Tensor4D was used to reconstruct the endoscopic tissues, and then the 2D enhancement methods were used for post-processing the rendered views, named “Tensor4D + *”. All comparative experiments were conducted with the same parameters such as training epochs, batch size, and learning rate. The synthesized results of novel views for different 2D image enhancement methods are shown in . From , we observe that using image enhancement methods to process endoscopic images can improve the visual quality of images. Different enhancement methods have different applicability to endoscopic images. Notably, HE led to substantial color distortions in the images. SRIE improved the brightness of the image, but lost some detail information. AGCWD and EndoMLE improved the brightness of the image, but EndoMLE produced blurred textures. Compared with other image enhancement methods, EDAU-Net can significantly improve the brightness and contrast of endoscopic images, and obtain the clearest details visually. 4.2. Quantitative Evaluation To comprehensively validate the performance of our proposed method, we used different evaluation metrics tailored to the specific goals of the comparisons. To further validate that our proposed method has excellent reconstruction capability, we evaluated the quality of the rendered views of all methods using three widely used reconstruction image quality assessment metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) . PSNR measures the pixel-level fidelity between reconstructed images and target images, indicating the effect of noise reduction. A higher PSNR indicates better reconstruction accuracy, as it reflects a lower level of noise and distortion. SSIM is designed to mimic the human visual perception of image quality in the context of structural fidelity, where higher values (closer to 1) indicate greater similarity to the target image structure. LPIPS assesses perceptual differences between the reconstructed images and the target images by leveraging deep feature representations. Lower LPIPS values indicate higher perceptual similarity and better preservation of the original scene’s visual integrity. As shown in , the evaluation metric values of TiNeuVox and NRFF are significantly worse than those of other methods. This is because these methods of improving computational efficiency have a certain impact on the quality of the reconstructed scene, resulting in distortion and blurring of the reconstructed results. Among the compared methods, Tensor4D achieved the highest PSNR value of 26.3423 and SSIM value of 0.8152, and 4D-GS achieved the lowest LPIPS value of 0.2166 averaged across the test scenes A-D. In contrast, our proposed method outperformed all other methods with an average PSNR of 29.6029, SSIM of 0.8596, and LPIPS of 0.1866. Compared to the best-performing baseline methods, the proposed method achieved a 12.38% improvement in PSNR, a 5.44% improvement in SSIM, and a 13.86% reduction in LPIPS. This indicates that our method produces novel views with a higher degree of similarity to the GT, accurately reconstructing the intricate structures of soft tissues within the human body. The real endoscopic images do not have corresponding high-quality images (i.e., GT) of the same scene. In order to effectively and objectively evaluate the performance of the EDAU-Net selected in this paper, three no-reference image quality assessment metrics specifically designed for enhancement tasks, namely Entropy, Contrast Improvement Index (CII), and Average Gradient (AG), were used to assess the quality of the enhanced views. Entropy quantifies the richness of image information, indicating the level of detail preserved or enhanced. CII measures the degree of contrast enhancement relative to the original image. AG reflects the sharpness and clarity of edges, which are critical factors for image enhancement. The comparison results are shown in . As observed, the performance of “* + Tensor4D” methods was generally inferior to that of “Tensor4D + *” methods. This may be because image enhancement techniques change the original color information and dynamic range of the image, which causes Tensor4D to learn the wrong scene information. In Scene C, Tensor4D + EDAU-Net outperformed all other methods, achieving improvements of 7.39% in Entropy, 4.89% in CII, and 1.30% in AG. In Scene D, Tensor4D + EDAU-Net demonstrated even greater gains, with improvements of 8.60% in Entropy and 29.66% in CII compared to the optimal values of these comparison methods. Although the AG was slightly lower than that of Tensor4D + HE in Scene D, the overall enhanced quality remained superior due to significant improvements in other metrics. Tensor4D + EDAU-Net achieved optimal comprehensive performance. Therefore, in this paper, we first improved Tensor4D to reconstruct the 3D scenes of endoscopic images, and then used EDAU-Net to post-process and enhance the rendered views. 4.3. Ablation Study The tissue mask-guided ray sampling strategy in utilizes the masks predicted by the SAM-EndoTissue model to reconstruct the deformable soft tissue regions while preventing ray sampling of invalid regions in endoscopic images. To evaluate its impact, we conducted ablation experiments comparing the reconstructed results with and without this module. The reconstruction pipeline was initially executed by removing the tissue mask-guided ray sampling strategy, allowing rays to be sampled uniformly across the entire scene. Next, the same reconstruction pipeline was achieved using the tissue mask-guided ray sampling, restricting sampling to deformable soft tissue regions predicted by the SAM-EndoTissue model. The qualitative and quantitative results are shown in and . a shows the reconstructed result after removing the tissue mask-guided ray sampling. b is the reconstructed result by using the tissue mask-guided ray sampling. b obviously has richer details and higher clarity than a. Moreover, we can see that the PSNR, SSIM, and LPIPS metrics of the rendered view by adopting tissue mask-guided ray sampling are significantly improved, as shown in . This demonstrates that the quality and structural integrity of the generated images are enhanced. In , the EDAU-Net is introduced to enhance the quality of rendered views. Its effectiveness was evaluated through an ablation study that compared the results with and without EDAU-Net processing. The rendered views generated by the reconstruction pipeline were directly compared in two scenarios: (1) without applying EDAU-Net, leaving the rendered views unprocessed, and (2) adopting the EDAU-Net to enhance the rendered views. shows the reconstructed results without and with EDAU-Net processing, respectively. Compared with a,b, it can be seen that after EDAU-Net processing, the structure of the submucosal blood vessels in the dark region of the image is clearer, and the texture information is richer. Moreover, there is no over-enhancement problem in the highlight region. To objectively assess the quality of generated images before and after being processed by EDAU-Net, we use Entropy, AG, and Average Brightness (AB) to evaluate the effect of image enhancement. AB measures the mean pixel intensity of an image, providing an objective indication of overall brightness and illumination. Although higher AB values indicate improved brightness in underexposed regions, excessively high AB values can lead to visual discomfort and loss of detail due to overexposure. Typically, an AB value close to 128 (in the range of 0–255) is considered optimal. The quantitative results are displayed in . The values of Entropy and AG are significantly improved after using EDAU-Net as shown in . This indicates that EDAU-Net effectively improves the details and clarity of the images, making the generated images contain richer details. The proposed method improves AB values compared to the original images. This improvement improves the brightness of the images and effectively addresses underexposed regions, making them more visually interpretable. 4.4. User Study We also conducted a user study involving two experienced clinicians and three senior medical students who had received training in endoscopy-related courses. They mainly evaluated the reconstructed results of NeRF, D-NeRF, DietNeRF, DS-NeRF, TiNeuVox, NRFF, 4D-GS, Tensor4D, and our proposed method on two randomly selected test images. To avoid bias, the images shown to participants were anonymized and the algorithms were typically labeled as Algorithm A, Algorithm B, etc. The experiment was conducted in a quiet and comfortable room to create an optimal environment for evaluation. All participants were assessed on the same high-resolution monitor, ensuring uniform viewing conditions across all sessions. To minimize potential distractions and allow for focused judgment, only one participant was present during each evaluation session. They took no less than 20 min each time. We used a Likert scale (1–5 points, where 1 indicates abysmal performance and 5 indicates excellent performance) and asked participants to rate the reconstructed results of these reconstruction algorithms based on image texture and the detail fidelity, structural integrity, and clinical applicability of the generated images. Before scoring, the three assessment criteria were briefly explained to these participants, and the output examples were presented to standardize their understanding of the scoring process. After completing the scoring, we immediately engaged each participant in a follow-up conversation to collect the qualitative feedback through open-ended questions. Each participant spent approximately 7–14 min participating in these conversations. Participants were encouraged to share specific thoughts on the strengths, weaknesses, and potential areas for improvement of these algorithms. Feedback was collected from five participants. The results of the user study are shown in . shows the average values of different test cases scored by different participants considering different aspects. As can be seen from , both clinicians and medical students evaluated our proposed method as significantly superior to these comparative algorithms in terms of the fidelity of texture and details of the reconstructed images. Earlier proposed NeRFs such as NeRF and D-NeRF had lower scores. In particular, D-NeRF only scored 1.3 points, indicating that it performed poorly in terms of fidelity of textures and details. In terms of restoring structural integrity, our proposed method received the highest average score (4.7). Compared with other comparison methods, Tensor4D performed better (reaching 4.2 points), but it was still inferior to our reconstruction method. TiNeuVox performed poorly in terms of structural integrity, with an average score of only 1.1. In this paper, our proposed 3D reconstruction method received the highest average score in terms of clinical applicability. The evaluation of the participants shows the potential of our method to be applied in clinical settings. However, other comparative algorithms such as NeRF, D-NeRF, DietNeRF, DS-NeRF, TiNeuVox, and NRFF had lower scores, indicating that their clinical application value is limited. Observing the line graph, it can be seen that the overall values of these algorithms show a trend of gradually increasing. This is because with the continuous development of NeRFs, the effect of such methods on scene reconstruction is constantly improving. In addition to the Likert scale ratings, we collected qualitative feedback from the five participants through open-ended questions. Their comments provided valuable insights into the clinical applicability and potential limitations of the proposed 3D reconstruction method. Several participants emphasized the potential advantages of the framework for surgical planning and diagnostics. For example, the second participant noted “The reconstruction models effectively capture structural integrity, which is critical for pre-surgical evaluation”. Similarly, the fourth participant commented that “The texture fidelity and spatial consistency of the reconstructed results are promising for training and educational purposes”. Furthermore, three participants, including experienced clinicians, emphasized the importance of optimizing both computational speed and system usability to enhance the framework’s feasibility for real-time clinical use. For example, the third participant highlighted that “A simplified user interface would be beneficial to improve workflow efficiency during routine clinical tasks”. The first participant raised a concern about “Whether the reconstruction speed could meet the demands of a live surgery scenario”. The fifth participant suggested that “Further optimization is needed to ensure the method can effectively reconstruct dynamic soft tissues”. This feedback highlights both the clinical potential of our method and the importance of addressing practical challenges. Feedback from participants provides valuable guidance for future research directions. 

To verify the effectiveness of our proposed method, we qualitatively compare it with state-of-the-art reconstruction methods, including NeRF , D-NeRF , DietNeRF , DS-NeRF , TiNeuVox , NRFF , 4D-GS , and Tensor4D . shows the reconstruction results of various methods for Scene A at two different time points, which further verifies their performance in solving the problems of invalid region occlusion and soft tissue deformation in endoscopic images. The reconstruction results of NeRF reveal noticeable blurring and artifacts when dealing with complex dynamic soft tissue scenes. Similarly, D-NeRF exhibits prominent blurring and artifacts, with minimal improvement compared to NeRF. This is because D-NeRF relies on dense temporal sampling to learn the continuity of dynamic changes. In the endoscopy scenarios, the non-rigid deformation of soft tissues and the interference of invalid pixels make it difficult to accurately capture subtle dynamic changes. DietNeRF struggles to effectively handle the interference caused by black boundaries and surgical tools, resulting in multiple artifacts in the image and blurring in the soft tissue regions, making it difficult to accurately reconstruct the structures of the soft tissues. DS-NeRF incorporates depth information, which helps alleviate the issue of being obstructed by invalid regions to some extent. However, it still produces blurred structures and textures in the synthesized views when handling soft tissue deformations in the scene. TiNeuVox and NRFF struggle to handle the issue of being occluded by complex invalid pixels and cannot accurately reconstruct the geometric structure of the scene, resulting in significant misalignment. Although 4D-GS demonstrates some advantages in handling dynamic scenes, it exhibits a prominent “hollow” phenomenon in the synthesized views at both time points. This limitation arises because the 4D-GS model has sparse Gaussians in regions with few feature points. Tensor4D performs well in handling soft tissue deformation but still cannot avoid the loss of detailed information. In addition, 4D-GS and Tensor4D exhibit pixel misalignment across the rendered image, meaning that noticeable shifts in overall pixel positions are observed in some areas. Our proposed method significantly improves the reconstruction quality of the soft tissue regions. Meanwhile, the proposed method can capture the dynamic changes in the soft tissues and achieve high-fidelity reconstruction of dynamic scenes. shows the reconstructed results of our method and state-of-the-art comparative methods at different viewpoints in scene B. From , we can observe that the soft tissue surfaces reconstructed by NeRF and NRFF exhibit varying degrees of blurring, indicating their limited adaptability to deformations. D-NeRF and TiNeuVox struggle to reconstruct the complete structural information of scene B, resulting in poor visual effects. 4D-GS still cannot reconstruct geometric structures with rich details due to its inherent limitations. The reconstructed soft tissue surfaces by DietNeRF and Tensor4D lack texture details. On the other hand, Tensor4D occasionally produces artifacts in regions undergoing rapid changes. Although DietNeRF performs relatively well in Scene B, it still has limitations in endoscopic scenes with non-rigid deformation of soft tissues and invalid pixel occlusion. Our proposed method achieves superior performance across all viewpoints compared with other reconstruction methods. Specifically, our method demonstrates significant advantages in the visual quality of rendered views, especially in generating texture details on the inner wall surface of the cavity. Moreover, our method can more accurately reconstruct the shape and appearance of the scene. To evaluate the effectiveness of EDAU-Net used in this paper, we compared EDAU-Net with classical 2D image enhancement methods, namely HE , AGCWD , SRIE , and EndoMLE . We designed two experimental setups for comparison. In the first step, 2D enhancement methods were used to pre-process the dataset, followed by using Tensor4D to reconstruct endoscopic tissues, referred to as “* + Tensor4D”. In the second step, Tensor4D was used to reconstruct the endoscopic tissues, and then the 2D enhancement methods were used for post-processing the rendered views, named “Tensor4D + *”. All comparative experiments were conducted with the same parameters such as training epochs, batch size, and learning rate. The synthesized results of novel views for different 2D image enhancement methods are shown in . From , we observe that using image enhancement methods to process endoscopic images can improve the visual quality of images. Different enhancement methods have different applicability to endoscopic images. Notably, HE led to substantial color distortions in the images. SRIE improved the brightness of the image, but lost some detail information. AGCWD and EndoMLE improved the brightness of the image, but EndoMLE produced blurred textures. Compared with other image enhancement methods, EDAU-Net can significantly improve the brightness and contrast of endoscopic images, and obtain the clearest details visually. 

To comprehensively validate the performance of our proposed method, we used different evaluation metrics tailored to the specific goals of the comparisons. To further validate that our proposed method has excellent reconstruction capability, we evaluated the quality of the rendered views of all methods using three widely used reconstruction image quality assessment metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) . PSNR measures the pixel-level fidelity between reconstructed images and target images, indicating the effect of noise reduction. A higher PSNR indicates better reconstruction accuracy, as it reflects a lower level of noise and distortion. SSIM is designed to mimic the human visual perception of image quality in the context of structural fidelity, where higher values (closer to 1) indicate greater similarity to the target image structure. LPIPS assesses perceptual differences between the reconstructed images and the target images by leveraging deep feature representations. Lower LPIPS values indicate higher perceptual similarity and better preservation of the original scene’s visual integrity. As shown in , the evaluation metric values of TiNeuVox and NRFF are significantly worse than those of other methods. This is because these methods of improving computational efficiency have a certain impact on the quality of the reconstructed scene, resulting in distortion and blurring of the reconstructed results. Among the compared methods, Tensor4D achieved the highest PSNR value of 26.3423 and SSIM value of 0.8152, and 4D-GS achieved the lowest LPIPS value of 0.2166 averaged across the test scenes A-D. In contrast, our proposed method outperformed all other methods with an average PSNR of 29.6029, SSIM of 0.8596, and LPIPS of 0.1866. Compared to the best-performing baseline methods, the proposed method achieved a 12.38% improvement in PSNR, a 5.44% improvement in SSIM, and a 13.86% reduction in LPIPS. This indicates that our method produces novel views with a higher degree of similarity to the GT, accurately reconstructing the intricate structures of soft tissues within the human body. The real endoscopic images do not have corresponding high-quality images (i.e., GT) of the same scene. In order to effectively and objectively evaluate the performance of the EDAU-Net selected in this paper, three no-reference image quality assessment metrics specifically designed for enhancement tasks, namely Entropy, Contrast Improvement Index (CII), and Average Gradient (AG), were used to assess the quality of the enhanced views. Entropy quantifies the richness of image information, indicating the level of detail preserved or enhanced. CII measures the degree of contrast enhancement relative to the original image. AG reflects the sharpness and clarity of edges, which are critical factors for image enhancement. The comparison results are shown in . As observed, the performance of “* + Tensor4D” methods was generally inferior to that of “Tensor4D + *” methods. This may be because image enhancement techniques change the original color information and dynamic range of the image, which causes Tensor4D to learn the wrong scene information. In Scene C, Tensor4D + EDAU-Net outperformed all other methods, achieving improvements of 7.39% in Entropy, 4.89% in CII, and 1.30% in AG. In Scene D, Tensor4D + EDAU-Net demonstrated even greater gains, with improvements of 8.60% in Entropy and 29.66% in CII compared to the optimal values of these comparison methods. Although the AG was slightly lower than that of Tensor4D + HE in Scene D, the overall enhanced quality remained superior due to significant improvements in other metrics. Tensor4D + EDAU-Net achieved optimal comprehensive performance. Therefore, in this paper, we first improved Tensor4D to reconstruct the 3D scenes of endoscopic images, and then used EDAU-Net to post-process and enhance the rendered views. 

The tissue mask-guided ray sampling strategy in utilizes the masks predicted by the SAM-EndoTissue model to reconstruct the deformable soft tissue regions while preventing ray sampling of invalid regions in endoscopic images. To evaluate its impact, we conducted ablation experiments comparing the reconstructed results with and without this module. The reconstruction pipeline was initially executed by removing the tissue mask-guided ray sampling strategy, allowing rays to be sampled uniformly across the entire scene. Next, the same reconstruction pipeline was achieved using the tissue mask-guided ray sampling, restricting sampling to deformable soft tissue regions predicted by the SAM-EndoTissue model. The qualitative and quantitative results are shown in and . a shows the reconstructed result after removing the tissue mask-guided ray sampling. b is the reconstructed result by using the tissue mask-guided ray sampling. b obviously has richer details and higher clarity than a. Moreover, we can see that the PSNR, SSIM, and LPIPS metrics of the rendered view by adopting tissue mask-guided ray sampling are significantly improved, as shown in . This demonstrates that the quality and structural integrity of the generated images are enhanced. In , the EDAU-Net is introduced to enhance the quality of rendered views. Its effectiveness was evaluated through an ablation study that compared the results with and without EDAU-Net processing. The rendered views generated by the reconstruction pipeline were directly compared in two scenarios: (1) without applying EDAU-Net, leaving the rendered views unprocessed, and (2) adopting the EDAU-Net to enhance the rendered views. shows the reconstructed results without and with EDAU-Net processing, respectively. Compared with a,b, it can be seen that after EDAU-Net processing, the structure of the submucosal blood vessels in the dark region of the image is clearer, and the texture information is richer. Moreover, there is no over-enhancement problem in the highlight region. To objectively assess the quality of generated images before and after being processed by EDAU-Net, we use Entropy, AG, and Average Brightness (AB) to evaluate the effect of image enhancement. AB measures the mean pixel intensity of an image, providing an objective indication of overall brightness and illumination. Although higher AB values indicate improved brightness in underexposed regions, excessively high AB values can lead to visual discomfort and loss of detail due to overexposure. Typically, an AB value close to 128 (in the range of 0–255) is considered optimal. The quantitative results are displayed in . The values of Entropy and AG are significantly improved after using EDAU-Net as shown in . This indicates that EDAU-Net effectively improves the details and clarity of the images, making the generated images contain richer details. The proposed method improves AB values compared to the original images. This improvement improves the brightness of the images and effectively addresses underexposed regions, making them more visually interpretable. 

We also conducted a user study involving two experienced clinicians and three senior medical students who had received training in endoscopy-related courses. They mainly evaluated the reconstructed results of NeRF, D-NeRF, DietNeRF, DS-NeRF, TiNeuVox, NRFF, 4D-GS, Tensor4D, and our proposed method on two randomly selected test images. To avoid bias, the images shown to participants were anonymized and the algorithms were typically labeled as Algorithm A, Algorithm B, etc. The experiment was conducted in a quiet and comfortable room to create an optimal environment for evaluation. All participants were assessed on the same high-resolution monitor, ensuring uniform viewing conditions across all sessions. To minimize potential distractions and allow for focused judgment, only one participant was present during each evaluation session. They took no less than 20 min each time. We used a Likert scale (1–5 points, where 1 indicates abysmal performance and 5 indicates excellent performance) and asked participants to rate the reconstructed results of these reconstruction algorithms based on image texture and the detail fidelity, structural integrity, and clinical applicability of the generated images. Before scoring, the three assessment criteria were briefly explained to these participants, and the output examples were presented to standardize their understanding of the scoring process. After completing the scoring, we immediately engaged each participant in a follow-up conversation to collect the qualitative feedback through open-ended questions. Each participant spent approximately 7–14 min participating in these conversations. Participants were encouraged to share specific thoughts on the strengths, weaknesses, and potential areas for improvement of these algorithms. Feedback was collected from five participants. The results of the user study are shown in . shows the average values of different test cases scored by different participants considering different aspects. As can be seen from , both clinicians and medical students evaluated our proposed method as significantly superior to these comparative algorithms in terms of the fidelity of texture and details of the reconstructed images. Earlier proposed NeRFs such as NeRF and D-NeRF had lower scores. In particular, D-NeRF only scored 1.3 points, indicating that it performed poorly in terms of fidelity of textures and details. In terms of restoring structural integrity, our proposed method received the highest average score (4.7). Compared with other comparison methods, Tensor4D performed better (reaching 4.2 points), but it was still inferior to our reconstruction method. TiNeuVox performed poorly in terms of structural integrity, with an average score of only 1.1. In this paper, our proposed 3D reconstruction method received the highest average score in terms of clinical applicability. The evaluation of the participants shows the potential of our method to be applied in clinical settings. However, other comparative algorithms such as NeRF, D-NeRF, DietNeRF, DS-NeRF, TiNeuVox, and NRFF had lower scores, indicating that their clinical application value is limited. Observing the line graph, it can be seen that the overall values of these algorithms show a trend of gradually increasing. This is because with the continuous development of NeRFs, the effect of such methods on scene reconstruction is constantly improving. In addition to the Likert scale ratings, we collected qualitative feedback from the five participants through open-ended questions. Their comments provided valuable insights into the clinical applicability and potential limitations of the proposed 3D reconstruction method. Several participants emphasized the potential advantages of the framework for surgical planning and diagnostics. For example, the second participant noted “The reconstruction models effectively capture structural integrity, which is critical for pre-surgical evaluation”. Similarly, the fourth participant commented that “The texture fidelity and spatial consistency of the reconstructed results are promising for training and educational purposes”. Furthermore, three participants, including experienced clinicians, emphasized the importance of optimizing both computational speed and system usability to enhance the framework’s feasibility for real-time clinical use. For example, the third participant highlighted that “A simplified user interface would be beneficial to improve workflow efficiency during routine clinical tasks”. The first participant raised a concern about “Whether the reconstruction speed could meet the demands of a live surgery scenario”. The fifth participant suggested that “Further optimization is needed to ensure the method can effectively reconstruct dynamic soft tissues”. This feedback highlights both the clinical potential of our method and the importance of addressing practical challenges. Feedback from participants provides valuable guidance for future research directions. 

In this paper, we constructed a dataset EndoTissue for soft tissue segmentation of endoscopic images for the first time. The EndoTissue dataset addresses the challenge of automatically segmenting soft tissue regions in endoscopic images, facilitating the effective and stable reconstruction of 3D scenes from endoscopic images. To achieve high-quality 3D reconstruction of deformable soft tissues in endoscopic images, a novel high-fidelity soft tissue reconstruction method is proposed in this paper. Our proposed method not only accounts for deformation factors but also introduces tissue mask-guided ray sampling and the EDAU-Net to improve the accuracy and stability of reconstruction, enabling the high-fidelity reconstruction of soft tissue structures from endoscopic image sequences. Experimental results show that compared with the state-of-the-art reconstruction methods, our proposed method has significant advantages in handling invalid region occlusion and dynamic changes in soft tissues in different scenes. It has also been highly praised by doctors and medical students. Therefore, the proposed method has important scientific research and application value in the field of endoscopic image processing. In the future, we plan to focus on the automatic and accurate segmentation of lesion regions in reconstructed endoscopic scenes. We are committed to providing reliable solutions for identifying and analyzing pathological areas. Additionally, based on the feedback from participants, we will focus on optimizing computational pipelines and developing user-friendly interfaces. This exploration is expected to improve diagnostic accuracy while providing valuable support for preoperative planning and intraoperative decision-making. 

