Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets
<|reference_start|>Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets: Surgical scene understanding is a key technical component for enabling intelligent and context aware systems that can transform various aspects of surgical interventions. In this work, we focus on the semantic segmentation task, propose a simple yet effective multi-modal (RGB and depth) training framework called SurgDepth, and show state-of-the-art (SOTA) results on all publicly available datasets applicable for this task. Unlike previous approaches, which either fine-tune SOTA segmentation models trained on natural images, or encode RGB or RGB-D information using RGB only pre-trained backbones, SurgDepth, which is built on top of Vision Transformers (ViTs), is designed to encode both RGB and depth information through a simple fusion mechanism. We conduct extensive experiments on benchmark datasets including EndoVis2022, AutoLapro, LapI2I and EndoVis2017 to verify the efficacy of SurgDepth. Specifically, SurgDepth achieves a new SOTA IoU of 0.86 on EndoVis 2022 SAR-RARP50 challenge and outperforms the current best method by at least 4%, using a shallow and compute efficient decoder consisting of ConvNeXt blocks.<|reference_end|>
@article{jamal2024rethinking,
    title={Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets},
    author={Muhammad Abdullah Jamal, Omid Mohareri},
    journal={arXiv preprint arXiv:2407.19714},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19714},
    primaryClass={cs.CV cs.AI}
}