Sparse Refinement for Efficient High-Resolution Semantic Segmentation
<|reference_start|>Sparse Refinement for Efficient High-Resolution Semantic Segmentation: Semantic segmentation empowers numerous real-world applications, such as autonomous driving and augmented/mixed reality. These applications often operate on high-resolution images (e.g., 8 megapixels) to capture the fine details. However, this comes at the cost of considerable computational complexity, hindering the deployment in latency-sensitive scenarios. In this paper, we introduce SparseRefine, a novel approach that enhances dense low-resolution predictions with sparse high-resolution refinements. Based on coarse low-resolution outputs, SparseRefine first uses an entropy selector to identify a sparse set of pixels with high entropy. It then employs a sparse feature extractor to efficiently generate the refinements for those pixels of interest. Finally, it leverages a gated ensembler to apply these sparse refinements to the initial coarse predictions. SparseRefine can be seamlessly integrated into any existing semantic segmentation model, regardless of CNN- or ViT-based. SparseRefine achieves significant speedup: 1.5 to 3.7 times when applied to HRNet-W48, SegFormer-B5, Mask2Former-T/L and SegNeXt-L on Cityscapes, with negligible to no loss of accuracy. Our "dense+sparse" paradigm paves the way for efficient high-resolution visual computing.<|reference_end|>
@article{liu2024sparse,
    title={Sparse Refinement for Efficient High-Resolution Semantic Segmentation},
    author={Zhijian Liu, Zhuoyang Zhang, Samir Khaki, Shang Yang, Haotian Tang,
  Chenfeng Xu, Kurt Keutzer, Song Han},
    journal={arXiv preprint arXiv:2407.19014},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19014},
    primaryClass={cs.CV}
}