Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving
<|reference_start|>Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving: Efficient data utilization is crucial for advancing 3D scene understanding in autonomous driving, where reliance on heavily human-annotated LiDAR point clouds challenges fully supervised methods. Addressing this, our study extends into semi-supervised learning for LiDAR semantic segmentation, leveraging the intrinsic spatial priors of driving scenes and multi-sensor complements to augment the efficacy of unlabeled datasets. We introduce LaserMix++, an evolved framework that integrates laser beam manipulations from disparate LiDAR scans and incorporates LiDAR-camera correspondences to further assist data-efficient learning. Our framework is tailored to enhance 3D scene consistency regularization by incorporating multi-modality, including 1) multi-modal LaserMix operation for fine-grained cross-sensor interactions; 2) camera-to-LiDAR feature distillation that enhances LiDAR feature learning; and 3) language-driven knowledge guidance generating auxiliary supervisions using open-vocabulary models. The versatility of LaserMix++ enables applications across LiDAR representations, establishing it as a universally applicable solution. Our framework is rigorously validated through theoretical analysis and extensive experiments on popular driving perception datasets. Results demonstrate that LaserMix++ markedly outperforms fully supervised alternatives, achieving comparable accuracy with five times fewer annotations and significantly improving the supervised-only baselines. This substantial advancement underscores the potential of semi-supervised approaches in reducing the reliance on extensive labeled data in LiDAR-based 3D scene understanding systems.<|reference_end|>
@article{kong2024multi-modal,
    title={Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving},
    author={Lingdong Kong and Xiang Xu and Jiawei Ren and Wenwei Zhang and Liang
  Pan and Kai Chen and Wei Tsang Ooi and Ziwei Liu},
    journal={arXiv preprint arXiv:2405.05258},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.05258},
    primaryClass={cs.CV cs.LG cs.RO}
}