Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
<|reference_start|>Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding: Recent vision-language pre-training models have exhibited remarkable generalization ability in zero-shot recognition tasks. Previous open-vocabulary 3D scene understanding methods mostly focus on training 3D models using either image or text supervision while neglecting the collective strength of all modalities. In this work, we propose a Dense Multimodal Alignment (DMA) framework to densely co-embed different modalities into a common space for maximizing their synergistic benefits. Instead of extracting coarse view- or region-level text prompts, we leverage large vision-language models to extract complete category information and scalable scene descriptions to build the text modality, and take image modality as the bridge to build dense point-pixel-text associations. Besides, in order to enhance the generalization ability of the 2D model for downstream 3D tasks without compromising the open-vocabulary capability, we employ a dual-path integration approach to combine frozen CLIP visual features and learnable mask features. Extensive experiments show that our DMA method produces highly competitive open-vocabulary segmentation performance on various indoor and outdoor tasks.<|reference_end|>
@article{li2024dense,
    title={Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding},
    author={Ruihuang Li, Zhengqiang Zhang, Chenhang He, Zhiyuan Ma, Vishal M.
  Patel, Lei Zhang},
    journal={arXiv preprint arXiv:2407.09781},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.09781},
    primaryClass={cs.CV}
}